Opnsense with Lightsail AWS VPS - Howto

[ljvb]

First post.. hopefully useful. 

I just switched over from PFSense.
Basic details of my network and setup (which will lead to my use of a VPS server).
Internal network
FIOS Internet to Opnsense Virtual machine with 2 intel gig cards via PCI Passthrough.  LAN Interface to Brocade switch with multiple vlans, trunked to VMWare Distributed switch for Virtual machines.  And then a bunch of other stuff not related to this post.

I tend to route all my traffic through a VPS running openvpn, previously I was running vanilla Freebsd 12 with machine implementations of openvpn using certificates.
Liightsail (aka AWS light) was chosen for a particular reason.  Sadly they do no offer console access like some other providers (I also use VULTR).  Everything had to be done via SSH with Lightsail.  Additionally, Lightsail does not offer FreeBSD 11.  This is a problem for Opnsense.

The reason I am dead set on using Lightsail, is that it uses AWS ip address space, and people companies Netflix and Amazon Prime block proxies/vpns (I have not checked Hulu), but as they are generally also AWS customers, they do not block AWS address space.  So my streaming is not affected unlike with Vultr and others I have tried.

So, since Lightsail does not offer FreeBSD 11, this posed a problem.  And here is the painful (and slow depending on your resources) solution which involves recompiling everything from scratch.

Spin up a FreeBSD 12 instance

Pray
(Optional) configure NFS or SSHFS, or similar, carefully configure your PF rules as you are transiting public networks.
(Optional) export /usr/src and /usr/obj from your Lightsail instance (remember to adjust the Lightsail firewall rules in addition to your local pf rules)
(Optional) mount /usr/src and /usr/obj on whichever machine you are going to compile the sources.

Regardless of whether you are using a build server or not, the process from here is the same.

Update your sources

Code: [Select]

svnlite checkout https://svn.freebsd.org/base/releases/11.2.0 /usr/src
Build your sources.  This will take a while no matter what you do.  I used a VM with 8 CPU cores, adjust your -j value according to your system specs, I typically use -j8 for 8, -j4 for 4 cores, etc.  Doing this on the VPS is painfully slow, moderately slow if you don't have gig internet service.  There are a few ways to do this, I just did it this way.

Code: [Select]

make -j8 buildworld buildkernel
Take a break, go eat dinner, watch a movie, catch a flight to a foreign country..  Took me about 30 min using my build server, the first time around using sshfs it took around 6 hours, will all depend on your resources.

Install the kernel first (you can do it either way, and generally you are supposed to boot between kernel and world, but you can get away without it)

Code: [Select]

make installkernel
make installworld

Merge the configs, this can be confusing at times, google mergemaster, I hate it.. but what can I do.

Code: [Select]

mergemaster -p
After making sure you still have a functioning password file (mergemaster has on occasion screwed up my password file, if it did, manually add and fix users and groups you need to start the bootstrap process, after which it makes no difference).

Fix your packages, Lightsail uses dual-dhcp and a few other packages.

Code: [Select]

pkg-static -f install pkg
pkg update
pkg upgrade

Reboot and pray
Assuming everything went well, you should be booted into a FreeBSD 11.2 instance.

Now, we can start the Opnsense installation we need to make a few changes.  I pulled this from the Github page for the Bootstrap tool

Code: [Select]

# pkg install ca_root_nss
# fetch https://raw.githubusercontent.com/opnsense/update/master/bootstrap/opnsense-bootstrap.sh
Before running the utility, edit it and comment out the reboot at the bottom of the file

Code: [Select]

# sh ./opnsense-bootstrap.sh

After this finished

You will need to edit /usr/local/etc/config.xml
Search for 192.168.  I change

Code: [Select]

<lan>
      <enable>1</enable>
      <if>mismatch0</if>
      <ipaddr>192.168.1.1</ipaddr>
      <subnet>24</subnet>
      <ipaddrv6>track6</ipaddrv6>
      <subnetv6>64</subnetv6>
      <media/>
      <mediaopt/>
      <track6-interface>wan</track6-interface>
      <track6-prefix-id>0</track6-prefix-id>
    </lan>
to

Code: [Select]

<lan>
      <enable>1</enable>
      <if>mismatch0</if>
      <ipaddr>dhcp</ipaddr>
      <subnet/>
      <gateway/>
      <ipaddrv6>track6</ipaddrv6>
      <subnetv6>64</subnetv6>
      <media/>
      <mediaopt/>
      <track6-interface>wan</track6-interface>
      <track6-prefix-id>0</track6-prefix-id>
    </lan>

If you don't have another route to the 172.26.x.x (or whatever your private address space assigned is), you will need to make sure that the refferer checks are disabled, as Lightsail does static real to rfc1918 address mapping. 

Code: [Select]

touch /tmp/disable_security_checks
chflags schg /tmp/disable_security_checks
The chflags command is to make sure /tmp is not cleaned on boot up.

After you make the changes to the config file, you can manually reboot.

If all went well, you should be able to pull up the web page to continue with the install. 

Go through and configure.  Be warned though, if you try to change the LAN to WAN you will get disconnected, and then have to start all over again..  Use snapshots judiciously between each major step..

I currently have my home Opnsense gateway connected to the Lightsail one via site to site VPN.  I am still figuring out Opnsense, a little bit of a change from PFSense and vanilla FBSD for me, and I only started playing with Opnsense for about 4 days (as of this post).

Good luck.. have fun..