Messages

1

General Discussion / Bhyve on OPNsense for virtualization in 2023

« on: March 04, 2023, 10:17:25 pm »

Greetings. Following in the footsteps of slipperyduck &co, I managed to convert my OPNsense router into a virtualization host, and I'd like to register my positive signal. One could debate the merits of running OPNsense on bare metal to host virtual machines vs running OPNsense itself as a virtual machine alongside other virtual machines vs using separate physical machines. Suffice it to say that there may be security implications of using OPNsense as a virtualization host that are unknown as of yet because this is an uncommon setup.

Don't use this in production. Do try this in your lab so we could exercise this code path and work out the kinks

Anyway, it all begins with the installation of the popular vm-bhyve system and required GRUB loader from the FreeBSD repository:

Code: [Select]

pkg lock -y pkg
sed -i '' 's/enabled: no/enabled: yes/' /usr/local/etc/pkg/repos/FreeBSD.conf
pkg install -y vm-bhyve grub2-bhyve
sed -i '' 's/enabled: yes/enabled: no/' /usr/local/etc/pkg/repos/FreeBSD.conf
pkg unlock -y pkg
Next, we prepare vm-bhyve (I'm using ZFS here):

Code: [Select]

zfs create zroot/vm
sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/vm"
vm init
cp /usr/local/share/examples/vm-bhyve/* /zroot/vm/.templates/
Here, we return to OPNsense's web UI to configure the network bridge. To make things easy, we could create a bridge for the LAN and then (later) use the same bridge for the VMs. My bridge is called bridge0 (the device name in the Device field).

In addition, I recommend doing packet filtering on the bridge interface instead of the individual ports, so we could connect any number of VMs without reconfiguring the firewall. In System > Settings > Tunables, set net.link.bridge.pfil_member=0 and net.link.bridge.pfil_bridge=1 (sysctl works too, but the settings wouldn't survive reboot).

Back to the command line, we tell vm-bhyve about our bridge:

Code: [Select]

vm switch create -t manual -b bridge0 public
Now let's choose an OS for our VM. I'm going lightweight with Alpine:

Code: [Select]

vm iso https://dl-cdn.alpinelinux.org/alpine/v3.17/releases/x86_64/alpine-standard-3.17.2-x86_64.iso
The Alpine template requires a little tweak:

Code: [Select]

sed -i '' 's/vanilla/lts/g' /zroot/vm/.templates/alpine.conf
And now, the moment of truth...

Code: [Select]

vm create -t alpine -s 1G myfirstvm
vm install -f myfirstvm alpine-standard-3.17.2-x86_64.iso
We should now be able to install the OS. I don't know how to exit the nmdm console during installation (~ ^D did not work for me here) short of disconnecting the SSH session; maybe y'all could help me out

After installation, we should be able to use the vm start myfirstvm and vm stop myfirstvm commands to manage the VM. Running vm console myfirstvm connects to the console (and here, ~ ^D did work to disconnect).

If all works well, we could make this VM start upon host boot:

Code: [Select]

sysrc vm_list="myfirstvm"
Protip: The bhyvectl tool could be used to manage the VMs directly if vm-bhyve ran into issues. If bhyvectl were used to stop a VM, the /zroot/vm/<name>/run.lock file should be removed to avoid confusing vm-bhyve.

Now all we need is a slick web UI