The Bhyve question

[slipperyduck]

My Small LIVA Z Plus Dual LAN BOX repurposed as a Firewall/Router.

Please note as has been said many times in the past, that running a VM on a firewall has some security implications and this is not recommended at all.
Whilst I will try make the installation a bit more secure, I have not completed penetration testing on it, so cannot vouch for the impact it has overall.
This excercise is intended for use as a homelab or development/experimental environment and is NOT recommended for a production environment.

I have seen quite a lot of discussions on the Web relating to running VM on OPNSense natively, but have not seen any actual implimentations at all.

Therefore this activity seeks to answer the question: Can it be done?
Answer: Yes and here's how

*Note this is only a test, do not put this in production - Also my document writing skills could probably need some tuning.

####################################################################################################################################################################
1.[INITIAL SETUP OF MY DEVICE]

This step only perains to my own device and preference, but seems to be a common problem on these devices, hence leaving this step in my run-down.
First I install nano as I find it tends to require less keystrokes when editing files and thus saves time and effort.

pkg install nano


My LIVA Box has a built in eMMC slot which causes timeout messages as it is probed at boot, so I'd rather disable it (since it is not used):
**Note standard /boot/loader.conf will be overwritten/reset by opnsense scripts so must rather create a .local version to load any boot options.

nano /boot/loader.conf.local

#Add this boot option/command to the file:
____________________________________________________________________________________________________________________________________________________________________

# Disabling Bad Card Reader Firmware load at boot:
hw.sdhci.enable_msi=0

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

Disable the Messages from these PCI Slot numbers:

nano /boot/device.hints
____________________________________________________________________________________________________________________________________________________________________

# Disabling Bad Card Reader Messages at boot:
hint.sdhci_pci.0.disabled="1"
hint.sdhci_pci.1.disabled="1"

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════




####################################################################################################################################################################
2. [SETTING UP BHYVE HYPERVISOR]

Bhyve is installed by default on the opnsense build which is fine for FreeBSD VMs, but in this scenario I require Linux and therefore need grub2-bhyve
so that Grub options can be set and used for the VM to load the linux kernel and config options.

We need to get grub2-bhyve which is required to load LINUX on Freebsd systems. This package is not available in the opnsense repo, but is available in the
FreeBSD repo. Luckily the FreeBSD repo is available on opnsense, but in a disabled state, here I will make use of this to get grub2-bhyve.

nano /usr/local/etc/pkg/repos/FreeBSD.conf
Change no to yes
____________________________________________________________________________________________________________________________________________________________________

FreeBSD: { enabled: yes }

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

To prevent FREEBSD Repo from updating opnsense default packages (like pkg itself) we must lock what we dont want updated, in this case pkg:
pkg lock pkg

Next we install grub2-bhyve from the FreeBSD repo:
pkg install grub2-bhyve

Now revert our changes Lock the grub2-bhyve package and disable the FreeBSD repo:
pkg unlock pkg
pkg lock grub2-bhyve

nano /usr/local/etc/pkg/repos/FreeBSD.conf
Change no to yes
____________________________________________________________________________________________________________________________________________________________________

FreeBSD: { enabled: no }

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════


3.[SYSTEM INFO SECTION]


The following section is informational and not required as steps/commands to complete this task, you can skip to the next section

## The grub2 linux command is the following (not in environmental path):      /usr/local/sbin/grub-bhyve

To show loaded Kernel Modules:
kldstat

To manually load VMM (Virtual Machine module) / NMDM (NullModem on-demand interface manager) / IF_BRIDGE (Bridge interfaces for freebsd) / IF_TAP (Tap interfaces)
kldload vmm
kldload nmdm
kldload if_bridge
kldload if_tap



####################################################################################################################################################################
4.[BHYVE PREPARATIONS]

Bridge and Tap are loaded by opnsense by default, so we want to add vmm and nmdm (to create a console connection to the VM)
As previously discussed, we can't use loader.conf as this is overwritten/reset at reboot so we create a new local loader where we can put our custom module loads.
In this case we are re-using the local and adding our extra options:

nano /boot/loader.conf.local

#At the top of the file add :
____________________________________________________________________________________________________________________________________________________________________

# ADD Bhyve required Modules and interfaces at boot
vmm_load="YES"
nmdm_load="YES"

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

In order to Enable TAP interfaces to come up when created we need systemcontol to allow this:
/sbin/sysctl net.link.tap.up_on_open=1
(this will enable TAP up during this current session, ie. not saved)

Now create A remote console terminal for VM :
/bin/echo 'vm1:dv=/dev/nmdb0B:br#9600:pa=none:' >> /etc/remote
(this command creates a tty serial session for our VM setup - which is not saved)

Now we need to go to OPNSense web interface and create a bridge:

[opnsense] [Interfaces] [Other Types]  [Bridge]   now click on {+ ADD]
Member interfaces -- Set LAN
Description - VMBRIDGE
  Click [SAVE]
 
[opnsense] [Interfaces] [Assignments]
Look for Bridge0 on the dropdown and click [+ADD]
  now click [SAVE]
 
[opnsense] [Interfaces]
Click on [OPT1]
Tick [✓] Enable interface
Description set to :  VMBRIDGE
  now click [SAVE]


Now lets add a TAP interface at boot, go back to the Console (or ssh session):

## Note found that opnsense does not autoload the interfaces via rc.conf here with nano /etc/defaults/rc.conf
## instead we will add it by creating our won startup script
# Using OPNSense manual for Bootup scripts, used the syshook directory to create a script that enables TAP and our terminal on boot:

nano /usr/local/etc/rc.syshook.d/start/50-tapstart
____________________________________________________________________________________________________________________________________________________________________

#!/bin/sh

/sbin/sysctl net.link.tap.up_on_open=1
/bin/echo 'vm1:dv=/dev/nmdb0B:br#9600:pa=none:' >> /etc/remote
/sbin/ifconfig tap0 create
/sbin/ifconfig bridge0 addm tap0
════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
(note added the TAP and TTY_Terminal at boot along with creating a TAP interface and binding it to the Bridge)

Now we must give the script permission to execute:
chmod +x /usr/local/etc/rc.syshook.d/start/50-tapstart

Now we can REBOOT!

Once it's back up we need to go to OPNSense web interface and commision the tap interface:

[opnsense] [Interfaces] [Assignments]
Look for tap0 on the dropdown and click [+ADD]
  now click [SAVE]
 
[opnsense] [Interfaces]
Click on [OPT2]
Tick [✓] Enable interface
Description set to :  VMGUEST_INT1
  now click [SAVE]


Now for Firewall Rules on Both VMBRIDGE and VMGUEST_INT1 Interfaces:
Add a Pass rule with from Source Subnet, i suggest first creating a [RealLAN] Network alias and using the alisa for the Source in your rule.
** Its up to you to make this secure, ie. lock it down as is required in your environment.



####################################################################################################################################################################
5.[BHYVE VIRTUAL MACHINE CREATION]

Now lets create the VM stuff using BHYVE.
First lets create a folder for our VM config and VM Boot ISOs.
In my case I wanted to run Debian10(Buster), so install wget package to pull this ISO image.

mkdir -p /vms/vm1
mkdir -p /vms/iso

cd /vms/iso
pkg install wget
wget -c https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.8.0-amd64-netinst.iso

cd /vms/vm1

Next I create a Disk Image file, in this case a 16Gb file for the installation of Debian:
truncate -s 16g /vms/vm1/debian1.img

Now I set a device map for the BHYVE VM to tell it the location of cd/ISO and DiskIMG:

nano /vms/vm1/device.map

____________________________________________________________________________________________________________________________________________________________________

(cd0) /vms/iso/debian-10.8.0-amd64-netinst.iso
(hd1) /vms/vm1/debain1.img

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

Next up we make sure there is no debian1 vm running followed by running the VM Boot to set GRUB options:
(*Note added it here at the top for quick reference as there is no graceful VM shutdown for BHYVE)

bhyvectl --destroy --vm=debian1

Next we use grub2-bhyve to initiate the grub boot sequence for the VM:

/usr/local/sbin/grub-bhyve -r cd0 -m /vms/vm1/device.map -M 2G debian1

The GRUB2-BHYVE window should show the Grub menu, we shoose 2nd option [Install] as this is the text mode installer which we need since we will use a serial terminal.

Now that Grub Kernel and options are set based on our grub2-bhyve command above we can launch the VM and associated interfaces:
bhyve -c 1 -m 2G -H -P -A -l com1,/dev/nmdm0A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3,ahci-cd,/vms/iso/debian-10.8.0-amd64-netinst.iso -s 4,virtio-blk,/vms/vm1/debian1.img debian1 &

OPEN a new SSH window so that we can use CU to open a serial terminal to our VM:
cu -l /dev/nmdm0B

From the Serial window we can progress through the Installation. Setting the Keyboard/Network Settings and note the creation of a user (we will use this instead of root later.
The Final stage we Select the listed /dev/vba device for Grub Boot Loader to be installed on the VMs virtual disk boot partion.
The window will eventually show completion and shutdown messages. DO NOT CLOSE THE SSH WINDOW WITH THE SERIAL TERMINAL - WE ARE GOING TO USE THIS AGAIN SHORTLY



Return to first ssh window where we started the configuration and destroy the VM that was just closed.
bhyvectl --destroy --vm=debian1

Here is the special sauce END OF INSTALL CHANGE GRUB ROOT TO DISK (instead of CD):

First we run the bhybe grub editor we set the loading kernel from DISK and location of the Grub menu (its not in a "standard" location):

/usr/local/sbin/grub-bhyve -r hd1,msdos1 -m /vms/vm1/device.map -M 2G -d /boot/grub debian1
(This tells Bhyve which disk parition(-r hd1,msdos1) from the device map to use as root disk and where to get boot info (-d /boot/grub on that disk))

Now we start the Virtual Machine and OMIT loading the ISO image (and use & to run this in the background)

bhyve -c 1 -m 2G -H -P -A -l com1,/dev/nmdm0A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 4,virtio-blk,/vms/vm1/debian1.img debian1 &




####################################################################################################################################################################
5.[BHYVE VM START AT BOOT]

Now lets start the VM at boot up by creating startup scripts. Scripts are executed in numerical order.

First up a script that kills our debain1 vm as it could be left in an unknown state.

nano /usr/local/etc/rc.syshook.d/start/70-debianvmkill
____________________________________________________________________________________________________________________________________________________________________

#!/bin/sh

/usr/sbin/bhyvectl --destroy --vm=debian1
/usr/sbin/bhyvectl --destroy --vm=debian1

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

Next we create a script that sets our grub boot options for the VM.

nano /usr/local/etc/rc.syshook.d/start/71-debianvmgrub
____________________________________________________________________________________________________________________________________________________________________

#!/bin/sh

/usr/local/sbin/grub-bhyve -r hd1,msdos1 -m /vms/vm1/device.map -M 2G -d /boot/grub debian1

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

nano /usr/local/etc/rc.syshook.d/start/72-debianvmstart

Here we wait 5 seconds for the grub boot selection script to finish then launch the VM
____________________________________________________________________________________________________________________________________________________________________

#!/bin/sh

/bin/sleep 5
/usr/sbin/bhyve -c 1 -m 2G -H -P -A -l com1,/dev/nmdm0A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 4,virtio-blk,/vms/vm1/debian1.img debian1

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

Now we must give it permission to execute:
chmod +x /usr/local/etc/rc.syshook.d/start/70-debianvmkill
chmod +x /usr/local/etc/rc.syshook.d/start/71-debianvmgrub
chmod +x /usr/local/etc/rc.syshook.d/start/72-debianvmstart

and thats it the VM should launch at startup.






####################################################################################################################################################################
7.[VM CONFIGURATION]

Now go back to the Serial Console SSH session and login as root so that we can allow SSH from root, this is temporary we will disable root SSH later:

nano /etc/ssh/sshd_config

Add the follwing
____________________________________________________________________________________________________________________________________________________________________

PermitRootLogin yes

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

Now restart SSH service:

/etc/init.d/ssh restart

We can now close the remote console ssh session.

Its time to setup the Debian VM in my case i setup with UNPRIVILIAGED LXC containers to run two instances of pihole to test feasibility and functionality.
SSH to Deiban VM as root (this time only)

run:
apt update
apt install sudo

Now I will add my user I added at installation time to the Sudo group
usermod -aG sudo slipperyduck

now run the visudo command to edit the sudoers permission and grant our user sudo access

visudo

#and at the end of the file add:
____________________________________________________________________________________________________________________________________________________________________

slipperyduck  ALL=(ALL) NOPASSWD:ALL

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

Now we will edit SSH server to disable Root over SSH (root becomes console only), since we will use the User created during the installation
nano /etc/ssh/sshd_config

Change the follwing (hash/comment-out the permitroot option)
____________________________________________________________________________________________________________________________________________________________________

#PermitRootLogin yes

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

Now restart SSH service:

/etc/init.d/ssh restart

We can now close the ssh session and open a new one to login as the user created during installation.

####################################################################################################################################################################
8.[CONCLUSION]

There we go we have a full linux VM running, I must say that running Unpriviliaged LXC containers is a must for security, but is a whole other installation document.
I have it fully operational in my LAB and if you want information on how to do that, I can add a post.

[itneau]

Hi there,

I hope you well.

I have a question conerning your post which is very informative. I have run through them a few time and get inconsistent results such as VMX errors 33 and unable to setup memory. I have not been able to boot opnwrt under bhyve as yet.

Do your instructions work with Opnsense 21.7.3-1 ?

Best

John

[banuseka]

I have it running with debian 10 on exact that OPNsense version and above instructions.

[banuseka]

I have it running with debian 10 on exact that OPNsense version and above instructions.

OK NOW I run into issues... I rebooted OPNsense and now my VM is not booting up anymore...

besides changing number of cores to use I didn't do anything else... any hints?

This is what my startupfiles look like:

/usr/local/etc/rc.syshook.d/start/70-debianvmkill

Code Select Expand


#!/bin/sh

/usr/sbin/bhyvectl --destroy --vm=debian1
/usr/sbin/bhyvectl --destroy --vm=debian1


/usr/local/etc/rc.syshook.d/start/71-debianvmgrub

Code Select Expand


#!/bin/sh

/usr/local/sbin/grub-bhyve -r hd1,msdos1 -m /vms/vm1/device.map -M 4G -d /boot/grub debian1


/usr/local/etc/rc.syshook.d/start/72-debianvmstart

Code Select Expand


#!/bin/sh

/bin/sleep 5
/usr/sbin/bhyve -c 2 -m 4G -H -P -A -l com1,/dev/nmdm0A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 4,virtio-blk,/vms/vm1/debian1.img debian1


As said earlier... it worked like a charme but after a reboot the VM doesn't start anymore...

Best, banuseka

[loredo]

You might miss the tap device setup from

Code Select Expand

/usr/local/etc/rc.syshook.d/start/50-tapstart

[banuseka]

tap is also running, although 50 is also used by wireguard, might that be a problem?


btw. my 50-tapstart:

Code Select Expand


#!/bin/sh

/sbin/sysctl net.link.tap.up_on_open=1
/bin/echo 'vm1:dv=/dev/nmdb0B:br#9600:pa=none:' >> /etc/remote
/sbin/ifconfig tap0 create
/sbin/ifconfig bridge0 addm tap0


this is what my /usr/local/etc/rc.syshook.d/start/ looks:

Code Select Expand


-rwxr-xr-x  1 root  wheel   647 Sep 30 15:28 10-newwanip
-rwxr-xr-x  1 root  wheel    78 Sep 30 15:28 20-freebsd
-rwxr-xr-x  1 root  wheel   225 Sep 30 15:28 21-syslog-ng
-rwxr-xr-x  1 root  wheel   173 Aug 27 13:52 50-tapstart
-rwxr-xr-x  1 root  wheel    83 Jul 23 09:43 50-wireguard
-rwxr-xr-x  1 root  wheel    95 Aug 27 15:16 70-debianvmkill
-rwxr-xr-x  1 root  wheel   103 Oct  9 14:02 71-debianvmgrub
-rwxr-xr-x  1 root  wheel   177 Oct  9 14:35 72-debianvmstart
-rwxr-xr-x  1 root  wheel  1541 Sep 30 15:28 90-carp
-rwxr-xr-x  1 root  wheel    60 Sep 30 15:28 90-cron
-rwxr-xr-x  1 root  wheel    31 Sep 30 15:28 95-beep


[banuseka]

OK, VM is running, however eversytime OPNsense reboots, the VM gets a new mac address and thus a different IP.

might

Code Select Expand

macchanger be a problem solver to alway fake the mac to a specific one?

Best, banu

[banuseka]

FYI:

After upgrading to 22.x, step 2 has to be repeated to get bhyve running again.

[banuseka]

I now run into trouble again... the VM cann acess the internet and I can acces the VM from any machine in the LAN . BUT the VM cannot acces anny machine on the LAN...