HOWTO: Setup OpenWRT Virtual Machine on OPNsense and use it to manage a WiFi AP

[zan]

A while ago I upgraded the WiFi device of my laptop and ended up with a spare Intel AX201 M.2 card.
I've been running OPNsense at home on a Intel 5105 NUC baremetal and it has a free M.2 slot.
Inspired by @pinako's post (https://forum.opnsense.org/index.php?topic=32813.0) and knowing abysmal WiFi support on FreeBSD, I was set to repurpose the card as a WiFi AP on my OPNsense, by using OpenWRT VM installed on FreeBSD's own hypervisor(bhyve) to manage.
This is a guide how to do it.

Prerequisite:

• OPNsense is running baremetal on a machine capable of IO virtualization and PCI passthrough (IOMMU aka VT-d on Intel, AMD-Vi on AMD).
• FreeBSD repo is enabled

1. Install vm-bhyve (https://github.com/churchers/vm-bhyve) to manage bhyve:

Code: [Select]

pkg install vm-bhyve grub2-bhyve bhyve-firmware
mkdir /home/vm   # or 'zfs create pool/vm' if you are using zfs
sysrc vm_enable="YES"
sysrc vm_dir="/home/vm"   # or "zfs:pool/vm"
vm init
2. In OPNsense web-GUI:

• Create a bridge interface in OPNsense GUI, assign an interface(eg: LAN1) as a member.
• Assign the newly created bridge iface (eg: bridge0) to a new iface (eg: VM_BRIDGE).
• We should configure IP address on the bridge iface, NOT on member iface. Assign an IP address (eg: 192.168.99.1) to VM_BRIDGE and unassign everything from LAN1.
• Enable DHCPDv4 on VM_BRIDGE, make sure gateway option in DHCPD is set to VM_BRIDGE's address (192.168.99.1).
• Create firewall rule(s) in VM_BRIDGE to pass traffic.
• Add the following tunables (System>Settings>Tunables) to enable filtering on bridge iface and disable filtering on member ifaces:

Code: [Select]

net.link.bridge.pfil_bridge -> 1
net.link.bridge.pfil_member -> 0

3. Create a vm-bhyve switch from our bridge and name it 'public':

Code: [Select]

vm switch create -t manual -b bridge0 public

4. Determine the PCI ID of WiFi device we want to pass through:

Code: [Select]

[root@router ~]# vm passthru
DEVICE     BHYVE ID     READY        DESCRIPTION
hostb0     0/0/0        No           -
vgapci0    0/2/0        No           JasperLake [UHD Graphics]
none0      0/4/0        No           Dynamic Tuning service
xhci0      0/20/0       No           -
iwlwifi0   0/20/3       No           Wi-Fi 6 AX201 160MHz    <--- This!
sdhci_pci0 0/20/5       No           -As we can see the device ID is 0/20/3 and being managed by iwlwifi driver.
We need to inform the kernel to exempt the device from loading its driver before we can pass through to OpenWRT, by adding these tunables via web-GUI again:

Code: [Select]

vmm_load -> YES
pptdevs -> 0/20/3Reboot and verify:

Code: [Select]

[root@router ~]# vm passthru
DEVICE     BHYVE ID     READY        DESCRIPTION
hostb0     0/0/0        No           -
vgapci0    0/2/0        No           JasperLake [UHD Graphics]
none0      0/4/0        No           Dynamic Tuning service
xhci0      0/20/0       No           -
ppt0       0/20/3       Yes          Wi-Fi 6 AX201 160MHz 
sdhci_pci0 0/20/5       No           -Good now the device is assigned to ppt driver and ready for use.


4. Create OpenWRT VM and download the latest OpenWrt image (https://downloads.openwrt.org/releases/22.03.5/targets/x86/64/). We are using one the EFI images:

Code: [Select]

vm create -s0 openwrt    # zero size image
cd /home/vm/openwrt      # or /pool/vm/openwrt
rm disk0.img             # we won't be using the default created img
fetch https://downloads.openwrt.org/releases/22.03.5/targets/x86/64/openwrt-22.03.5-x86-64-generic-ext4-combined-efi.img.gz
gunzip openwrt-22.03.5-x86-64-generic-ext4-combined-efi.img.gz

5. Edit and modify openwrt.conf:

Code: [Select]

loader="uefi"
cpu=2
memory=512M   # 256MB should be enough but just in case
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="openwrt-22.03.5-x86-64-generic-ext4-combined-efi.img"
passthru0="0/20/3=7:0"    # We map the device to slot 7 on OpenWrt*Do not change 'uuid' & 'network0_mac' values


6. Start the VM and attach to its console:

Code: [Select]

vm start -f openwrt

7. If all is well we are now in OpenWRT shell as root. We can then perform OpenWRT initial setup:

Code: [Select]

passwd # set root password
/etc/init.d/odhcpd disable # Disable OpenWRT DHCP & DNS servers
/etc/init.d/dnsmasq disable
uci set network.lan.ipaddr='192.168.99.2/24'
uci set network.lan.gateway='192.168.99.1'
uci set network.lan.dns='192.168.99.1'
uci commit
/etc/init.d/network restart
opkg update
opkg install luci
Find the firmware for our device at https://openwrt.org/packages/index/firmware, eg: iwlwifi-firmware-ax210 for Intel AX210

Code: [Select]

opkg install kmod-iwlwifi iwlwifi-firmware-ax210 hostapd-openssl
reboot

8. After rebooting, OpenWRT should be reachable by 192.168.99.2 (Web-GUI and SSH). All that's left to do is to configure a non-routing AP(Dumb AP), continue by following the OpenWRT guide (https://openwrt.org/docs/guide-user/network/wifi/dumbap).
Once the AP is up and running any client connected to it will be getting IP address from OPNsense DHCPD.

[catchyuser]

Thank you Zan for sharing the guide for Wifi AP installation. I have ordered the Wifi card today and will try it out once I receive it.

[zan]

If you are getting a new card I advise not to get Intel cards because Intel does not allow AP mode on 5GHz.

Get something like Mediatek MT7912K, my colleague uses it on his OpenWRT and able to get 1.2Gb rate on WiFi-6 mode.

[catchyuser]

oops, I already ordered the intel Ax210 card, will retun it and order the Mediatek MT7912K model.
Thank you for the heads up!

[TheHellSite]

Very cool guide and thank you for sharing it here!   


Out of interest...
Why not just install a hypervisor, like Proxmox, and create two VMs (one for OPNsense and one for OpenWrt)?
This way you wouldn't have to "mess" with the OPNsense install that much.

[zan]

Hi @TheHellSite,

I have a server at home running Proxmox for my NAS and various VMs and I've been running pfSense/OPNsense for years before.
Now I just prefer my firewall not depending on anything else to run, that's why I bought this small appliance.
To me, having bhyve virtualizes OpenWRT is not 'messing' OPNsense much, it fits my use case.
I treat it as running a jail capable of PCI passthrough.
The OpenWRT essentially becomes a part of OPNsense and the current setup is lightweight enough the added CPU usage is barely noticeable.

[catchyuser]

Hello Zan,

I am trying to install the Wifi adapter on Opnsense, but I am stuck at the step after

vm_load -> YES
pptdevs -> 2/0/0

I have updated Tunables settings as above, reboot/restarted the router but when i check vm passthru, it doesn't show pptdevs, it is still showing device iwlwifi0. Please refer to the attached screenshot.

Can you please help and advise how to resolve this issue?

Thanks in advance.

[zan]

You did a reboot after adding the tunables right?
When using OPNsense GUI to set tunables it will auto-add double quotes so make sure you don't enclose double quotes to the values, check "/boot/loader.conf" to verify.

Also make sure you can load the ppt driver for that device on-demand:

Code: [Select]

devctl set driver -f pci0:2:0:0 ppt

[catchyuser]

It worked after I entered this command --> devctl set driver -f pci0:2:0:0 ppt

Now I can see the device ppt0 --> 2/0/0 Yes

Thank you so much for the quick help.

Sorry one more question, I started VM openwrt ..it says booting and I get the prompt again ..How can I connect to console for the next steps.

I did reboot several times but the pptdevs value did not show up.

Please find the loader.conf file below, there are entries for pptdev and vm_load

 ##############################################################
# This file was auto-generated using the rc.loader facility. #
# In order to deploy a custom change to this installation,   #
# please use /boot/loader.conf.local as it is not rewritten, #
# or better yet use System: Settings: Tunables from the GUI. #
##############################################################

loader_brand="opnsense"
loader_logo="hourglass"
loader_menu_title=""

autoboot_delay="3"

# Vital modules that are not in FreeBSD's GENERIC
# configuration will be loaded on boot, which makes
# races with individual module's settings impossible.
carp_load="YES"
if_bridge_load="YES"
if_enc_load="YES"
if_gif_load="YES"
if_gre_load="YES"
if_lagg_load="YES"
if_tap_load="YES"
if_tun_load="YES"
if_vlan_load="YES"
pf_load="YES"
pflog_load="YES"
pfsync_load="YES"

# ZFS standard environment requirements
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
vfs.zfs.min_auto_ashift=12
opensolaris_load="YES"
zfs_load="YES"

# dynamically generated console settings follow
#comconsole_speed
#boot_multicons
#boot_serial
#kern.vty
console="vidconsole"

# dynamically generated tunables settings follow
hw.ibrs_disable="0"
hw.ixl.enable_head_writeback="0"
hw.syscons.kbd_reboot="0"
kern.ipc.maxsockbuf="4262144"
kern.randompid="1"
net.enc.in.ipsec_bpf_mask="2"
net.enc.in.ipsec_filter_mask="2"
net.enc.out.ipsec_bpf_mask="1"
net.enc.out.ipsec_filter_mask="1"
net.inet.icmp.drop_redirect="1"
net.inet.icmp.icmplim="0"
net.inet.icmp.log_redirect="0"
net.inet.icmp.reply_from_interface="1"
net.inet.ip.accept_sourceroute="0"
net.inet.ip.forwarding="1"
net.inet.ip.intr_queue_maxlen="1000"
net.inet.ip.portrange.first="1024"
net.inet.ip.random_id="1"
net.inet.ip.redirect="0"
net.inet.ip.sourceroute="0"
net.inet.tcp.blackhole="2"
net.inet.tcp.delayed_ack="0"
net.inet.tcp.drop_synfin="1"
net.inet.tcp.log_debug="0"
net.inet.tcp.recvspace="65228"
net.inet.tcp.sendspace="65228"
net.inet.tcp.syncookies="1"
net.inet.tcp.tso="1"
net.inet.udp.blackhole="1"
net.inet.udp.checksum="1"
net.inet.udp.maxdgram="57344"
net.inet6.ip6.forwarding="1"
net.inet6.ip6.intr_queue_maxlen="1000"
net.inet6.ip6.prefer_tempaddr="0"
net.inet6.ip6.redirect="0"
net.inet6.ip6.use_tempaddr="0"
net.link.bridge.pfil_bridge="1"
net.link.bridge.pfil_local_phys="0"
net.link.bridge.pfil_member="0"
net.link.bridge.pfil_onlyip="0"
net.link.ether.inet.log_arp_movements="1"
net.link.ether.inet.log_arp_wrong_iface="1"
net.link.tap.user_open="1"
net.link.vlan.mtag_pcp="1"
net.local.dgram.maxdgram="8192"
net.pf.share_forward="1"
net.pf.share_forward6="1"
net.route.multipath="0"
pptdevs="2/0/0"
security.bsd.see_other_gids="0"
security.bsd.see_other_uids="0"
vfs.read_max="32"
vm.pmap.pti="1"
vm_load="YES"

[catchyuser]

Hi Zan,

I am looking at the log and see that there is some error "bhyve exited with status 4"

Any clues what is causing this error and how I can fix it?

Thanks in advance for the help!

[zan]

I just realized I made a grave typo error, the setting should be "vmm_load="YES".
I edited the original post, really sorry for this.

Please try again.

it says booting and I get the prompt again ..How can I connect to console for the next steps.

If you start the the VM with -f flag it will auto-attach to VM's console. If you get kicked out to the OPNsense prompt again most likely the VM failed to start. You can check the vm-bhyve.log in VM's directory to find out why.

To connect to a running VM:

Code: [Select]

vm list
vm console <vm-name>
Do check vm-bhyve github page for list of commands. The Wiki page there is very informative.

[catchyuser]

Thanks Zan for all the help!

It worked after I changed the Tunable parameter to vmm_load = "YES"

The openwrt installation is done successfully and I can access the console.

Now I am getting error installing the firmware --> opkg install kmod-iwlwifi iwlwifi-firwmware-ax210

I did a reboot and see the following errors for iwlwifi

[    5.607673] Intel(R) Wireless WiFi driver for Linux
[    5.610248] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-66.ucode failed with error -2
[    5.611845] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-66.ucode
[    5.618274] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-65.ucode failed with error -2
[    5.619874] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-65.ucode
[    5.624009] PPP generic driver version 2.4.2
[    5.624823] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-64.ucode failed with error -2
[    5.626419] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-64.ucode
[    5.628096] NET: Registered protocol family 24
[    5.630984] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-63.ucode failed with error -2
[    5.632596] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-63.ucode
[    5.634966] kmodloader: done loading kernel modules from /etc/modules.d/*
[    5.636682] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-62.ucode failed with error -2
[    5.638276] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-62.ucode
[    5.641459] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-61.ucode failed with error -2
[    5.643097] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-61.ucode
[    5.646426] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-60.ucode failed with error -2
[    5.648065] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-60.ucode
[    5.651354] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-59.ucode failed with error -2
[    5.652955] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-59.ucode
[    5.656201] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-58.ucode failed with error -2
[    5.657834] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-58.ucode
[    5.661201] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-57.ucode failed with error -2
[    5.662812] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-57.ucode
[    5.666067] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-56.ucode failed with error -2
[    5.667711] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-56.ucode
[    5.670898] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-55.ucode failed with error -2
[    5.672519] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-55.ucode
[    5.675738] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-54.ucode failed with error -2
[    5.677377] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-54.ucode
[    5.680786] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-53.ucode failed with error -2
[    5.682369] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-53.ucode
[    5.685620] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-52.ucode failed with error -2
[    5.687226] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-52.ucode
[    5.690483] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-51.ucode failed with error -2
[    5.692117] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-51.ucode
[    5.695367] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-50.ucode failed with error -2
[    5.697003] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-50.ucode
[    5.700186] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-49.ucode failed with error -2
[    5.701799] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-49.ucode
[    5.705003] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-48.ucode failed with error -2
[    5.706604] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-48.ucode
[    5.709787] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-47.ucode failed with error -2
[    5.711386] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-47.ucode
[    5.714628] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-46.ucode failed with error -2
[    5.716227] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-46.ucode
[    5.719439] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-45.ucode failed with error -2
[    5.721045] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-45.ucode
[    5.724306] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-44.ucode failed with error -2
[    5.725959] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-44.ucode
[    5.729171] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-43.ucode failed with error -2
[    5.730783] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-43.ucode
[    5.734043] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-42.ucode failed with error -2
[    5.735648] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-42.ucode
[    5.738848] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-41.ucode failed with error -2
[    5.740456] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-41.ucode
[    5.743685] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-40.ucode failed with error -2
[    5.745287] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-40.ucode
[    5.748554] iwlwifi 0000:00:08.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-39.ucode failed with error -2
[    5.750165] iwlwifi 0000:00:08.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-39.ucode
[    5.753427] iwlwifi 0000:00:08.0: no suitable firmware found!
[    5.754377] iwlwifi 0000:00:08.0: minimum version required: iwlwifi-ty-a0-gf-a0-39
[    5.755603] iwlwifi 0000:00:08.0: maximum version supported: iwlwifi-ty-a0-gf-a0-66
[    5.756866] iwlwifi 0000:00:08.0: check git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
[    7.041348] br-lan: port 1(eth0) entered blocking state
[    7.042234] br-lan: port 1(eth0) entered disabled state
[    7.043131] device eth0 entered promiscuous mode
[    7.044258] br-lan: port 1(eth0) entered blocking state
[    7.045137] br-lan: port 1(eth0) entered forwarding state
[    8.126737] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready

[zan]

Looks like the correct firmware of your card is not in the iwlwifi-firwmware-ax210 package.

[    5.754377] iwlwifi 0000:00:08.0: minimum version required: iwlwifi-ty-a0-gf-a0-39
[    5.755603] iwlwifi 0000:00:08.0: maximum version supported: iwlwifi-ty-a0-gf-a0-66

It's looking for iwlwifi-ty-a0-gf-a0 firmware compatible to your current OpenWrt kernel.
You can get it straight from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/

Look for iwlwifi-ty-a0-gf-a0-66.ucode and save it to /lib/firmware directory.

Code: [Select]

cd /lib/firmware
wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/iwlwifi-ty-a0-gf-a0-66.ucode

[catchyuser]

Thank you so much Zan for the help!

Happy Friday!

I downloaded the firmware iwlwifi-ty-a0-gf-a0-66.ucode and rebooted the router.

I am still getting the same error message (screenshot attached)

Can you please help again..

Once again thank you for help in advance.

[zan]

Sorry I pasted a wrong link. Try this one https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/iwlwifi-ty-a0-gf-a0-66.ucode

Once again thank you for help in advance.

Always happy to help. You are very welcome. Hope you get it up and running asap!