[HOWTO] OpnSense under virtualisation (Proxmox et.al.)

[alestark]

Did you enable multiqueue on the VM NIC interfaces in Proxmox? The throughput you are getting suggests, you did not.

If you refer to this settings, I have: https://imgur.com/a/K3upFP1

[meyergru]

I always use 4 cores and 4 queues. iperf needs a -P4 as well, a single thread will max out at ~600 Mbps for these CPUs.

[alestark]

I am kinda limited by the host cpu. Being a N5105 i only have 4 cores available, hence, i've given firewall vm 2 cores.
However, I did bump the queues to 4, but it's actually the same. Cpu spikes near 100% and same speed...

OPNSense:

Code Select Expand

-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.2.204, port 54384
[  5] local 192.168.2.1 port 5201 connected to 192.168.2.204 port 54390
[  8] local 192.168.2.1 port 5201 connected to 192.168.2.204 port 54398
[ 10] local 192.168.2.1 port 5201 connected to 192.168.2.204 port 54412
[ 12] local 192.168.2.1 port 5201 connected to 192.168.2.204 port 54428
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  4.12 MBytes  34.5 Mbits/sec                 
[  8]   0.00-1.00   sec  3.88 MBytes  32.4 Mbits/sec                 
[ 10]   0.00-1.00   sec  36.0 MBytes   301 Mbits/sec                 
[ 12]   0.00-1.00   sec  3.75 MBytes  31.4 Mbits/sec                 
[SUM]   0.00-1.00   sec  47.8 MBytes   400 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.01   sec  10.8 MBytes  89.2 Mbits/sec                 
[  8]   1.00-2.01   sec  9.88 MBytes  81.9 Mbits/sec                 
[ 10]   1.00-2.01   sec  3.25 MBytes  27.0 Mbits/sec                 
[ 12]   1.00-2.01   sec  10.1 MBytes  84.0 Mbits/sec                 
[SUM]   1.00-2.01   sec  34.0 MBytes   282 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.01-3.01   sec  13.5 MBytes   113 Mbits/sec                 
[  8]   2.01-3.01   sec  3.38 MBytes  28.3 Mbits/sec                 
[ 10]   2.01-3.01   sec  25.9 MBytes   217 Mbits/sec                 
[ 12]   2.01-3.01   sec  2.75 MBytes  23.1 Mbits/sec                 
[SUM]   2.01-3.01   sec  45.5 MBytes   382 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.01-4.01   sec  15.8 MBytes   132 Mbits/sec                 
[  8]   3.01-4.01   sec  14.2 MBytes   120 Mbits/sec                 
[ 10]   3.01-4.01   sec  24.1 MBytes   202 Mbits/sec                 
[ 12]   3.01-4.01   sec  4.00 MBytes  33.6 Mbits/sec                 
[SUM]   3.01-4.01   sec  58.1 MBytes   488 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.01-5.01   sec  10.6 MBytes  89.1 Mbits/sec                 
[  8]   4.01-5.01   sec  22.2 MBytes   187 Mbits/sec                 
[ 10]   4.01-5.01   sec   896 KBytes  7.34 Mbits/sec                 
[ 12]   4.01-5.01   sec  3.25 MBytes  27.3 Mbits/sec                 
[SUM]   4.01-5.01   sec  37.0 MBytes   310 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.01-6.20   sec  1.75 MBytes  12.3 Mbits/sec                 
[  8]   5.01-6.26   sec  31.4 MBytes   211 Mbits/sec                 
[ 10]   5.01-6.26   sec   384 KBytes  2.53 Mbits/sec                 
[ 12]   5.01-6.26   sec  9.38 MBytes  63.2 Mbits/sec                 
[SUM]   5.01-6.20   sec  42.9 MBytes   302 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.20-7.01   sec  6.00 MBytes  62.6 Mbits/sec                 
[  8]   6.26-7.01   sec  21.4 MBytes   239 Mbits/sec                 
[ 10]   6.26-7.01   sec  0.00 Bytes  0.00 bits/sec                 
[ 12]   6.26-7.01   sec  2.50 MBytes  28.0 Mbits/sec                 
[SUM]   6.20-7.01   sec  29.9 MBytes   312 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.01-8.05   sec  12.6 MBytes   101 Mbits/sec                 
[  8]   7.01-8.05   sec  10.2 MBytes  82.1 Mbits/sec                 
[ 10]   7.01-8.05   sec  25.5 MBytes   204 Mbits/sec                 
[ 12]   7.01-8.06   sec  9.00 MBytes  72.0 Mbits/sec                 
[SUM]   7.01-8.05   sec  57.4 MBytes   460 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.05-9.01   sec  13.0 MBytes   114 Mbits/sec                 
[  8]   8.05-9.01   sec  6.38 MBytes  55.8 Mbits/sec                 
[ 10]   8.05-9.01   sec  1.00 MBytes  8.75 Mbits/sec                 
[ 12]   8.06-9.01   sec  12.2 MBytes   107 Mbits/sec                 
[SUM]   8.05-9.01   sec  32.6 MBytes   286 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.01-10.01  sec  21.1 MBytes   177 Mbits/sec                 
[  8]   9.01-10.01  sec  0.00 Bytes  0.00 bits/sec                 
[ 10]   9.01-10.01  sec  0.00 Bytes  0.00 bits/sec                 
[ 12]   9.01-10.01  sec  16.5 MBytes   138 Mbits/sec                 
[SUM]   9.01-10.01  sec  37.6 MBytes   316 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.01-10.02  sec   128 KBytes   136 Mbits/sec                 
[  8]  10.01-10.02  sec  0.00 Bytes  0.00 bits/sec                 
[ 10]  10.01-10.02  sec  0.00 Bytes  0.00 bits/sec                 
[ 12]  10.01-10.02  sec   128 KBytes   134 Mbits/sec                 
[SUM]  10.01-10.02  sec   256 KBytes   272 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.02  sec   109 MBytes  91.6 Mbits/sec                  receiver
[  8]   0.00-10.02  sec   123 MBytes   103 Mbits/sec                  receiver
[ 10]   0.00-10.02  sec   117 MBytes  97.9 Mbits/sec                  receiver
[ 12]   0.00-10.02  sec  73.6 MBytes  61.6 Mbits/sec                  receiver
[SUM]   0.00-10.02  sec   423 MBytes   354 Mbits/sec                  receiver

host proxmox:

Code Select Expand

[root@pve-02]: ~ $ iperf3 -c 192.168.2.1 -P4
Connecting to host 192.168.2.1, port 5201
[  5] local 192.168.2.204 port 54390 connected to 192.168.2.1 port 5201
[  7] local 192.168.2.204 port 54398 connected to 192.168.2.1 port 5201
[  9] local 192.168.2.204 port 54412 connected to 192.168.2.1 port 5201
[ 11] local 192.168.2.204 port 54428 connected to 192.168.2.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.23 MBytes  43.8 Mbits/sec    1   1.41 KBytes       
[  7]   0.00-1.00   sec  5.23 MBytes  43.8 Mbits/sec    1   1.41 KBytes       
[  9]   0.00-1.00   sec  38.9 MBytes   326 Mbits/sec  197    782 KBytes       
[ 11]   0.00-1.00   sec  4.95 MBytes  41.4 Mbits/sec    1   1.41 KBytes       
[SUM]   0.00-1.00   sec  54.3 MBytes   455 Mbits/sec  200             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  12.2 MBytes   103 Mbits/sec    1    400 KBytes       
[  7]   1.00-2.00   sec  10.9 MBytes  91.9 Mbits/sec    7    376 KBytes       
[  9]   1.00-2.00   sec  3.75 MBytes  31.5 Mbits/sec    1   1.41 KBytes       
[ 11]   1.00-2.00   sec  11.4 MBytes  95.5 Mbits/sec   17    373 KBytes       
[SUM]   1.00-2.00   sec  38.3 MBytes   322 Mbits/sec   26             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  13.3 MBytes   111 Mbits/sec   11    529 KBytes       
[  7]   2.00-3.00   sec  4.23 MBytes  35.4 Mbits/sec   11    379 KBytes       
[  9]   2.00-3.00   sec  27.5 MBytes   231 Mbits/sec  157    803 KBytes       
[ 11]   2.00-3.00   sec  3.11 MBytes  26.1 Mbits/sec   10    366 KBytes       
[SUM]   2.00-3.00   sec  48.1 MBytes   404 Mbits/sec  189             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  16.2 MBytes   136 Mbits/sec    5    293 KBytes       
[  7]   3.00-4.00   sec  13.4 MBytes   113 Mbits/sec    1    427 KBytes       
[  9]   3.00-4.00   sec  22.5 MBytes   189 Mbits/sec  232    416 KBytes       
[ 11]   3.00-4.00   sec  4.35 MBytes  36.5 Mbits/sec    7    192 KBytes       
[SUM]   3.00-4.00   sec  56.5 MBytes   474 Mbits/sec  245             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  12.5 MBytes   105 Mbits/sec    0    324 KBytes       
[  7]   4.00-5.00   sec  22.4 MBytes   188 Mbits/sec    0    478 KBytes       
[  9]   4.00-5.00   sec  2.50 MBytes  21.0 Mbits/sec    0    419 KBytes       
[ 11]   4.00-5.00   sec  3.17 MBytes  26.6 Mbits/sec    1    204 KBytes       
[SUM]   4.00-5.00   sec  40.5 MBytes   340 Mbits/sec    1             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  2.50 MBytes  21.0 Mbits/sec    0    328 KBytes       
[  7]   5.00-6.00   sec  25.5 MBytes   214 Mbits/sec    0    510 KBytes       
[  9]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec    0    416 KBytes       
[ 11]   5.00-6.00   sec  6.40 MBytes  53.7 Mbits/sec    0    225 KBytes       
[SUM]   5.00-6.00   sec  35.6 MBytes   299 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  3.75 MBytes  31.5 Mbits/sec    0    338 KBytes       
[  7]   6.00-7.00   sec  29.0 MBytes   243 Mbits/sec    0    530 KBytes       
[  9]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0    416 KBytes       
[ 11]   6.00-7.00   sec  5.34 MBytes  44.8 Mbits/sec    0    242 KBytes       
[SUM]   6.00-7.00   sec  38.1 MBytes   319 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  12.5 MBytes   105 Mbits/sec    1    365 KBytes       
[  7]   7.00-8.00   sec  10.1 MBytes  85.0 Mbits/sec    0    537 KBytes       
[  9]   7.00-8.00   sec  22.5 MBytes   189 Mbits/sec    0    460 KBytes       
[ 11]   7.00-8.00   sec  8.76 MBytes  73.5 Mbits/sec    0    267 KBytes       
[SUM]   7.00-8.00   sec  53.9 MBytes   452 Mbits/sec    1             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  12.5 MBytes   105 Mbits/sec    0    390 KBytes       
[  7]   8.00-9.00   sec  6.71 MBytes  56.3 Mbits/sec    0    547 KBytes       
[  9]   8.00-9.00   sec  3.75 MBytes  31.5 Mbits/sec    0    464 KBytes       
[ 11]   8.00-9.00   sec  11.7 MBytes  98.0 Mbits/sec    0    301 KBytes       
[SUM]   8.00-9.00   sec  34.6 MBytes   291 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  21.2 MBytes   178 Mbits/sec    0    431 KBytes       
[  7]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0    547 KBytes       
[  9]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0    464 KBytes       
[ 11]   9.00-10.00  sec  16.9 MBytes   142 Mbits/sec    0    341 KBytes       
[SUM]   9.00-10.00  sec  38.2 MBytes   320 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   112 MBytes  94.0 Mbits/sec   19             sender
[  5]   0.00-10.02  sec   109 MBytes  91.6 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   127 MBytes   107 Mbits/sec   20             sender
[  7]   0.00-10.02  sec   123 MBytes   103 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   123 MBytes   103 Mbits/sec  587             sender
[  9]   0.00-10.02  sec   117 MBytes  97.9 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec  76.0 MBytes  63.8 Mbits/sec   36             sender
[ 11]   0.00-10.02  sec  73.6 MBytes  61.6 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec   438 MBytes   368 Mbits/sec  662             sender
[SUM]   0.00-10.02  sec   423 MBytes   354 Mbits/sec                  receiver

iperf Done.


[mmosa]

I have a Proxmox server with a single NIC that's connected to a MikroTik router.

In Proxmox, the default bridge is vmbr0.
On the MikroTik side, I created a VLAN (e.g., VLAN 100) and set it as a DHCP server.

On the Proxmox host, I added an interface vmbr0.100 (for VLAN 100), and it gets an IP automatically via DHCP from the MikroTik VLAN.

Also, the Proxmox host has a Cloudflare Tunnel set up, which gives remote access to all services running on the VMs, including the Proxmox web UI itself.

Now, I also have an OPNsense instance running.


What I want to do is:

Route all VM and LXC traffic in Proxmox through VLANs provided by OPNsense.

And I still want to access everything via the Cloudflare Tunnel, routed through the Proxmox host.

Is this kind of setup possible? Any best practices or recommendations?

[meyergru]

That is a very specific setup that should be put into a thread on its own.

[spetrillo]

I am not sure of my problem...but here is what I have. I hope you can point me in the right direction!

I have a Lenovo M720q PC with a 4 port Intel I350 network adapter. I am going to use the onboard NIC for other VMs, as we as Proxmox mgmt. VLANs are configured on 3 of the 4 I350 ports, with the 4th port going to the Internet. The Proxmox config to support this is in attachment 1. The OPNsense config in Proxmox is in attachment 2.

I have a connection from my PC directly to port 1 of the I350. I have setup the VLAN on my PC connection to VLAN 1, which matches the OPNsense config for port 1. How am supposed to get to the GUI, so I can continue my config efforts? I am completely lost here.

[meyergru]

First off, VLAN 1 is mostly "special" - many switches consider this to be the untagged LAN. I would rather not use it.

Then, you can imagine any bridge device on Proxmox as a switch where you can plug in network ports - these can either be physical NICs or VM vtnet ports.

So, the usual setup with a LAN and a WAN would be vmbr0 connected to the physical LAN interface (in your case vmbr1) and vmbr1 connected to the physical WAN interface (yours is vmbr4) with Proxmox having an IP and a subnet on LAN. You can configure the latter on vmbr0 or the physical interface, both should work.

Your OpnSense VM and would then have vtnet0 on vmbr0 as LAN with a configuration you specify on OpnSense only and vtnet1 on vmbr1 likewise.

The LAN bridge vmbr0 (in your case vmbr1) would then be connected to a pyhsical LAN NIC connected to a switch, internally given a LAN IP for Proxmox and be connected to the vtnet0 interface, which is LAN on OpnSense.

[zuppaduppa]

Hi everyone,

I'm hitting a wall with a VLAN issue where tagged traffic seems to be processed incorrectly by my OPNsense VM, despite extensive diagnostics showing the Proxmox bridge and physical network are working correctly.

My Setup:

    - Host: Proxmox VE 8.4.14 (Kernel 6.8.12-16-pve).

    - Hardware: CWWK Mini PC (N100/N150 model) with 4x Intel i226-V 2.5GbE NICs.

    - VM: OPNsense 25.7 (VM 100).

    - Network: UniFi Switch (USW Flex) & AP (U6 IW).

    - VLANs: LAN (untagged, Native VLAN 1), IOT (VLAN 100), GUEST (VLAN 200).

Problem: Traffic from my IOT VLAN (e.g., Chromecast, 192.168.100.100) destined for a server on the LAN (192.168.10.5:443) is being processed on my LAN interface instead of my LAN_IOT interface. The OPNsense firewall log shows this traffic being passed by the LAN interface rules, completely bypassing my LAN_IOT rules. This happens with both OPNsense network configurations I've tried (see below).

Troubleshooting & Evidence:

    - Proxmox Bridge (vmbr1): Is "VLAN aware" (bridge-vlan-aware yes) and the config file confirms bridge-vids 2-4094.

    - tcpdump Tests:

        - tcpdump on the physical NIC (enp2s0) shows VLAN 100 tags arriving from the UniFi switch.

        - tcpdump on the bridge (vmbr1) also shows the VLAN 100 tags are present and being passed to the VM.

    - "Smoking Gun" LXC Test:

        - I created a new Alpine LXC (CT 102) on the same host.

        - I gave it two vNICs on vmbr1: net0 (untagged, 192.168.10.250) and net1 (VLAN Tag 100, 192.168.100.250).

        - I successfully pinged both interfaces from my laptop (pinging .100.250 while on the IOT VLAN, pinging .10.250 while on the  LAN).

Conclusion: This proves my Proxmox bridge (vmbr1) is working correctly and is handling tagged/untagged traffic to an LXC perfectly. The problem is isolated to the OPNsense VM (KVM/QEMU) or its interaction with the bridge.

Failed Fixes (The problem persists after all these steps):

    - Architecture 1 (Router-on-a-Stick): OPNsense VM with one VirtIO vNIC (vtnet1 on vmbr1, no tag), OPNsense handles VLANs internally (vlan01, vlan02 parented to vtnet1). -> Result: Leak.

    - Architecture 2 (PVE-handled VLANs): OPNsense VM with separate VirtIO vNICs on vmbr1 (net1 untagged, net4 with tag=100, net5 with tag=200). OPNsense interfaces assigned directly to vtnet1, vtnet4, vtnet5. -> Result: Same leak.

    - Alternative vNIC Drivers: Changing all OPNsense vNICs to E1000 or vmxnet3 causes the OPNsense VM to kernel panic and fail to boot. Only VirtIO boots, but it has this leak.

    - Host/Driver Fixes:

        - Rebooted Proxmox host multiple times.

        - Reset OPNsense state table.

        - Added bridge-mcsnoop 0 to the bridge config.

        - Disabled the Proxmox firewall on all OPNsense vNICs.

        - Disabled EEE (EEE status: disabled) and GRO (ethtool -K enp2s0 gro off) on the host's physical NIC.

    - IPv6: Allow IPv6 is disabled in OPNsense settings, so this is not an IPv6 leak.

I am completely out of ideas. It seems only the VirtIO vNIC boots, but it's not handling the tagged traffic correctly inside OPNsense, even though the bridge is proven to be working. What else could cause this?

Thanks for any help!

[meyergru]

First: The successful ping does only show connectivity, it might go over a gateway, so this is no absolute proof that your VLAN configuration works. Please show the content of /etc/network/interfaces, too.

It also begs the question: In order to make those VLANs work without the OpnSense VM, you probably have another router, likely a Unifi dreambox or their likes. Could it be that this interferes with the addition of OpnSense, resulting in the "leaks" (which you did not describe exactly what you mean by it).

Second: any configuration involving tagged and untagged packets on the same OpnSense interface is discouraged, because FreeBSD is not particularly good at discriminating between a parent and a VLAN interface. I.e.: the parent interface should not be configured with a subnet.

This rules out Architecture 1, but also: You say you used vlan01 and vlan02, whereas the connection from your physical network to Proxmox has only one (tagged) VLAN and an untagged LAN. So how would you map the untagged LAN to your OpnSense over the interface other than using no tag in the vtnet1 definition?

Note that you still can a standard Unifi environment, where the default LAN is untagged, but other VLANs can exist. Preferably, would then delegate the VLAN handling to Proxmox. This means that you create two vtnet interfaces, one with a VLAN tag and one with your respective VLAN. On OpnSense, both vtnet interfaces are UNTAGGED, because Proxmox handles the tagging in this situation.

Alternatively, you can also use vtnet0 as untagged and have vtnet1 also untagged in Proxmox and then create VLANs on vtnet1 inside OpnSense as needed. That way, vtnet0 only uses untagged frames, whereas vtnet1 only uses tagged VLANs, which also satifies the above condition.

Third: you give vtnet1 with Architecture 1, which implies another interface vtnet0 still being present, potentially causing problems. Same goes for Architecture 2, where net4 and net5 show up, potentially leaving net0-3 as being problematic.

Fourth: Did you follow the guide w/r to tuneables to make vtnet interfaces work?

Also, please create your own thread for this specific question, because I know for a fact that OpnSense can handle VLANs under Proxmox just fine.