Virtio performance routing across VLANS

[toxic]

Hello,


I have a virtualized opnsense router and can't seem to manage to get decent performance while routing packets between vlans.


On PvE I defined vmbr0

Code: [Select]

auto vmbr0

iface vmbr0 inet manual

        bridge-ports bond0

        bridge-stp on

        bridge-fd 0

        bridge-vlan-aware yes

        bridge-vids 1-4094

        pre-up ethtool -G bond0 rx 1024 tx 1024

        pre-up ethtool -K bond0 tx off gso off

        post-up ethtool -K vmbr0 tx off gso off

#Bridge All VLANs to SWITCH

Now I pass vmbr0 to my opnsenseVM as virtio, it extracts vtnet0_vlan2 and vtnet0_vlan3 properly, serves DHCP properly, and routes traffic between the vlans according to fw rules.


For testing I use an LXC attached to vmbr0 using vlan tag 3, and the PvE host itself attached to vmbr2 as follows

Code: [Select]

auto vmbr2

iface vmbr2 inet static

        address 10.2.2.2/24

        gateway 10.2.2.1

        bridge-ports vmbr0.2

        bridge-stp on

        bridge-fd 0

        post-up   ip rule add from 10.2.2.0/24 table 2Vlan prio 1

        post-up   ip route add default via 10.2.2.1 dev vmbr2 table 2Vlan

        post-up   ip route add 10.2.2.0/24 dev vmbr2 table 2Vlan

        pre-up ethtool -G vmbr0.2 rx 1024 tx 1024

        pre-up ethtool -K vmbr0.2 tx off gso off

        post-up ethtool -K vmbr2 tx off gso off

#VMs bridge

I have in opnsense the settings to disable everything: CRC offloading, TSO, LRO and VLAN offloading as well.


All CPU monitoring I can do show that during an iperf3 across vlans there is ample idle time on all CPU (80%) on all 3 nodes involved (it's a homelab nothing else is stressing anything here)


And yet I get 800-900MB/s when crossing vlans...

On the same vlan I get 18-19GB/s

I also managed to get 12GB/s from one VLan to the router but that was only by enabling the CRC offloading in the opnsense virtual router... But enabling CRC offloads breaks inter-vlan communication, the same opnsense VM, no rules changes, CRC offloaded = 12GB/s in one VLan but no Vlan 2 to 3 communication possible, or CRC not offloaded and only 850MB/s...


I'm getting stuck...

The HW NIC behind bonds is an Intel I225V-rev04, it's alone in the bond, later it will be bonded with a gigabit real Tek in case I plug the cable in the wrong NIC


If you have any ideas as to how I should set it up to achieve>10GB/s between VMs and LXCs regardless of the VLAN I put them on, anything would be helpful here I think.


Thanks for the reading and thanks in advance for any idea!

[Patrick M. Hausen]

IMHO you should leave the VLAN handling to the hypervisor and give one virtual interface per VLAN to OPNsense.

[toxic]

Thanks for the reply @Patrick
I tried that too, adding a virtio nic for some of my vmbr0.2 and vmbr0.3 and reassigned those in opnsense. Same behavior though, CRC offloaded means no inter-vlan communication and CRC not offloaded meaning it works but 900MB/s only.

Yesterday night I tried applying the settings found here https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/
It got me up to 2GB/s inter network but still quite far from the 12GB/s I get in the same Linux bridge...

My CPU is a AMD Ryzen 7 5700U, maybe I'm dreaming and routing from 1 Linux bridge vmbr0.2 to the next vmbr0.3 can't happen any faster than 2GB/s on my machine? I already had to give it all CPU cores available and quite a bit of ram...

Best I think would be to find a way to leave CRC on and still make the routing work across VLANS... But I don't know enough to understand which device is it actually offloaded to, seems a Linux bridge has nothing visible from ethtool in terms of offloading features so maybe my above -K on vmbr0 or any of it's VLan is doing nothing anyway...

[Patrick M. Hausen]

IMHO CRC on does not make sense for traffic that is routed *through* a device.

Apart from that you seem to have hit a limit here. Do you have 100 Gbit/s interfaces or how do you intend to reach >10 G/s speeds?

[toxic]

For now the HW only has 2.5G NIC but I'll upgrade to 10G when I get the money, might be a while seeing I 'll need managed or at least vlan capable switches and I refuse using power hungry old enterprise HW going forward 😉

So anything above 10G is not needed nor my target but still quite a bit of room between the current 2G and the target 10G.

Faster than 10G can still be useful within this machine as it my NAS with several SSDs and some VMs are using it over this virtual network. Faster "remote" storage is always a plus I figured 😉
Will make another attempt at passing individual vlans to opnsense and trying to play around with my post up rules find out if one setup allows me to enable CRC offloading in opnsense seeing it seemed to make a world of difference at least within the same vlan between an LXC on the Linux bridge and opnsense on the same bridge

[Patrick M. Hausen]

There's a bug in KVM in combination with VirtIO interfaces and FreeBSD guests. You definitely must disable all checksum offloading.

[toxic]

Yeah, I thought I had read something like this... Any chance you have some infos on where exactly this bug is being tracked and if I can follow it's eventual resolution hopefully ?
Still in the proces of setting up my test setup to play around with offloading anf confirm I'm stuck at 2GB/s as my max throughput with opnSense, takes more than anticipates, need a third interface to administer opnsense with the gui, and a fourth to give WAN access else I can't apt isntall iperf3... Will post my findings soon but I expect I'll hit this bug yeah...

[Patrick M. Hausen]

Any chance you have some infos on where exactly this bug is being tracked and if I can follow it's eventual resolution hopefully ?

Nothing in the FreeBSD bug tracker as far as I am aware. I just heard directly from Antranig (https://antranigv.am) that he is working on the issue with Vultr.com. Seems it must be fixed on the KVM side.

Set a loader.conf tunable:

hw.vtnet.csum_disable="1"

and all will work well for now.