default hw.vtnet.csum_disable=1 causes network slowness

[jauling]

Running OPNsense 25.1(.1) on Proxmox 8.3.3 on an N100 box with 4x i226-v NICs, and I've been running into an iperf3 performance issue. I'm using virtio for all the VM networking, no pass-through.

With Ubuntu and FreeBSD VMs, I would see around 15-18Gbps between them, and around 20-25Gbps with Proxmox. Even with OPNsense 24.7, I saw similar numbers. But with OPNsense 25.1, I would hit a 5Gbps ceiling no matter to VMs or Proxmox. After many days of testing, I found that OPNsense 25.1 by default disables checksum offloading AND disables the GUI offload control. I've been banging my head why I couldn't toggle the RXCSUM/TXCSUM/LRO/TSO options via ifconfig like I could with OPNsense 24.7. I've found that when hw.vtnet.csum_disable=1, even things like TSO and LRO cannot be toggled on. The options ALWAYS look like this:

Code Select Expand

vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN (lan)
        options=90028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether bc:24:11:55:6e:37
        inet 10.4.2.2 netmask 0xffffff00 broadcast 10.4.2.255
        inet6 fe80::be24:11ff:fe55:6e37%vtnet0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Regardless if you try to do things like 'ifconfig vtnet0 lro' or 'ifconfig vtnet tso4', they wouldn't do anything.

Are LRO/TSO supposed to be affected by this tunable? Whatever the case is, setting hw.vtnet.csum_disable=0 then enabling the 3 offload settings in the GUI then rebooting, has restored the iperf3 performance in OPNsense 25.1 (for me).

NB: The manpage for vtnet seems to suggest that hw.vtnet.csum_disable SHOULD NOT have any affect on LRO or TSO4.
https://man.freebsd.org/cgi/man.cgi?vtnet(4)

[meyergru]

IDK if the settings influence LRO or TSO4, clearly, they should not. However, note that there was a reason to disable HW checksumming, especially under virtualisation.

You might also try enabling multiqueue.

[Patrick M. Hausen]

Thanks for the link, @meyergru - I would have stepped in here, anyway, but you were faster.

I suggested the change in the tunable because of the linked issues that manifest themselves in various KVM based cloud hosting providers. I am really surprised it results in a performance degradation for you.

But one question: are you running iperf3 on OPNsense itself for one end of the connection? Why? A firewall forwards and filters packets and is rarely the end of a TCP connection. That's why when submitting the change I was confident it would not be noticeable in cases you are not hit by the cloud KVM problem.

If yes, what performance do you get with LRO and TSO disabled and both endpoints on two different sides of the firewall?


What I think we probably should do:

- keep the tunable visible in the default set of tunables in the UI
- improve the documentation by adding a section about this issue in the performance tuning chapter of the manual
- come to a consent and decice if it is better to have the default at 0 (off) or 1 (on)

@franco, what do you think? I could do the documentation in rather short time.

Kind regards,
Patrick

[jauling]

Apologies for the reponse lag.

You guys were right, as well as basically everything else I've read about vtnet offloading. With the checksum offload enabled, I could get some decent iperf3 numbers directly to/from my opnsense vm, but that was basically a moot point, since this it couldn't route! It is very strange, since I could ping through the router from lan1 to lan2 via opnsense, but I couldn't tcp connect between the lans. I've never seen anything like it.

Once I disabled checksum offload (I forget if it was tx or rx) routing was working, though I'm still surprised I'm seeing less than 4Gbps between vms on the same proxmox host when traversing opnsense. I guess I can try a different router to compare.

FWIW, setting hw.vtnet.csum_disable=1 in a FreeBSD 14.2 vm also resulted in LRO and TSO to become permanently disabled as well, so this seems to be more of a freebsd kernel bug than an opnsense one.

[franco]

I think this is a case of "no good deed goes unpunished". I don't mind either way... the variable is visible by default in the tunables so the use is explicit.

However, the vtnet code seems to make its intention clear WRT CSUM/LRO/TSO being intertwined:

https://github.com/opnsense/src/blob/e33a9f9b088b14acf4edf405a3043c2215f0d492/sys/dev/virtio/network/if_vtnet.c#L635-L645


Cheers,
Franco

[meyergru]

...but that was basically a moot point, since this it couldn't route! It is very strange, since I could ping through the router from lan1 to lan2 via opnsense, but I couldn't tcp connect between the lans. I've never seen anything like it.

Once I disabled checksum offload (I forget if it was tx or rx) routing was working, ...

I do not see this, there must be something else wrong with that. What are lan1 and lan2? Do they have separate subnets or are they bridged? If the latter, did you set all the neccessary tuneables?

[jauling]

I do not see this, there must be something else wrong with that. What are lan1 and lan2? Do they have separate subnets or are they bridged? If the latter, did you set all the neccessary tuneables?

lan1=10.4.2.0/24
lan2=10.4.3.0/24
opnsense vm sits between on 10.4.2.2 and 10.4.3.2
i have a live edgerouter serving my internet at 10.4.2.1

my proxmox host nics -> linux bridges -> opnsense virtio
enp1s0->vmbr0->wan(disabled)
enp2s0->vmbr1->lan1
enp3s0->vmbr2->lan2
enp4s0->vmbr3->unused

The ubuntu vms I move between lan1 and lan2 for iperf3 testing.

I've set a boatload of tuneables as per what I read from here: https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/

Summarized:

Code Select Expand

hw.ibrs_disable=1
net.isr.maxthreads=-1
net.isr.bindthreads=1
net.isr.dispatch=deferred
net.inet.rss.enabled=1
net.inet.rss.bits=1
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=4194304
net.inet.tcp.recvspace=65536
net.inet.tcp.sendbuf_inc=65536
net.inet.tcp.sendbuf_max=4194304
net.inet.tcp.sendspace=65536
net.inet.tcp.soreceive_stream=1
net.inet.tcp.mssdflt=1240
net.inet.tcp.abc_l_var=52
net.inet.tcp.minmss=536
kern.random.fortuna.minpoolsize=128
net.isr.defaultqlimit=2048
I've also got all the hardware offload and filtering disabled. This is my first foray into proxmox and opnsense, so I hope I'm not doing anything obviously wrong? It is quite possible though!

One thing I haven't yet done is reset the BIOS of my N100 proxmox server.

[meyergru]

What I do not get is why turning off hardware offloading (which is the default) does break routing. I would expect that maybe your gateway settings are off, given that you do not use OpnSense as your main gateway and that the WAN on OpnSense is disabled.

If that was the case, you could well ping any address on either side of OpnSense, yet a machine from one side will not ping any machine on the other because its default gateway is not OpnSense, but your main router.

[Patrick M. Hausen]

Turning it on breaks forwarding in certain KVM based environments. The bootloader variable is named blah.blah.disable 🙂

[jauling]

What I do not get is why turning off hardware offloading (which is the default) does break routing. I would expect that maybe your gateway settings are off, given that you do not use OpnSense as your main gateway and that the WAN on OpnSense is disabled.

If that was the case, you could well ping any address on either side of OpnSense, yet a machine from one side will not ping any machine on the other because its default gateway is not OpnSense, but your main router.

Crap, did I write it wrong? Sorry if it wasn't clear. Enabling hardware offloading, specifically checksum offloading, breaks routing for me (though ICMP ping goes through). Once I disable checksum offloading, routing works again. Oddly enough, traceroutes also fail when csum offload is enabled.
FWIW, I know that hw.vtnet.csum_disable=1 (and reboot) is basically the same as ifconfig vtnetX -txcsum -rxcsum

I have double checked my gateways just now, and also have static routes added.

lan1 vm:

Code Select Expand

default via 10.4.2.1 dev enp6s18 proto dhcp src 10.4.2.184 metric 100
10.4.2.0/24 dev enp6s18 proto kernel scope link src 10.4.2.184 metric 100
10.4.3.0/24 via 10.4.2.2 dev enp6s18
lan2 vm:

Code Select Expand

default via 10.4.3.2 dev enp6s18 proto dhcp src 10.4.3.101 metric 100
10.4.3.0/24 dev enp6s18 proto kernel scope link src 10.4.3.101 metric 100

FWIW, your tagline/signature looks like a similar fanless box as what I'm running. You're right. My opnsense box right now is purely being used as a router, forwarding packets between lan1 and lan2. I tried disabling the opnsense firewall too, but that also had no effect on iperf3 performance between vms on either side.

I just stood up a openwrt router, and out of the box I'm seeing around 9.3Gbps iperf3 between lan1 and lan2. I'm turning it off though, openwrt really looks horrible and the GUI gives me a headache.