Running OPNsense 25.1(.1) on Proxmox 8.3.3 on an N100 box with 4x i226-v NICs, and I've been running into an iperf3 performance issue. I'm using virtio for all the VM networking, no pass-through.
With Ubuntu and FreeBSD VMs, I would see around 15-18Gbps between them, and around 20-25Gbps with Proxmox. Even with OPNsense 24.7, I saw similar numbers. But with OPNsense 25.1, I would hit a 5Gbps ceiling no matter to VMs or Proxmox. After many days of testing, I found that OPNsense 25.1 by default disables checksum offloading AND disables the GUI offload control. I've been banging my head why I couldn't toggle the RXCSUM/TXCSUM/LRO/TSO options via ifconfig like I could with OPNsense 24.7. I've found that when hw.vtnet.csum_disable=1, even things like TSO and LRO cannot be toggled on. The options ALWAYS look like this:
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN (lan)
options=90028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether bc:24:11:55:6e:37
inet 10.4.2.2 netmask 0xffffff00 broadcast 10.4.2.255
inet6 fe80::be24:11ff:fe55:6e37%vtnet0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Regardless if you try to do things like 'ifconfig vtnet0 lro' or 'ifconfig vtnet tso4', they wouldn't do anything.
Are LRO/TSO supposed to be affected by this tunable? Whatever the case is, setting hw.vtnet.csum_disable=0 then enabling the 3 offload settings in the GUI then rebooting, has restored the iperf3 performance in OPNsense 25.1 (for me).
NB: The manpage for vtnet seems to suggest that hw.vtnet.csum_disable SHOULD NOT have any affect on LRO or TSO4.
https://man.freebsd.org/cgi/man.cgi?vtnet(4)
IDK if the settings influence LRO or TSO4, clearly, they should not. However, note that there was a reason to disable HW checksumming (https://forum.opnsense.org/index.php?msg=216918), especially under virtualisation.
You might also try enabling multiqueue.
Thanks for the link, @meyergru - I would have stepped in here, anyway, but you were faster.
I suggested the change in the tunable because of the linked issues that manifest themselves in various KVM based cloud hosting providers. I am really surprised it results in a performance degradation for you.
But one question: are you running iperf3 on OPNsense itself for one end of the connection? Why? A firewall forwards and filters packets and is rarely the end of a TCP connection. That's why when submitting the change I was confident it would not be noticeable in cases you are not hit by the cloud KVM problem.
If yes, what performance do you get with LRO and TSO disabled and both endpoints on two different sides of the firewall?
What I think we probably should do:
- keep the tunable visible in the default set of tunables in the UI
- improve the documentation by adding a section about this issue in the performance tuning chapter of the manual
- come to a consent and decice if it is better to have the default at 0 (off) or 1 (on)
@franco, what do you think? I could do the documentation in rather short time.
Kind regards,
Patrick
Apologies for the reponse lag.
You guys were right, as well as basically everything else I've read about vtnet offloading. With the checksum offload enabled, I could get some decent iperf3 numbers directly to/from my opnsense vm, but that was basically a moot point, since this it couldn't route! It is very strange, since I could ping through the router from lan1 to lan2 via opnsense, but I couldn't tcp connect between the lans. I've never seen anything like it.
Once I disabled checksum offload (I forget if it was tx or rx) routing was working, though I'm still surprised I'm seeing less than 4Gbps between vms on the same proxmox host when traversing opnsense. I guess I can try a different router to compare.
FWIW, setting hw.vtnet.csum_disable=1 in a FreeBSD 14.2 vm also resulted in LRO and TSO to become permanently disabled as well, so this seems to be more of a freebsd kernel bug than an opnsense one.
I think this is a case of "no good deed goes unpunished". I don't mind either way... the variable is visible by default in the tunables so the use is explicit.
However, the vtnet code seems to make its intention clear WRT CSUM/LRO/TSO being intertwined:
https://github.com/opnsense/src/blob/e33a9f9b088b14acf4edf405a3043c2215f0d492/sys/dev/virtio/network/if_vtnet.c#L635-L645
Cheers,
Franco
Quote from: jauling on February 19, 2025, 12:43:47 AM...but that was basically a moot point, since this it couldn't route! It is very strange, since I could ping through the router from lan1 to lan2 via opnsense, but I couldn't tcp connect between the lans. I've never seen anything like it.
Once I disabled checksum offload (I forget if it was tx or rx) routing was working, ...
I do not see this, there must be something else wrong with that. What are lan1 and lan2? Do they have separate subnets or are they bridged? If the latter, did you set all the neccessary tuneables?
Quote from: meyergru on February 19, 2025, 08:42:07 AMI do not see this, there must be something else wrong with that. What are lan1 and lan2? Do they have separate subnets or are they bridged? If the latter, did you set all the neccessary tuneables?
lan1=10.4.2.0/24
lan2=10.4.3.0/24
opnsense vm sits between on 10.4.2.2 and 10.4.3.2
i have a live edgerouter serving my internet at 10.4.2.1
my proxmox host nics -> linux bridges -> opnsense virtio
enp1s0->vmbr0->wan(disabled)
enp2s0->vmbr1->lan1
enp3s0->vmbr2->lan2
enp4s0->vmbr3->unused
The ubuntu vms I move between lan1 and lan2 for iperf3 testing.
I've set a boatload of tuneables as per what I read from here: https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/
Summarized:
hw.ibrs_disable=1
net.isr.maxthreads=-1
net.isr.bindthreads=1
net.isr.dispatch=deferred
net.inet.rss.enabled=1
net.inet.rss.bits=1
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=4194304
net.inet.tcp.recvspace=65536
net.inet.tcp.sendbuf_inc=65536
net.inet.tcp.sendbuf_max=4194304
net.inet.tcp.sendspace=65536
net.inet.tcp.soreceive_stream=1
net.inet.tcp.mssdflt=1240
net.inet.tcp.abc_l_var=52
net.inet.tcp.minmss=536
kern.random.fortuna.minpoolsize=128
net.isr.defaultqlimit=2048I've also got all the hardware offload and filtering disabled. This is my first foray into proxmox and opnsense, so I hope I'm not doing anything obviously wrong? It is quite possible though!
One thing I haven't yet done is reset the BIOS of my N100 proxmox server.
What I do not get is why turning off hardware offloading (which is the default) does break routing. I would expect that maybe your gateway settings are off, given that you do not use OpnSense as your main gateway and that the WAN on OpnSense is disabled.
If that was the case, you could well ping any address on either side of OpnSense, yet a machine from one side will not ping any machine on the other because its default gateway is not OpnSense, but your main router.
Turning it on breaks forwarding in certain KVM based environments. The bootloader variable is named blah.blah.disable 🙂
Quote from: meyergru on February 19, 2025, 05:30:59 PMWhat I do not get is why turning off hardware offloading (which is the default) does break routing. I would expect that maybe your gateway settings are off, given that you do not use OpnSense as your main gateway and that the WAN on OpnSense is disabled.
If that was the case, you could well ping any address on either side of OpnSense, yet a machine from one side will not ping any machine on the other because its default gateway is not OpnSense, but your main router.
Crap, did I write it wrong? Sorry if it wasn't clear. Enabling hardware offloading, specifically checksum offloading, breaks routing for me (though ICMP ping goes through). Once I disable checksum offloading, routing works again. Oddly enough, traceroutes also fail when csum offload is enabled.
FWIW, I know that hw.vtnet.csum_disable=1 (and reboot) is basically the same as ifconfig vtnetX -txcsum -rxcsum
I have double checked my gateways just now, and also have static routes added.
lan1 vm:
default via 10.4.2.1 dev enp6s18 proto dhcp src 10.4.2.184 metric 100
10.4.2.0/24 dev enp6s18 proto kernel scope link src 10.4.2.184 metric 100
10.4.3.0/24 via 10.4.2.2 dev enp6s18 lan2 vm:
default via 10.4.3.2 dev enp6s18 proto dhcp src 10.4.3.101 metric 100
10.4.3.0/24 dev enp6s18 proto kernel scope link src 10.4.3.101 metric 100 FWIW, your tagline/signature looks like a similar fanless box as what I'm running. You're right. My opnsense box right now is purely being used as a router, forwarding packets between lan1 and lan2. I tried disabling the opnsense firewall too, but that also had no effect on iperf3 performance between vms on either side.
I just stood up a openwrt router, and out of the box I'm seeing around 9.3Gbps iperf3 between lan1 and lan2. I'm turning it off though, openwrt really looks horrible and the GUI gives me a headache.
Hi everyone,
I'd like to highlight the importance of thoroughly documenting—and ideally resolving—this bug. Late last year, our organization decided to standardize on OPNSense as our routing/firewall platform for both virtual and hardware environments.
We began by deploying several OPNSense virtual instances on QEMU-KVM, running on Linux. From installation to configuration and general operation, everything performed admirably—until we conducted non-functional performance tests. At that point, we discovered a significant bottleneck: intra-virtualization host forwarding/routing (VLAN) throughput was roughly a third of what we achieved with a simple Linux router (6 Gbit/s vs 19 Gbit/s) and used triple the CPU in comparison.
Initially, we assumed tuning was the solution, given the many articles, Reddit discussions, and blog posts stressing various parameters to adjust. Unfortunately, none of those tweaks addressed our performance gap. Further investigation led us to this long-standing issue:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059
(https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059)
Although enabling certain hardware offloading settings appears to restore performance, it disrupts forwarding between VMs on the same host due to an incomplete implementation in the virtio net driver: https://github.com/freebsd/freebsd-src/blob/9efd215411bb5ead2bc0ab208b4c19e46da0d2c9/sys/dev/virtio/network/if_vtnet.c#L1761-L1777 (https://github.com/freebsd/freebsd-src/blob/9efd215411bb5ead2bc0ab208b4c19e46da0d2c9/sys/dev/virtio/network/if_vtnet.c#L1761-L1777)
We're reaching out to ask if there are any plans to address this. Virtual environments are becoming ever more critical across organizations, and the Linux QEMU-KVM stack is especially prevalent both on-premises and in the cloud. Given these issues, we've decided to suspend our OPNSense rollout for now. Notably, many authors of the tuning guides and related posts we've come across seem to have encountered the same problem and ultimately moved on to Linux-based router/firewall platforms.
Thank you for your time. We hope this issue can be resolved soon.
Kind Regards,
Vasco
Quote from: vsc on March 06, 2025, 04:54:26 PMInitially, we assumed tuning was the solution, given the many articles, Reddit discussions, and blog posts stressing various parameters to adjust. Unfortunately, none of those tweaks addressed our performance gap. Further investigation led us to this long-standing issue:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059
(https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059)
Thank you for bringing this link to my attention. I have been following this problem for 2-3 years now and I cannot remember to have come across this one.
As a base line do you agree that hw.vtnet.csum_disable=1 is a sensible default given that without that setting using FreeBSD as a router inside KVM does not work at all? At least that's what I observe and what led me to suggest this default.
Or is it working for you without hw.vtnet.csum_disable=1 only not up to the performance figures that Linux achieves?
I will probably take this from here directly to that bug report and my personal "FreeBSD bubble" of developers. It's hardly anything OPNsense can fix on their own, it seems like it needs a serious effort by upstream.
This particularly struck me:
QuoteThis is simply because FreeBSD does not know how to forward and NAT packets with offload.
KVM forwards packets to a virtual machine that can be up to 64 kilobytes in size and without checksums.
This is optimization!!!
Kind regards,
Patrick
I remember that for: You knew that one once (https://forum.opnsense.org/index.php?msg=216918), Patrick...
Quote from: meyergru on March 06, 2025, 08:25:21 PMI remember that for: You knew that one once (https://forum.opnsense.org/index.php?msg=216918), Patrick...
Thanks I guess :-P
:-P
Still, it is bad, because one way, you will have no routing and the other, speed will be slower, at least when the underlying link speeds are higher than 1 Gbps...
Quote from: Patrick M. Hausen on March 06, 2025, 06:56:33 PMThank you for bringing this link to my attention. I have been following this problem for 2-3 years now and I cannot remember to have come across this one.
As a base line do you agree that hw.vtnet.csum_disable=1 is a sensible default given that without that setting using FreeBSD as a router inside KVM does not work at all? At least that's what I observe and what led me to suggest this default.
Or is it working for you without hw.vtnet.csum_disable=1 only not up to the performance figures that Linux achieves?
Hi Patrick,
Thank you for looking into this.
Regarding your question, in my opinion, hw.vtnet.csum_disable=1 is counterproductive both theoretically and practically. Theoretically, because OPNsense exposes these configurations in the main web interface under Interface > Settings, yet this "obscure," undocumented default parameter overrides (with no warning) the explicit main UI settings. Furthermore, those UI settings are already disabled by default.
In practical terms, I spent a considerable part of two working days struggling to diagnose the discrepancy between versions 24 and 25. Only after encountering a Reddit post by someone doing a similar investigation did I discover that hw.vtnet.csum_disable=1 was the culprit.
As a side note, the comment in the bug thread describing this as an "optimization" is alarming. An issue that causes broken routing/forwarding or a three- to six-fold loss in performance (in terms of bandwidth and CPU usage) isn't just an optimization problem—it's a serious bug.
Kind Regards,
Vasco
Quote from: vsc on March 07, 2025, 12:26:42 PMAs a side note, the comment in the bug thread describing this as an "optimization" is alarming. An issue that causes broken routing/forwarding or a three- to six-fold loss in performance (in terms of bandwidth and CPU usage) isn't just an optimization problem—it's a serious bug.
The comment refers to the KVM side not calculating the checksum to save that effort.
The bug is FreeBSD not implementing dealing with KVMs partial checksums.
The problem has some "management attention" now. If that leads to anything is difficult to predict in an open source project.
The "optimization" remark was for the vtnet subsystem in Linux/KVM. It refers to being able to pass 64K at a time between KVM hosts and Linux VMs, where it works automagically.
The problem is that the FreeBSD vtnet driver currently does not take care of the specific flags to correctly implement checksumming when packets are routed.
(Too late, Patrick just wrote it - the sad part is that it must be fixed upstream in FreeBSD and because of the nature of the bug manifesting predominantly in routers, it might get dismissed as "downstream" only)
Using a default value to make something broken work seems more appropriate than to use a value that definitely breaks things, IMHO.
Actually, the imposed limit by doing the checksumming in software is somewhere above 1 GBit/s, so I would argue that the percentage of people complaining about a default that breaks vtnet is bigger than the percentage of people who cannot reach speeds above 1.5 GBit/s.
Most people get KVM virtualisation wrong even without a hurdle like this.
However, until this gets finally fixed, maybe the implications of the setting should be documented in the official docs. @Monviech to the rescue!
I have included a discussion of this in my HOWTO (https://forum.opnsense.org/index.php?topic=44159.0).
Quote from: Patrick M. Hausen on March 07, 2025, 12:32:14 PMThe comment refers to the KVM side not calculating the checksum to save that effort.
The bug is FreeBSD not implementing dealing with KVMs partial checksums.
The problem has some "management attention" now. If that leads to anything is difficult to predict in an open source project.
Thank you for your effort Patrick.
Do you have any visibility if this is being worked on? In our use case this would give an instant 6x boost.
Kind Regards,
Vasco
Nothing beyond the bug tracker, sorry.