Messages

The comment refers to the KVM side not calculating the checksum to save that effort.
The bug is FreeBSD not implementing dealing with KVMs partial checksums.

The problem has some "management attention" now. If that leads to anything is difficult to predict in an open source project.

Thank you for your effort Patrick.

Do you have any visibility if this is being worked on? In our use case this would give an instant 6x boost.

Kind Regards,
Vasco

Thank you for bringing this link to my attention. I have been following this problem for 2-3 years now and I cannot remember to have come across this one.

As a base line do you agree that hw.vtnet.csum_disable=1 is a sensible default given that without that setting using FreeBSD as a router inside KVM does not work at all? At least that's what I observe and what led me to suggest this default.

Or is it working for you without hw.vtnet.csum_disable=1 only not up to the performance figures that Linux achieves?

Hi Patrick,

Thank you for looking into this.

Regarding your question, in my opinion, hw.vtnet.csum_disable=1 is counterproductive both theoretically and practically. Theoretically, because OPNsense exposes these configurations in the main web interface under Interface > Settings, yet this "obscure," undocumented default parameter overrides (with no warning) the explicit main UI settings. Furthermore, those UI settings are already disabled by default.

In practical terms, I spent a considerable part of two working days struggling to diagnose the discrepancy between versions 24 and 25. Only after encountering a Reddit post by someone doing a similar investigation did I discover that hw.vtnet.csum_disable=1 was the culprit.

As a side note, the comment in the bug thread describing this as an "optimization" is alarming. An issue that causes broken routing/forwarding or a three- to six-fold loss in performance (in terms of bandwidth and CPU usage) isn't just an optimization problem—it's a serious bug.

Kind Regards,
Vasco

Hi everyone,

I'd like to highlight the importance of thoroughly documenting—and ideally resolving—this bug. Late last year, our organization decided to standardize on OPNSense as our routing/firewall platform for both virtual and hardware environments.

We began by deploying several OPNSense virtual instances on QEMU-KVM, running on Linux. From installation to configuration and general operation, everything performed admirably—until we conducted non-functional performance tests. At that point, we discovered a significant bottleneck: intra-virtualization host forwarding/routing (VLAN) throughput was roughly a third of what we achieved with a simple Linux router (6 Gbit/s vs 19 Gbit/s) and used triple the CPU in comparison.

Initially, we assumed tuning was the solution, given the many articles, Reddit discussions, and blog posts stressing various parameters to adjust. Unfortunately, none of those tweaks addressed our performance gap. Further investigation led us to this long-standing issue:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059


Although enabling certain hardware offloading settings appears to restore performance, it disrupts forwarding between VMs on the same host due to an incomplete implementation in the virtio net driver: https://github.com/freebsd/freebsd-src/blob/9efd215411bb5ead2bc0ab208b4c19e46da0d2c9/sys/dev/virtio/network/if_vtnet.c#L1761-L1777

We're reaching out to ask if there are any plans to address this. Virtual environments are becoming ever more critical across organizations, and the Linux QEMU-KVM stack is especially prevalent both on-premises and in the cloud. Given these issues, we've decided to suspend our OPNSense rollout for now. Notably, many authors of the tuning guides and related posts we've come across seem to have encountered the same problem and ultimately moved on to Linux-based router/firewall platforms.

Thank you for your time. We hope this issue can be resolved soon.

Kind Regards,
Vasco