default hw.vtnet.csum_disable=1 causes network slowness

[vsc]

Thank you for bringing this link to my attention. I have been following this problem for 2-3 years now and I cannot remember to have come across this one.

As a base line do you agree that hw.vtnet.csum_disable=1 is a sensible default given that without that setting using FreeBSD as a router inside KVM does not work at all? At least that's what I observe and what led me to suggest this default.

Or is it working for you without hw.vtnet.csum_disable=1 only not up to the performance figures that Linux achieves?

Hi Patrick,

Thank you for looking into this.

Regarding your question, in my opinion, hw.vtnet.csum_disable=1 is counterproductive both theoretically and practically. Theoretically, because OPNsense exposes these configurations in the main web interface under Interface > Settings, yet this "obscure," undocumented default parameter overrides (with no warning) the explicit main UI settings. Furthermore, those UI settings are already disabled by default.

In practical terms, I spent a considerable part of two working days struggling to diagnose the discrepancy between versions 24 and 25. Only after encountering a Reddit post by someone doing a similar investigation did I discover that hw.vtnet.csum_disable=1 was the culprit.

As a side note, the comment in the bug thread describing this as an "optimization" is alarming. An issue that causes broken routing/forwarding or a three- to six-fold loss in performance (in terms of bandwidth and CPU usage) isn't just an optimization problem—it's a serious bug.

Kind Regards,
Vasco

[Patrick M. Hausen]

As a side note, the comment in the bug thread describing this as an "optimization" is alarming. An issue that causes broken routing/forwarding or a three- to six-fold loss in performance (in terms of bandwidth and CPU usage) isn't just an optimization problem—it's a serious bug.

The comment refers to the KVM side not calculating the checksum to save that effort.
The bug is FreeBSD not implementing dealing with KVMs partial checksums.

The problem has some "management attention" now. If that leads to anything is difficult to predict in an open source project.

[meyergru]

The "optimization" remark was for the vtnet subsystem in Linux/KVM. It refers to being able to pass 64K at a time between KVM hosts and Linux VMs, where it works automagically.

The problem is that the FreeBSD vtnet driver currently does not take care of the specific flags to correctly implement checksumming when packets are routed.

(Too late, Patrick just wrote it - the sad part is that it must be fixed upstream in FreeBSD and because of the nature of the bug manifesting predominantly in routers, it might get dismissed as "downstream" only)

Using a default value to make something broken work seems more appropriate than to use a value that definitely breaks things, IMHO.
Actually, the imposed limit by doing the checksumming in software is somewhere above 1 GBit/s, so I would argue that the percentage of people complaining about a default that breaks vtnet is bigger than the percentage of people who cannot reach speeds above 1.5 GBit/s.

Most people get KVM virtualisation wrong even without a hurdle like this.

However, until this gets finally fixed, maybe the implications of the setting should be documented in the official docs. @Monviech to the rescue!

I have included a discussion of this in my HOWTO.

[vsc]

The comment refers to the KVM side not calculating the checksum to save that effort.
The bug is FreeBSD not implementing dealing with KVMs partial checksums.

The problem has some "management attention" now. If that leads to anything is difficult to predict in an open source project.

Thank you for your effort Patrick.

Do you have any visibility if this is being worked on? In our use case this would give an instant 6x boost.

Kind Regards,
Vasco

[Patrick M. Hausen]

Nothing beyond the bug tracker, sorry.

[vsc]

Nothing beyond the bug tracker, sorry.

Good news Patrick, https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059 is progressing with a commit in main branch to fix this bug.

[vsc]

Hi all ( @Patrick M. Hausen, @meyergru @franco)

Since the bug fix is committed to stable 14 and 15, when can we expect it to land on a opnsense release?

Thanks
Vasco

[Patrick M. Hausen]

When it makes its way into e.g. 14.4 ...

[vsc]

FreeBSD release 14.4 is scheduled for March 2026. We can expect it around that time but isn't there a use/business case to backport it in OPNSense? It's a dramatic networking performance penalty in virtual environments.
Kind Regards,
Vasco

[jauling]

Would it make sense to track this topic in the 26.1 Series forum? I checked the 26.1 release notes, did not find anything related to hw, hardware, offload.

TBH, I haven't revisited this topic since my last post in this thread. I'm assuming this issue is still open.

[franco]

Why are we always asking about backports when that's our standard procedure? 26.1 has all the latest backports as the are available on stable/14 and it's going to be in the business edition on 26.4.


Cheers,
Franco

[jauling]

Because I'm a n00b and wanted to confirm. Thanks, and apologies.

[skunkwerks]

There have been some changes upstream in FreeBSD that should be of interest here, both in STABLE 14 & 15:

- https://reviews.freebsd.org/D51475 sctp, tcp, udp: improve deferred computation of checksums
- https://reviews.freebsd.org/D51686 vtnet: fix translation between mbuf and virtio flags

[added as this thread is one of the top google searches related to freebsd tcp offload & checksum - I hope that's helpful for somebody]

[franco]

> - https://reviews.freebsd.org/D51475 sctp, tcp, udp: improve deferred computation of checksums

% git describe 2927ebde300
25.7.5-20-g2927ebde3001

> - https://reviews.freebsd.org/D51686 vtnet: fix translation between mbuf and virtio flags

% git describe 52cbb08e60e
25.7.5-6-g52cbb08e60ed

That means both went into 25.7.8 in November.


Cheers,
Franco

[grb]

Are those changes applied already? I was testing OPNsense with many different options, best I had with those set up:

Code Select Expand

hw.vtnet.csum_disable 0
hw.vtnet.tso_disable 0
hw.vtnet.lro_disable 0I was reaching around 16Gbps speeds but then my PPPOE WAN decreased... from 0.6Gbps to 0.2Gbps
When I set options to:

Code Select Expand

hw.vtnet.csum_disable 1
hw.vtnet.tso_disable 1
hw.vtnet.lro_disable 1I can get max 4.5Gbit/sec OPNsense to other VMs.

When I go through physical network card, I'm reaching near max - 9Gbps. ( but that's probably  hiccups on the other side)



I was testing other VMs:
debian to debian on same network reaches 20Gbps
vanilla pfsense 2.8.1 to debian 4.46 Gbits/sec
and also vanilla freebsd 16.0 current to debian - and that's what worries me, because I get 17Gbps one way, and 5Gbps other way.

Does it mean that even in near future I won't achieve over >15 Gbps speed on OPNsense/FreeBSD?
Or haven't setup up sth obvious?

I was also testing vmxnet3 but speeds were only around 0.5 Gbits/sec

My setup  OPNsense (with extra 10Gbps network card passthrough exclusive) in Proxmox with few VMs.
and current settings are:

Code Select Expand

## System Tunables
| Tunable | Value | Description |
|---------|---------|---------|
| hw.vtnet.csum_disable | 1 | OWN Disables receive and send checksum offload |
| hw.vtnet.tso_disable | 1 | OWN Enables TCP Segmentation Offload |
| hw.vtnet.lro_disable | 1 | OWN Enables Large Receive Offload |
| vm.pmap.pti | 0 | OWN metldown mitigation |
| hw.ibrs_disable | 1 | OWN spectre mitigation |
| net.isr.bindthreads | 1 | OWN performance |
| net.isr.maxthreads | -1 | OWN performance |
| net.inet.rss.enabled | 1 | OWN performance |
| net.inet.rss.bits | 2 | OWN performance |
| net.isr.dispatch | deferred | OWN performance PPPoE |
| hw.vtnet.rx_process_limit | 2048 | OWN performance |