Bandwidth limitations problems without any traffic shaping rules

[jinxray]

We have Opnsense instance in cloud environment between separated from each other networks. For some reason, there is serious speed limitations: ~300KB/S-800KB/S. Cloud provider didnt register any problems with cloud platform. If traffic dont go through Opnsense, speed is normal. There is no traffic shaping rules. Rebooted opnsense, didnt work either. Can you point, where should I look and what exactly?

[Patrick M. Hausen]

KVM based cloud and VirtIO network interfaces? Please check and report back.

[jinxray]

KVM based cloud and VirtIO network interfaces? Please check and report back.

Yes, https://yandex.cloud/en/docs/vpc/concepts/software-accelerated-network#reg-vm and https://yandex.cloud/en/docs/compute/operations/image-create/custom-image#requirements

[Patrick M. Hausen]

Try this:

1. System > Settings > Tunables
2. Add tunable, name=hw.vtnet.csum_disable, value=1
3. Save and reboot

[jinxray]

Try this:

1. System > Settings > Tunables
2. Add tunable, name=hw.vtnet.csum_disable, value=1
3. Save and reboot

Thanks! It worked out. Can you explain nature of the problem and why it happened? So, it is some checksum calculation? Am I getting it right, without checksum there will be more resource utilization?

[Patrick M. Hausen]

There's a bug or a common misconfiguration at cloud providers in KVM that works out to our (FreeBSD) disadvantage. The hypervisor announces the virtual network hardware would perform checksum offloading but then does not actually do it. FreeBSD thinks "fine, HW offloading is great" and enables it, which leads to checksum errors, which leads to retransmissions, which leads to orders of magnitude too low throughput.

I identified at least these bug reports having a high probability of being due to this problem:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263229
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277718
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276760
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235607
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259249

So people are searching for FreeBSD bugs where there aren't any.

This has been discussed in the regular bhyve production users call and people are working on it together with Vultr and Digitalocean if I am not mistaken. In the meantime apply the workaround and you will be fine.

It disables checksum offloading, not checksums. The performance penalty is probably negligible.

I'll file a merge request - the tunable does not hurt if you are not running on KVM, and if you are then it most probably helps.

[Patrick M. Hausen]

@jinxray

Following my proposed change to the OPNsense defaults to include the `hw.vtnet.csum_disable` tunable I was reminded that checksum offloading is in fact disabled by default for new OPNsense installations.

Did you initially change any of these settings before you noticed the low performance?