Ookla speedtest high jitter, latency and packet loss on download

[chadima]

Hi,
I'm trying to replace some (cca. 10+) OpenWRT firewalls with OPNsense.

Internet connection:
optical network 10/10 Gbps
Hardware:
Proxmox virtualization on AMD EPYC 9745 256 threads and lot of RAM
(doesn't matter, if i use Proxmox on a completely different HW)
Network cards:
Broadcom BCM57414 through VirtIO or PCIe passthrough
(doesn't matter, if virtualized or directly using HW)

They all report high jitter, latency and packet loss on download,
if i use Ookla speedtest via the CLI directly on the firewall itself
(there should be no NAT involved if used this way).

https://www.speedtest.net/apps/cli

Code Select Expand

pkg add --force "https://install.speedtest.net/app/cli/ookla-speedtest-1.2.0-freebsd13-x86_64.pkg"

Code Select Expand


   Speedtest by Ookla

      Server: 10G-CZNET.CZ - Prague (id: 21429)
         ISP: CESNET z.s.p.o.
Idle Latency:     0.79 ms   (jitter: 0.03ms, low: 0.78ms, high: 0.84ms)
    Download:  7473.87 Mbps (data used: 9.4 GB)                                                   
                 56.42 ms   (jitter: 54.22ms, low: 0.87ms, high: 495.29ms)
      Upload:  5101.07 Mbps (data used: 8.8 GB)                                                   
                  1.15 ms   (jitter: 0.17ms, low: 0.89ms, high: 3.10ms)
Packet Loss:    29.6%
  Result URL: https://www.speedtest.net/result/c/ce4f06d7-db26-42f3-b207-6bb2a5abe15a


I've tried almost all the advice available here.
(do not check gateway etc...)
The OPNsense firewall is set on default values.
Any advice?

PS the same environment with OpenWRT:

Code Select Expand


   Speedtest by Ookla

      Server: ISP Alliance a.s. - Prague (id: 4162)
         ISP: CESNET z.s.p.o.
Idle Latency:     0.77 ms   (jitter: 0.03ms, low: 0.73ms, high: 0.83ms)
    Download:  9043.70 Mbps (data used: 9.5 GB)                                                   
                  4.93 ms   (jitter: 0.24ms, low: 0.80ms, high: 6.77ms)
      Upload:  6626.55 Mbps (data used: 7.1 GB)                                                   
                  1.47 ms   (jitter: 0.48ms, low: 0.86ms, high: 3.43ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/92f73887-737d-411d-95e4-5ae75ac6cce

(the different servers Ookla speedtest connects to in these examples do not matter)

[chadima]

one month spend in trying different "tunables"
and other configuration options...

[OPNenthu]

You may have some success lowering latency with this: https://docs.opnsense.org/manual/how-tos/shaper_bufferbloat.html

It doesn't work well for all ISPs, but when it does, it makes a huge difference.

[chadima]

first thing i have tried

there was a limit and the maximum speed for bufferbloat is 4Gbps (i have networks 10-25Gbps...)
and it doesn't remove the problem with packet loss completely (sometimes is the latency so high -> packet loss)

there is no problem with VyOS, Debian, Redhat, OpenWRT etc.

[slackadelic]

Being virtualized, did you turn off all the hardware checksum offloading in OPNsense?

[REB00T]

You could try setting the tunable `kern.hz` = 1000. In VMs FreeBSD defaults to 100, which saves some CPU but messes up timer accuracy and such. You could try even higher if using shapers at those speeds, but 1000 is the default for bare metal and seems to work well for me at least.

[chadima]

thx
but this didn't help
all the latency is under 1ms and just sometimes during the tst it spikes up to cca 500ms an gets therefore packet lose