How to improve internet speed per connection

[ybalbert]

Hello everyone, I'm new to OpnSense and have only used it since a month ago. I like it so far thanks to the friendly UI and the comprehensive features. I feel there's a lot of to learn from it :)

I installed OpnSense directly on a mini-PC with 14 cores CPU and plenty of RAM (probably overkill I know), and enabled Zenarmor on the LAN/vLAN interfaces. The inter vLAN speed is ~5gbits/second using iPerf which works for me (and probably have room to tune), but the internet speed is slower than I expected. With the default settings of either speedtest.net or fast.com, the download speed is around 400mbits/s while the upload speeds is 900+mbits/s. After I change the "maximum connection" setting on fast.com from 8 (default value) to 16 connections, I get the expected download speed of 940mbits/s.

Does anyone know how to increase the speed per connection? Is it because my CPU's single core performance is not as good (however the CPU usage is quite low on any core), or there's some settings I set incorrectly? This issue doesn't happen when I use a different router or connect the internet to my PC directly. Thanks for the help in advance!

[bartjsmit]

Does anyone know how to increase the speed per connection?

Emigrate to South Korea?  :) What are you running that actually saturates 900 Mbps? Just curious.

[ybalbert]

Emigrate to South Korea?

One day maybe :)

What are you running that actually saturates 900 Mbps?

Not much, only occasionally when I download some LLM models which opens up enough connections to saturate the bandwidth. I just want to learn if there's any setting I'm missing. The internet speed test result makes me sad a little bit (and I can't show off to my wife :P)

[bartjsmit]

Better reason than most  - the learning, not the bragging 8)

Pare down the config to just NAT and test again. Tweak the NIC (in OPNsense) / CPU / RAM (in BIOS) settings to see if any increase speed. Once you're maxed out, add features that you need and then features that you want. Note where you see a drop and decide if the feature is worth it.

Bart...

[Greg_E]

I would disable ZenArmor and try again, that might be the issue combined with single core speed (as mentioned).

Are you also running IDS/IPS (Suricata)? That might be slowing things down a bit too.

I get a similar slowdown (fast.com) with both running on my AMD V1756b with 16gb of ram and a known good gigabit connection. I'm guessing this is mostly ZenArmor and it's single threaded operation. I think I tried with Zenarmor disable and got a much higher test, but that was months ago and I don't really remember.

I just tested 620 down and 770 up with IDS/IPS on the wan and Zenarmor on the lan and crowdsec somewhere in the middle.

[ybalbert]

Are you also running IDS/IPS (Suricata)? That might be slowing things down a bit too.

No, I'm not using IDS/IPS right now. I enabled it on WAN for a while but it didn't report any alert, so I disabled it to focus on troubleshooting other parts of the system.

Just played around with the settings a bit more. Some updates:

• With the current hardware and default OpnSense settings (freshly installed OpnSense with just WAN and LAN interface), the internet speed is still capped at 300mbps with the default 8 connections on fast.com.
• With Zenarmor on and limiting the processing to a single core, one core reached 70% of usage when I change the number of connections to 16 on fast.com (the speed is 900+mbps). With the default 8 connections, this core is using ~30%, so the single CPU core doesn't seem to be the bottleneck.
• Disabling Zenarmor helped with the inter-VLAN speed which went up to ~10gbps from ~5gbps. However, it doesn't affect the internet speed test result.
• I checked the inter-VLAN speed and router CPU usage w/ and wo Zenarmor enabled. When Zenarmor is disabled, the cores (threads) are used evenly (~30% per core when reaching the max speed of 10gbps). When Zenarmor is on, one core reaches 100% and rest is close to 0%. I guess multi-core on Zenarmor is not supported yet (probably will be there later this year? https://www.zenarmor.com/roadmap). To run this test, I enabled rss and isr thread binding.
• The Zenarmor setting, "Do not pin engine packet processors to dedicate CPU cores" (in Zenarmor
  -> settings -> configuration), helped to increase the VLAN speed a little bit (from 5gbps to 7gbps). It's still heavily using one core, but the other cores see some usage.

The original problem is still not resolved yet unfortunately, but I'm learning and hopefully will find the answer one day  :-*

[Greg_E]

I wonder why your system is performing so much differently from mine. Mine is very vanilla on 24.1.8 CE (still haven't had the time to build up my business licensed machine). I may change that "do not pin" setting in Zenarmor to see if speed between networks increases, but I'm mostly still getting really good speeds between them (all gigabit and pretty close to max speeds).

Do you have all of the offloading in the NICs turned off? That was kind of required with IDS/IPS and to an extent Zenarmor.

And yes I'm looking forward to the Zenarmor multicore update, many of us are looking forward to this. More processor should mean more throughput.

[ybalbert]

Do you have all of the offloading in the NICs turned off?

I disabled the three hardware offload settings.

I don't see anything obviously wrongly with the OpnSense configs. I have even tried to install pfSense on the same hardware to benchmark, and the result was the same. So I guess it's a mostly likely a hardware thing or a general FreeBSD limit. Unfortunately I don't have another box to test with. Will probably leave it like this for now.