Messages

Do you have all of the offloading in the NICs turned off?

I disabled the three hardware offload settings.

I don't see anything obviously wrongly with the OpnSense configs. I have even tried to install pfSense on the same hardware to benchmark, and the result was the same. So I guess it's a mostly likely a hardware thing or a general FreeBSD limit. Unfortunately I don't have another box to test with. Will probably leave it like this for now.

Are you also running IDS/IPS (Suricata)? That might be slowing things down a bit too.

No, I'm not using IDS/IPS right now. I enabled it on WAN for a while but it didn't report any alert, so I disabled it to focus on troubleshooting other parts of the system.

Just played around with the settings a bit more. Some updates:

• With the current hardware and default OpnSense settings (freshly installed OpnSense with just WAN and LAN interface), the internet speed is still capped at 300mbps with the default 8 connections on fast.com.
• With Zenarmor on and limiting the processing to a single core, one core reached 70% of usage when I change the number of connections to 16 on fast.com (the speed is 900+mbps). With the default 8 connections, this core is using ~30%, so the single CPU core doesn't seem to be the bottleneck.
• Disabling Zenarmor helped with the inter-VLAN speed which went up to ~10gbps from ~5gbps. However, it doesn't affect the internet speed test result.
• I checked the inter-VLAN speed and router CPU usage w/ and wo Zenarmor enabled. When Zenarmor is disabled, the cores (threads) are used evenly (~30% per core when reaching the max speed of 10gbps). When Zenarmor is on, one core reaches 100% and rest is close to 0%. I guess multi-core on Zenarmor is not supported yet (probably will be there later this year? https://www.zenarmor.com/roadmap). To run this test, I enabled rss and isr thread binding.
• The Zenarmor setting, "Do not pin engine packet processors to dedicate CPU cores" (in Zenarmor
  -> settings -> configuration), helped to increase the VLAN speed a little bit (from 5gbps to 7gbps). It's still heavily using one core, but the other cores see some usage.

The original problem is still not resolved yet unfortunately, but I'm learning and hopefully will find the answer one day  :-*

Emigrate to South Korea?

One day maybe :)

What are you running that actually saturates 900 Mbps?

Not much, only occasionally when I download some LLM models which opens up enough connections to saturate the bandwidth. I just want to learn if there's any setting I'm missing. The internet speed test result makes me sad a little bit (and I can't show off to my wife :P)

Hello everyone, I'm new to OpnSense and have only used it since a month ago. I like it so far thanks to the friendly UI and the comprehensive features. I feel there's a lot of to learn from it :)

I installed OpnSense directly on a mini-PC with 14 cores CPU and plenty of RAM (probably overkill I know), and enabled Zenarmor on the LAN/vLAN interfaces. The inter vLAN speed is ~5gbits/second using iPerf which works for me (and probably have room to tune), but the internet speed is slower than I expected. With the default settings of either speedtest.net or fast.com, the download speed is around 400mbits/s while the upload speeds is 900+mbits/s. After I change the "maximum connection" setting on fast.com from 8 (default value) to 16 connections, I get the expected download speed of 940mbits/s.

Does anyone know how to increase the speed per connection? Is it because my CPU's single core performance is not as good (however the CPU usage is quite low on any core), or there's some settings I set incorrectly? This issue doesn't happen when I use a different router or connect the internet to my PC directly. Thanks for the help in advance!