OPNsense Forum

English Forums => Hardware and Performance => Topic started by: ybalbert on June 10, 2024, 05:08:10 PM

Title: How to improve internet speed per connection
Post by: ybalbert on June 10, 2024, 05:08:10 PM
Hello everyone, I'm new to OpnSense and have only used it since a month ago. I like it so far thanks to the friendly UI and the comprehensive features. I feel there's a lot of to learn from it :)

I installed OpnSense directly on a mini-PC with 14 cores CPU and plenty of RAM (probably overkill I know), and enabled Zenarmor on the LAN/vLAN interfaces. The inter vLAN speed is ~5gbits/second using iPerf which works for me (and probably have room to tune), but the internet speed is slower than I expected. With the default settings of either speedtest.net or fast.com, the download speed is around 400mbits/s while the upload speeds is 900+mbits/s. After I change the "maximum connection" setting on fast.com from 8 (default value) to 16 connections, I get the expected download speed of 940mbits/s.

Does anyone know how to increase the speed per connection? Is it because my CPU's single core performance is not as good (however the CPU usage is quite low on any core), or there's some settings I set incorrectly? This issue doesn't happen when I use a different router or connect the internet to my PC directly. Thanks for the help in advance!
Title: Re: How to improve internet speed per connection
Post by: bartjsmit on June 10, 2024, 06:41:38 PM
Quote from: ybalbert on June 10, 2024, 05:08:10 PM
Does anyone know how to increase the speed per connection?
Emigrate to South Korea?  :) What are you running that actually saturates 900 Mbps? Just curious.
Title: Re: How to improve internet speed per connection
Post by: ybalbert on June 11, 2024, 12:03:30 AM
QuoteEmigrate to South Korea?
One day maybe :)

QuoteWhat are you running that actually saturates 900 Mbps?
Not much, only occasionally when I download some LLM models which opens up enough connections to saturate the bandwidth. I just want to learn if there's any setting I'm missing. The internet speed test result makes me sad a little bit (and I can't show off to my wife :P)
Title: Re: How to improve internet speed per connection
Post by: bartjsmit on June 11, 2024, 08:13:21 AM
Better reason than most  - the learning, not the bragging 8)

Pare down the config to just NAT and test again. Tweak the NIC (in OPNsense) / CPU / RAM (in BIOS) settings to see if any increase speed. Once you're maxed out, add features that you need and then features that you want. Note where you see a drop and decide if the feature is worth it.

Bart...
Title: Re: How to improve internet speed per connection
Post by: Greg_E on June 11, 2024, 03:46:25 PM
I would disable ZenArmor and try again, that might be the issue combined with single core speed (as mentioned).

Are you also running IDS/IPS (Suricata)? That might be slowing things down a bit too.

I get a similar slowdown (fast.com) with both running on my AMD V1756b with 16gb of ram and a known good gigabit connection. I'm guessing this is mostly ZenArmor and it's single threaded operation. I think I tried with Zenarmor disable and got a much higher test, but that was months ago and I don't really remember.

I just tested 620 down and 770 up with IDS/IPS on the wan and Zenarmor on the lan and crowdsec somewhere in the middle.
Title: Re: How to improve internet speed per connection
Post by: ybalbert on June 12, 2024, 06:04:41 PM
QuoteAre you also running IDS/IPS (Suricata)? That might be slowing things down a bit too.
No, I'm not using IDS/IPS right now. I enabled it on WAN for a while but it didn't report any alert, so I disabled it to focus on troubleshooting other parts of the system.

Just played around with the settings a bit more. Some updates:

The original problem is still not resolved yet unfortunately, but I'm learning and hopefully will find the answer one day  :-*
Title: Re: How to improve internet speed per connection
Post by: Greg_E on June 13, 2024, 03:50:59 PM
I wonder why your system is performing so much differently from mine. Mine is very vanilla on 24.1.8 CE (still haven't had the time to build up my business licensed machine). I may change that "do not pin" setting in Zenarmor to see if speed between networks increases, but I'm mostly still getting really good speeds between them (all gigabit and pretty close to max speeds).

Do you have all of the offloading in the NICs turned off? That was kind of required with IDS/IPS and to an extent Zenarmor.

And yes I'm looking forward to the Zenarmor multicore update, many of us are looking forward to this. More processor should mean more throughput.
Title: Re: How to improve internet speed per connection
Post by: ybalbert on June 14, 2024, 04:36:46 PM
QuoteDo you have all of the offloading in the NICs turned off?
I disabled the three hardware offload settings.

I don't see anything obviously wrongly with the OpnSense configs. I have even tried to install pfSense on the same hardware to benchmark, and the result was the same. So I guess it's a mostly likely a hardware thing or a general FreeBSD limit. Unfortunately I don't have another box to test with. Will probably leave it like this for now.