Slow Download Speed

[Lruts]

OPNsense 24.1.1-amd64 connected directly (no modem/router) hosted on a Protectli Vault FW6A.
There is no special configuration besides DHCP on IPV4 and IPV6 disable.
I have 1 GB service with Frontier, but tests ran by Speedtest plugin are showing downloads of 350 Mbps and uploads of 900 Mbps consistently.
What is weird is that every time I restart the device, the first two or three tests will show downloads of 900 MBps but after 30 seconds it will go to an average 350sh Mbps, the uploads will remain above 900 Mbps.
    Average Download speed:    429.98 Mbps (min: 239.03 Mbps, max: 940.48 Mbps)
    Average Upload speed:   917.12 Mbps (min: 854.17 Mbps, max: 940.77 Mbps)
    Average Latency:       6.99 ms (min: 6.07 ms, max: 7.42 ms)


Any pointers will be greatly appreciated.

[johnmcallister]

Do you have a baseline of earlier tests done on the same hardware & same connection, using older versions of OPNsense?

If yes, consider summarizing those tests here.
If no, try using a series of other speed testing providers for comparison. (Fast.com is one example, there are several others.)

It's possible that Frontier provides 'burstable' 900 Mbps downstream but after repeated bursts from a single host or within a certain time-frame they rate-limit your downstream connection temporarily.

[Lruts]

Unfortunately, I don't have a history of tests, I just started today after several days of lagging.
It doesn't seem to me that Frontier is limiting the downloads, as I still get 350 Mbps after waiting for an hour, but as soon as I reboot Opnsense it goes to 900 for a short time.
I am suspecting that something is getting loaded that is limiting to around 375 Mbps as I can't go beyond that on all further tests.

[johnmcallister]

I would perhaps approach this by using a separate local bare-metal device, connected directly or through a reliable and otherwise-unloaded GB or 10GB-capable switch, to test Opnsense's max bandwidth. (Using iperf & similar tools.)

You might be able to reproduce the issue with your local test setup, which would rule out anything to do with the ISP.

Or, if you can't reproduce it locally that might tend to point back towards something specific to the Frontier connection.

[Allistaire]

Unfortunately, I don't have a history of tests, I just started today after several days of lagging.
It doesn't seem to me that Frontier is limiting the downloads, as I still get 350 Mbps after waiting for an hour, but as soon as I reboot Opnsense it goes to 900 for a short time.
I am suspecting that something is getting loaded that is limiting to around 375 Mbps as I can't go beyond that on all further tests.

After Upgrading to 24.1 it was fine for a while, then I noticed intermittent speed problems. Especially on Download speed, but not limited to it.  When running speed test My download speed would be 30Mb but the OpnSense router would show 120Mb (speed of my line), I eventually downgraded to 23.7 and the problem is gone.

[Lruts]

After some testing, I found that the IPS mode (under Services: Intrusion Detection: Administration) was the one causing the problem.
Once disabled the test results (download/upload) are consistent (900 Mbps)

[sektor]

Thank you Lruts , early morning here , was driving myself crazy , got the same numbers stuck on 200 300 Mbps , now I just disabled the IDS and booooooooooom 1 Gbps again

Thank youuuuuuuuuuuuu

[Melroy vd Berg]

I would still like to understand why there is such a big performance impact (I also see a massive reduction in speeds) when enabling IPS Mode.

The reason that I find it strange is that when I only enable  Intrusion Detection (so just IDS), it's very fast. But the system is still checking all the packages, after all it will alert on find matches.

And IPS Mode is basically doing all of the same, but then it will also drop the connection on the rules instead of only an alert. So why is IPS mode so much worse performance wise? I still do not get it.

[meyergru]

IDS mode can inspect packets after the fact and only generate an alarm. At this time, the packet was already processed. IPS mode has to actually check all the rules before it will decide on whether to actually allow the packet to pass.

[Melroy vd Berg]

IDS mode can inspect packets after the fact and only generate an alarm. At this time, the packet was already processed. IPS mode has to actually check all the rules before it will decide on whether to actually allow the packet to pass.

Now I think about this, this makes of course a lot of sense! So IDS mode is just written differently. And indeed IPS needs to be blocking the traffic if it does find something, so it can not do any post-processing after the fact. Thanks!

[Melroy vd Berg]

@meyergru Can I ask one final question?

I notice that IDS/IPS can optionally also include block lists like Spamhaus, right? However, I notice some people will use maybe a firewall alias for Spamhaus and block the traffic under firewall rules..

So my question would be: Would it be more performant (keep higher throughput) if some of these checks like Spamhaus will be done under the firewall settings rather than under Instruction Detection? Since for some reason, I have the feeling that Instruction Detection is much more demanding than just a firewall block, while again a simple block list like Spamhaus doesn't necessary need to be part of IDS/IDS, Spamhaus (and alike) can be part of a block list...?

I hope my story/question is clear.

[meyergru]

I think (i.e. not know) that when IDS has spotted an IP to do something weird, it will put it on an internal blocklist, which could operate just as efficiently as any other blocklist, but IDK the details of it.

When you have very large blocklists, it depends on how you implement the matching algorithm on how fast it can get. Most surely, a simple sequential scan won't cut it for larger lists - I would make use of some kind of hash tables. It might be the case that one or the other implementation is better or worse at that, also depending on the actual (and expected) size of the blocklist in question.

But then again, IDK how that is implemented in detail.