Crazy High Latency While Other Clients Are Downloading..

[twoflekc]

After updating to 14.1 I've begun noticing crazy high latency while my other clients are downloading files. Even running a speed test on my phone over wifi will cause the download and upload latency/jitter to go from 3-4
ms unloaded to over 30ms. Not too sure what to look at as my system doesn't seem taxed at all. As strange as it may sound my biggest telltale sign of this is while playing competitive COD - I go from 10ms to over 150ms when something is being downloaded by another client.

Setup:
Firewall:  N100 DDR5 version. 8GB 4800mhz Samsung RAM. 250GB Samsung 980 pro https://www.aliexpress.us/item/3256804173757529.html

2.5gb switch connected with ethernet to the firewall - Have 2 gaming PCs and unraid server connected to this with ethernet. https://a.co/d/6OAOe3A

Ubiquiti US-48 connected with 10gb SFP to the switch above https://a.co/d/13G9670

Google Fiber ISP

Appreciate any suggestions :)

[IdleBit]

Hi, I have a similar problem since the upgrade to 24.1. As soon as I run a larger download via Steam or PSN, the download rate drops from 1gb/s to about 400mb/s and for the remaining network clients the internet breaks down.   

The hardware does not show high CPU or RAM utilization. I am also a bit at a loss as to the best way to proceed.

My setup:
OPNsense: 24.1.1-amd64 FreeBSD: 13.2-RELEASE-p9 OpenSSL: 3.0.13

Hardware: Low Energy Server LES compact 4L (https://www.thomas-krenn.com/de/produkte/rack-server/technisches-datenblatt/frame.only_content/key.21578.html)

CPU type: Intel(R) Celeron(R) CPU J3160 @ 1.60GHz (4 cores, 4 threads)
RAM: 4GB

Vodafone Cabel ISP

I am grateful for any tips :)

[meyergru]

That sounds like a bufferbloat problem. Some setups show that behaviour, especially when upstream and downstream are asymmetric. There is a good test for this here.

It often shows more when a high BDP is present, such when the other side of a connection has a high latency.

I am at a loss as to why this should happen with 24.1, but probably something has changed in the network stack - I cannot tell since I have not updated yet. Sometimes the bottleneck is outside of OpnSense itself, like when the modem or ONT has too small (or too big) buffers.
You can (sometimes) fix this by using a traffic shaper that limits your up- and/or downstream to what your ISP delivers in order not to overwhelm the connection.

[twoflekc]

Hi, I have a similar problem since the upgrade to 24.1. As soon as I run a larger download via Steam or PSN, the download rate drops from 1gb/s to about 400mb/s and for the remaining network clients the internet breaks down.   

The hardware does not show high CPU or RAM utilization. I am also a bit at a loss as to the best way to proceed.

My setup:
OPNsense: 24.1.1-amd64 FreeBSD: 13.2-RELEASE-p9 OpenSSL: 3.0.13

Hardware: Low Energy Server LES compact 4L (https://www.thomas-krenn.com/de/produkte/rack-server/technisches-datenblatt/frame.only_content/key.21578.html)

CPU type: Intel(R) Celeron(R) CPU J3160 @ 1.60GHz (4 cores, 4 threads)
RAM: 4GB

Vodafone Cabel ISP

I am grateful for any tips :)

Glad I'm not the only one! Everything was completely fine before upgrading to 24.1.1.. I've read into having traffic shaping rules in place but would rather not as it was not needed pre-upgrade. Hopefully, someone can comment further on what is causing this. Fingers crossed!

[twoflekc]

That sounds like a bufferbloat problem. Some setups show that behaviour, especially when upstream and downstream are asymmetric. There is a good test for this here.

It often shows more when a high BDP is present, such when the other side of a connection has a high latency.

I am at a loss as to why this should happen with 24.1, but probably something has changed in the network stack - I cannot tell since I have not updated yet. Sometimes the bottleneck is outside of OpnSense itself, like when the modem or ONT has too small (or too big) buffers.
You can (sometimes) fix this by using a traffic shaper that limits your up- and/or downstream to what your ISP delivers in order not to overwhelm the connection.

Yeah, it's for sure a buffer bloat issue.. Just not sure what changed in 24.1 to begin causing this.

Don't believe it to be an issue with the ONT. I have the same one as in the post here https://www.dslreports.com/forum/r33627113-

[opnfwb]

Fellow google fiber user here but I'm on the 500/500 service. Also running 24.1.1 and I haven't noticed an increase in bufferbloat issues.

This test was done via a wired 1gb client just now: https://www.waveform.com/tools/bufferbloat?test-id=44cce432-825a-4677-85c6-05f9d5eba2cb

My OPNsense instance does not currently have any limiters or queue management enabled. I have used it in the past, also with good results but found it wasn't needed to get A scores in the tests.

[twoflekc]

Lucky! What ONT do you have?

[emmitt]

Same here...  :(

My setup:
OPNsense: 24.1.1-amd64 FreeBSD: 13.2-RELEASE-p9 OpenSSL: 3.0.13

Hardware: apu2 Board

CPU type: AMD GX-412TC SOC (4 cores, 4 threads)
RAM: 4GB

GlassFibre

With 23.7 no issues. With 24.1:

[IdleBit]

That sounds like a bufferbloat problem. Some setups show that behaviour, especially when upstream and downstream are asymmetric. There is a good test for this here.

It often shows more when a high BDP is present, such when the other side of a connection has a high latency.

I am at a loss as to why this should happen with 24.1, but probably something has changed in the network stack - I cannot tell since I have not updated yet. Sometimes the bottleneck is outside of OpnSense itself, like when the modem or ONT has too small (or too big) buffers. You can (sometimes) fix this by using a traffic shaper that limits your up- and/or downstream to what your ISP delivers in order not to overwhelm the connection.

Thanks for your tip, but unfortunately the settings have not led to any improvement for me.
https://www.waveform.com/tools/bufferbloat?test-id=641f82dc-3a99-45ce-9bdc-042fc3cd1981

[opnfwb]

Lucky! What ONT do you have?

Mine is an older install, I believe its a GFLT110.

Router hardware is a passively cooled J3455 with Dual Intel IGB NICs

[opnfwb]

For bufferbloat and using FQ Codel, this site was very helpful for me. Not my site but I followed these steps exactly to get the limiters and queue managers setup.

https://maltechx.de/en/2021/03/opnsense-setup-traffic-shaping-and-reduce-bufferbloat/

[Seimus]

That latency during DL/UL does it happen when your connection is maxed out?

If yes its 99,99% buffer bloat related stuff. As advised by @opnfwb try to configure shaper with fq_codel. At 500Mbit connections usually you don't need to even tune it out. But as I tested myself sometimes tuning is needed depending from Telco to Telco.

I had two Telcos for asymmetric 500/30.
On one of them fq_codel run with default settings.
The other one I had to tune fq_codel.



Regards,
S.

[twoflekc]

Setting up the traffic shaper fixed my issue.. but I lost around 300mbps of throughput. Had to keep turning it down to get the result in the screenshot.

Yes, it fixed it. Is this still an issue? Yes. I had none of these issues on 23.7

[axsdenied]

Setting up the traffic shaper fixed my issue.. but I lost around 300mbps of throughput. Had to keep turning it down to get the result in the screenshot.

Yes, it fixed it. Is this still an issue? Yes. I had none of these issues on 23.7

Roll it back and test?  No way to know if this is causation or correlation without more data than "I just updated".

[axsdenied]

What was the results prior to upgrade?

Same here...  :(

My setup:
OPNsense: 24.1.1-amd64 FreeBSD: 13.2-RELEASE-p9 OpenSSL: 3.0.13

Hardware: apu2 Board

CPU type: AMD GX-412TC SOC (4 cores, 4 threads)
RAM: 4GB

GlassFibre

With 23.7 no issues. With 24.1: