fq_codel console flood

[saulsutherland]

I have no idea why you think a quantum of 2400 is good. (I am one of the authors of the fq_codel algorithm).

I started a project recently to try and collect all the complaints about fq_codel and shaping and see if they were a bug in the implementation or not. It was an outgrowth of the reddit thread here:
https://www.reddit.com/r/opnsense/comments/18z68ec/anyone_who_hasnt_configured_codel_limiters_do_it/

So far the openbsd implementation is proving out ok, but I have not had the time, money or chops, to get someone to rigorously test the freebsd (opnsense version) enough to prove to me that it is indeed implemented correctly. Anyone out there capable of running some serious flent tests? I outlined basic tests
to the openbsd person on the case here: https://marc.info/?l=openbsd-tech&m=170782792023898&w=2

I am perversely happy that a probable source of speed complaints about it is it doing too aggressive logging under stress, as reported on this bug here today.

The freebsd bug is being tracked here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276890

Until this is resolved should I disable my inbound shaping rule?

[Fenuxx]

I think I may have found the cause to this, and that being setting the FQ-CoDel limit on the pipe too low, which, from the looks of it, the message it's spamming is (possibly) the number of packets it's unable to process because the queue is full.

Reading this document (posted elsewhere on the forum) https://datatracker.ietf.org/doc/html/rfc8290 showed me a number of the common recommendations are just, well, wrong.

FQ-CoDel quantum should be set at your WAN MTU (in my case, 1514 bytes)
FQ-CoDel limit doesn't really need messed with, this setting defines the maximum number of packets that CAN be queued.  The default is 10240.  Most recommendations are to drop this value significantly, thus, causing the console flood messages.  For me, on my download pipe, I left this at the default of 10240.  For my upload pipe (with a max speed of 40Mbps) I halved it to 5120.  No real reason to, but my uploads will never saturate it.

Since changing these things, I have zero issues.  I can post my complete config if anyone's interested.
https://www.waveform.com/tools/bufferbloat?test-id=d11e32d1-02df-45d8-b5e9-2628fbbcd78c

[Koldnitz]

The ping command that pops up when you google when you search about finding WAN MTU this is not very useful.

If anyone is wondering how to determine your MTU on OPNsense:

Code Select Expand

sudo ping -D -s 1472 1.1.1.1
"1472" is just right for me (and probably most people not on PPPoe ... but your mileage may vary; this is my hobby).

You need to adjust the number "1472" greater and or less than until you get:

Code Select Expand

sudo ping -D -s 1472 1.1.1.1                                                                                                                                                            ─╯
Password:
PING 1.1.1.1 (1.1.1.1): 1472 data bytes
1480 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=8.168 ms
1480 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=7.644 ms
1480 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=8.822 ms
1480 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=8.274 ms
^C
--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 7.644/8.227/8.822/0.418 ms

For Example:

Code Select Expand

sudo ping -D -s 1473 1.1.1.1                                                                                                                                                            ─╯
PING 1.1.1.1 (1.1.1.1): 1473 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
^C
--- 1.1.1.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

"1473" is too high.

Please also note:

Code Select Expand

sudo ping -D -s 1471 1.1.1.1                                                                                                                                                            ─╯
PING 1.1.1.1 (1.1.1.1): 1471 data bytes
1479 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=11.158 ms
1479 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=14.895 ms
1479 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=8.327 ms
1479 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=11.253 ms
^C
--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 8.327/11.408/14.895/2.331 ms

"1471" works but since "1472" works as well this is too low.

I hope this helps someone.

Cheers,

[jrichey98]

Reading this document (posted elsewhere on the forum) https://datatracker.ietf.org/doc/html/rfc8290 showed me a number of the common recommendations are just, well, wrong.

I ran into a lot of that as well when I was first figuring things out.

FQ-CoDel quantum should be set at your WAN MTU (in my case, 1514 bytes)

100% true, don't know why people would want to split packets up.

FQ-CoDel limit doesn't really need messed with, this setting defines the maximum number of packets that CAN be queued.  The default is 10240.  Most recommendations are to drop this value significantly, thus, causing the console flood messages.  For me, on my download pipe, I left this at the default of 10240.  For my upload pipe (with a max speed of 40Mbps) I halved it to 5120.  No real reason to, but my uploads will never saturate it.

Scaling the limit with bandwidth can help with Bufferbloat. I have ATT 500mbps symetric fiber, and the first hop is the converter box that I have bypassed. With a 2ms target and 4ms interval, 1200 limit is best and I never get any of the warnings. With a higher limit, it has a tendency to burst much higher for a longer interval which increases packet latency.

But yeah, the OP's limit is set too low if he's getting those warnings. It needs to be increased.

Recipe for ATT 500mbps (they over provision to about 640mbps):
Pipe: ATT-IN
Bandwidth: 560
Scheduler Type: fq_codel
Target: 2ms (should be much higher for many, ping your upstream gateway to get good numbers).
Interval: 4ms
Quantum: 1514 (Packet Size, see https://man.freebsd.org/cgi/man.cgi?ipfw)
Limit: 1200 (about perfect for 500mbps, 1500 is worse for me)
*only enable ECN on outgoing pipes/queues.

Pipe: ATT-OUT
Bandwidth: 560
Scheduler Type: fq_codel
Target: 2ms (should be much higher for many, ping your upstream gateway to get good numbers).
Interval: 4ms
Quantum: 1514 (Packet Size, see https://man.freebsd.org/cgi/man.cgi?ipfw)
Limit: 1200 (about perfect for 500mbps, 1500 is worse for me)
ECN: Enabled

Results: https://www.waveform.com/tools/bufferbloat?test-id=1ece2bf6-b7a2-4a40-9817-c90db7eafcd1
Best: 13ms+0+0
Any Day: 14ms+1+1/15ms+0+0

[MagikMark]

The only way right now to addressed the flooding:

Quantum: 1514
Limit: 20480
Flow: 65535

This problem is also present in pfSense

[xPliZit_xs]

I think I may have found the cause to this, and that being setting the FQ-CoDel limit on the pipe too low, which, from the looks of it, the message it's spamming is (possibly) the number of packets it's unable to process because the queue is full.

Reading this document (posted elsewhere on the forum) https://datatracker.ietf.org/doc/html/rfc8290 showed me a number of the common recommendations are just, well, wrong.

FQ-CoDel quantum should be set at your WAN MTU (in my case, 1514 bytes)
FQ-CoDel limit doesn't really need messed with, this setting defines the maximum number of packets that CAN be queued.  The default is 10240.  Most recommendations are to drop this value significantly, thus, causing the console flood messages.  For me, on my download pipe, I left this at the default of 10240.  For my upload pipe (with a max speed of 40Mbps) I halved it to 5120.  No real reason to, but my uploads will never saturate it.

Since changing these things, I have zero issues.  I can post my complete config if anyone's interested.
https://www.waveform.com/tools/bufferbloat?test-id=d11e32d1-02df-45d8-b5e9-2628fbbcd78c

i would like to see your pipe configs.
Thank you.

[AhnHEL]

While obviously off the Main Topic but applicable as related as I've been following this thread closely.

I've been using FQ-PIE as Scheduler in my Pipes for my Asymmetric 940/35 Spectrum Cable Internet with great results.

No Quantum or Limit settings or concern over ECN on Inbound or Outbound Queues, and no more Flood messages in Console either.

It just works.  I'm not getting a perfect +0 Download on the Waveform Test but I am consistently getting a +3 Download and +0 Upload with an A+ Score which to me is perfectly acceptable considering what I get with Traffic Shaping off.

The amount of time I've spent trying to fine tune the settings using CoDel with inconsistent results is embarrassing to state.   PIE might not work out for you but I just wanted to share my experiences since I've been plagued with all of the problems and confusion that have been posted in this thread and many others here.

<Edited for spelling and grammar error>

[keropiko]

While obviously off the Main Topic but applicable as related as I've been following this thread closely.

I've been using FQ-PIE as Scheduler in my Pipes for my Asymmetric 940/35 Spectrum Cable Internet with great results.

No Quantum or Limit settings or concern over ECN on Inbound or Outbound Queues, and no more Flood messages in Console either.

It just works.  I'm not getting a perfect +0 Download on the Waveform Test but I am consistently getting a +3 Download and +0 Upload with an A+ Score which to me is perfectly acceptable considering what I get with Traffic Shaping off.

The amount of time I've spent trying to fine tune the settings using CoDel with inconsistent results is embarrassing to state.   PIE might not work out for you but I just wanted to share my experiences since I've been plagued with all of the problems and confusion that have been posted in this thread and many others here.

<Edited for spelling and grammar error>

Hi,

Have you enabled only the scheduler or the "Pie" option too?

What's your config?

[AhnHEL]

I'm just using the Scheduler - FlowQueue-PIE with no other checkboxes.  I could avoid the Queues all together and use the Rules with just the Pipes but for whatever reason this gives less than optimal results.

Attached are pics. 

First the Pipes

[AhnHEL]

The Queues

[AhnHEL]

And the Rules

[xPliZit_xs]

Hi all,

seeing this again with 24.7.12.

[Seimus]

The excessive logging + FQ_C implementation was not yet fixed, the BUG is still open on FreeBSD.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276890

Regards,
S.