IPv6 Control Plane with FQ_CoDel Shaping

[Seimus]

I think maybe these:

https://forum.opnsense.org/index.php?topic=43856.0
https://forum.opnsense.org/index.php?topic=45135.0

It'll take some time to play but if I find a way to improve the Household test score I will post back.

Nice you found it!

I should maybe again reconsider, to test and maybe write about Weighted scheduler + Latency Queue management. I previously did think about it, but it does require more configuration and could confuse the general user.

Regards,
S.

[OPNenthu]

I think the CoDel options outside of the FQ_CoDel scheduler are not effective, based on some preliminary testing.

I tried two things:

1)
Pipes set to WFQ
Queues set to CoDel + ECN
Result: bufferbloat score went way down (B-C range, +80ms on upload latency)

2)
Pipes set to WFQ + CoDel + ECN
Queues set plain (no options)
Result: same as above

There seems to be something special about the FQ_CoDel scheduler on pipes that makes them effective for Bufferbloat management which other CoDel options do not have.

If that's the case, then it won't be practical to classify and prioritize traffic types as the latency won't be acceptable.

The only way might be to use separate pipes with FQ_CoDel but I don't want to carve up my bandwidth.

Are you seeing the same?

[Seimus]

2)
Pipes set to WFQ + CoDel + ECN
Queues set plain (no options)
Result: same as above

This actually will not use the CoDel in the Queues (tab). If you enable CoDel in the Pipe you need to attach your rules directly to the Scheduler instead of Queues you manually Created. Because CoDel in Pipes is configured on the dynamic created Queues. When you attach your rules to the manually created Queues, they use FiFO qdisc.

There seems to be something special about the FQ_CoDel scheduler on pipes that makes them effective for Bufferbloat management which other CoDel options do not have.

If that's the case, then it won't be practical to classify and prioritize traffic types as the latency won't be acceptable.

The only way might be to use separate pipes with FQ_CoDel but I don't want to carve up my bandwidth.

The magic most likely is due to the FQ scheduler, + the fact FQ_C allows to set quantum, it does not create such overhead as WFQ or QFQ. FQ_C is a fine tuned version of CoDel. CoDel itself because is just a qdisc needs a proper scheduler. Its usually advised to use CoDel with QFQ which performs better.

https://www.bufferbloat.net/projects/codel/wiki/

Are you seeing the same?

Honestly I did not had the chance to test this yet :)

Regards,
S.

[OPNenthu]

That makes sense regarding #2, @Seimus.  Thanks.

I had tried #1 with QFQ instead of WFQ but it was no difference.

[Seimus]

How are you testing #1
Do you have multiple Queues each with different Weights?
Can you try to set all the Queues with same Weight?
Can you try to run only one any/any Queue per direction?

Regards,
S.

[OPNenthu]

(Post #1 of 4: Test 1)

In this post I've kept my existing setup with the separate control plane for ICMP/ICMPv6, but I've changed the pipes on the data plane to use QFQ.  The queues are as before, except now they are using CoDel+ECN at the queue level.  Queue weights within the data plane are all 100.

Setup images attached.

The router was rebooted to ensure a clean pick-up of the new configs.

[OPNenthu]

(Post #2 of 4: Test 1 Results)

Results images attached.

[OPNenthu]

(Post #3 of 4: Test 2)

In this post I've removed the existing data queues/rules and instead used a single queue and rule (any/any) per each direction.

There is a problem observed now that I am getting a lot of messages like this in the console:

Code Select Expand

config_aqm Unable to configure flowset, flowset busy!

I rebooted the router twice and I also completely cleared the shaper configs and started over with just an upload pipe and a download pipe (no control plane stuff).  The messages did not go away.

There seems to be a bug.

[OPNenthu]

(Post #4 of 4: Test 2 Results)

Results images attached.

Also, the console showing the flood of 'flowset busy' messages mentioned in the previous post.

[Seimus]

That error looks like is due to

https://github.com/opnsense/core/issues/1279#issuecomment-3417927175

That error tells you that you are trying to reconfigure a flowset (pipe or queue) while there is an actual traffic uses that flowset (it has an active scheduler).
However, there is an easy workaround to avoid this error. If you make sure that there's no traffic passes through the pipe/queue that you want to reconfigure, then you can reconfigure the pipe/queue without problems.

I would advice to disable the rules, and than try to reapply the settings.

Regards,
S.

[Seimus]

Honestly considering, you run Weighted scheduler with a CoDel on Queues these still good results even thought not the desired.
Basically, CoDel isn't aware on any BW target, so its managing the latency only based per packet sojourn time.

The Scheduler set to QFQ with a Weighted Queue with No MASK, has only the BW Available per the weight ratio per Queue. But here is the main point, its just a Queue without any flow recognition. So basically 1st flow comes in 1st flow gets out & the BW. Once full 50 packets size Queue, anything that will come after TAIL Drops.

For CoDel in Queue you can adjust two parameters TARGET & INTERVAL, the defaults should be enough but maybe you can try to tune them. You can focus on the Upload and set it so CoDel will more aggressively drop or ECN flag packets. Potentially as well try to disable ECN, as instead of flagging first and dropping then ECN supported flows it will drop them right away.

Regards,
S.

[OPNenthu]

I appreciate the feedback...

That error tells you that you are trying to reconfigure a flowset (pipe or queue) while there is an actual traffic uses that flowset (it has an active scheduler).
However, there is an easy workaround to avoid this error. If you make sure that there's no traffic passes through the pipe/queue that you want to reconfigure, then you can reconfigure the pipe/queue without problems.

I would advice to disable the rules, and than try to reapply the settings.

Hmm, but, as I mentioned it persisted across reboots.  I can set it up again to make sure the rules were disabled, but if it happened on bootup then I'm not confident it won't get into that state again on the next router reboot.  The messages come early, even before OPNsense is fully booted.

The Scheduler set to QFQ with a Weighted Queue with No MASK, has only the BW Available per the weight ratio per Queue. But here is the main point, its just a Queue without any flow recognition. So basically 1st flow comes in 1st flow gets out & the BW. Once full 50 packets size Queue, anything that will come after TAIL Drops.

For CoDel in Queue you can adjust two parameters TARGET & INTERVAL, the defaults should be enough but maybe you can try to tune them. You can focus on the Upload and set it so CoDel will more aggressively drop or ECN flag packets. Potentially as well try to disable ECN, as instead of flagging first and dropping then ECN supported flows it will drop them right away.

Disabling ECN didn't seem to have a noticeable effect though I didn't try TARGET & INTERVAL.  I think I've accepted that FQ_CoDel itself is good enough and not worth the trouble to try and prioritize further (you were right, it's difficult) :)

I'm running with the original setup now (FQ_CoDel data pipes + QFQ control pipes for ICMP only) and for whatever reason the Waveform test is not stalling.  I still don't know if the test itself is sometimes faulty or if it was something on my end that got cleared, but I'm keeping an eye on it.

I gave up on trying to get an 'A' on the LibreQoS Houshold test.  I had seen a couple F's before, but mostly it's giving me a B.  I think @meyergru is seeing 'B' as well, and I should not presume that I can beat him :)

[Seimus]

At the end all what matters is how the real live experience feels.

Even if you get in an artificial benchmark/test worse or unwanted score, at the end its just an artificial test. For example, I am getting on waveform and cloudflare A+, but on libre C or D. But in reality, my latency across services in congestion state is fantastic and more close to the results of waveform.

Honestly I am not sure if the libre test is working properly or what the drill with it is. Or even how exactly it works. But it doesn't bother me much as in real life all works as it should.

Regards,
S.