traffic shaping, especially fq_codel and WFQ

[bimbar]

So I use fq_codel because it's a huge plus in general for the usability of internet connections under load.

It still became important to reduce the priority of a specific flow in relation to the others (it's a backup job to the cloud and takes up all bandwidth).

Apparently with fq_codel that's not possible since it ignores the weights chosen in the queue.
If WFQ is used in the pipe, prioritization works as expected, but the queue management is gone - the "Enable codel" checkboxes seem to do almost nothing - or rather, the codel part without the fq is not hugely beneficial.

Cake seems to solve this, but it's not available on dummynet.

Any other ideas?

[mimugmail]

CoDel on the Pipe or the Queue?

[bimbar]

pipe so far.

But maybe I should do queue with mask and then do codel on the queue?

[mimugmail]

Maybe worth a try

[meyergru]

Just an idea: I had SQM with fq_codel running under 21.7, but less than impressive results, until I stumbled over this guide:

https://maltechx.de/en/2021/03/opnsense-setup-traffic-shaping-and-reduce-bufferbloat/

After doing it like that, I have an A+ rating on Dslreports and +0ms on Waveform bufferbloat tests.

[Animosity]

I use 3 queues for upload/download and rules to drop things into queue.

I use weights on each of the queues 100/50/1 respectively and works like a champ.

I have a high / default. / low queue and for me, I drop all my 'backup' type traffic in the low queue and never hit any issues.

I thought the weighting with FQ_Codel was more about packets it sent rather than priority. I recall reading something about that but the docs are pretty spotty.

My queues look like:

https://imgur.com/gallery/HQDLF77

Rules are like this as a filter out a specific IP to my low queue. Rules should be top down with the most restrictive first as I want my low matches than highs with a catch all default at the end.

https://imgur.com/a/HjMhuxn

I have a gibabit FIOS link and always get A+ on waveform regardless of load on my line or what's happening.

[bimbar]

The problem is, if you use fq_codel on the pipe, what you do in queues largely doesn't matter anymore.

So if you want to do fq_codel and prioritize something as well, you're out of luck.

Unfortunately fq_codel without priorities does run much better than WFQ with priorities.

[Animosity]

I was going by this:

https://docs.opnsense.org/manual/how-tos/shaper_prioritize_using_queues.html

and this:

https://docs.opnsense.org/manual/shaping.html

queue
A queue is an abstraction used to implement the WF2Q+ (Worstcase Fair Weighted Fair Queueing) policy, which is an efficient variant of the WFQ policy. The queue associates a weight and a reference pipe to each flow, and then all backlogged (i.e., with packets queued) flows linked to the same pipe share the pipe’s bandwidth proportionally to their weights. Note that weights are not priorities; a flow with a lower weight is still guaranteed to get its fraction of the bandwidth even if a flow with a higher weight is permanently backlogged.

In my testing, if I max out a low queue, the other ones suffer no loss and work and I can see traffic flowing through my high/default queues.

It's very easy to test with iperf and using a particular IP for rules.

Example of my rules/test with iperf3.

https://imgur.com/a/oeufGnN

[bimbar]

I was going by this:

https://docs.opnsense.org/manual/how-tos/shaper_prioritize_using_queues.html

and this:

https://docs.opnsense.org/manual/shaping.html

queue
A queue is an abstraction used to implement the WF2Q+ (Worstcase Fair Weighted Fair Queueing) policy, which is an efficient variant of the WFQ policy. The queue associates a weight and a reference pipe to each flow, and then all backlogged (i.e., with packets queued) flows linked to the same pipe share the pipe’s bandwidth proportionally to their weights. Note that weights are not priorities; a flow with a lower weight is still guaranteed to get its fraction of the bandwidth even if a flow with a higher weight is permanently backlogged.

In my testing, if I max out a low queue, the other ones suffer no loss and work and I can see traffic flowing through my high/default queues.

It's very easy to test with iperf and using a particular IP for rules.

Example of my rules/test with iperf3.

https://imgur.com/a/oeufGnN

The problem with fq_codel is that it doesn't support weights, so your weights in the queues that feed into the same fq_codel pipe don't have any effect.
It tested that with 2 queues, one with weight 100, the other with weight 1, and they shared their bandwidth 50% / 50% if the pipe was configured with fq_codel. It just doesn't support weights.

[senser]

@bimbar maybe you can somehow tell the backup tool to use a specified maximum amount of bandwidth (throttle upload bandwidth) instead. Just an idea.

[Animosity]

Thanks for sticking with this.

I think I found a pretty decent explanation and have a better understanding why weights do nothing and I had a placebo effect as my queues/rules were working but not because of the weights.

This article had a very good explanation:

https://community.ui.com/questions/If-youre-looking-to-better-understand-what-fqcodel-HTB-and-BQL-is-/edbeb291-83d8-45e4-953f-c0d13ec5689f

So basically, if you are defaults, you get 1024 flows that are 'fairly' used so based on the concept of any FQ_xxxx active scheduled, it fairly distributes it so you can't prioritize something as it's contrary to the fair part.

So if you have 2 large apps, they'll share bandwidth and other flows also get bandwidth and ensure they work.

I simplified my setup and just made 1 queue/1 in rule / 1 out rule since it doesn't matter.

I tested with iperf3 and validated that everything is shared as you described which makes total sense now.

So you'd have to use a different scheduler if you wanted true priority as FQ_Codel and FQ_PIE will split amongst their flows.

[bimbar]

Cake can do that, but cake is not available in FreeBSD so far.

[Animosity]

Yeah, I don't think that's coming to FreeBSD anytime soon unfortunately.

I tried a few different things and nothing seems to work with any weighting that really helps with bloat as well which is unfortunate.

IPFire is testing out Cake currently but that just doesn't work for me as it is too unstable a platform.

My use case, I'm fine with shared bandwidth as it doesn't really matter to me but would be nice to get prioritized bandwidth as that's what I thought I was getting..