IPv6 Control Plane with FQ_CoDel Shaping

[OPNenthu]

To Mr. Täht, who tamed our networks.  🥃

[OPNenthu]

I think I cracked it!  DOCSIS asymmetry was the main problem here and IPv6 was also masking an issue.

The Waveform test is IPv4 only but I didn't realize that before.  The LibreQoS test supports both but it was defaulting to IPv6.  When I toggled my connection to use IPv4 and ran the LibreQoS test I noticed that the IPv4 path was performing significantly worse than the IPv6 path, and this was the first clue as to why the Waveform test was maybe stalling.  So for the remainder of the exercise I focused on shaping IPv4 first, then made sure my adjustments carried over to IPv6 (and they did).

The next thing I noticed, because I happen to have separate queues for TCP ACKs, is that >98% of the packets on the upload pipe during the tests were ACKs:

You cannot view this attachment.

I never really paid attention before but it hit me: my upload pipe can't keep up with the size of the download pipe and it's causing ACK congestion on the upstream.  That's why no matter how much I tweaked the upload pipe it made no difference, because I only have a very small upload in comparison to the download and the tweaks made only marginal differences.

So the fix became clear- I needed to give up a lot of download bandwidth.  I played around and found that 600Mbps was the sweet spot to balance out my upload.  My current plan is 1000/35 (advertised) and measures 1200/40 in practice due to over-provisioning.  That's a 28-30x difference.

Maybe this can help some others with significant bandwidth asymmetries.  Tuning for anti-bufferbloat isn't only about the individual pipes, it's also about the ratio of bandwidths.  The pros already know; this is just an enthusiast having an "aha!" moment ;)

[meyergru]

You are correct about that there is a certain relation between up- and downstream that must be met in order to allow traffic at all. That is because the ACK stream takes up upstream bandwidth.

However, I measured during the downstream part of the Waveform test and got these results:

You cannot view this attachment.

This shows 4 GByte downstream data and ~130 MByte Upstream, of which 80% was TCP ACKs, so roughly a 3.25% of the downstream needed for upstream. AFAIR, that is about to be expected at a theoretical worst case of ~4% and a more practical 2% (RFC 1122).
AFAIK, that should also explain your rate of 1000/35 Mbps: Your ISP wants you to have full 1000 Mbps downstream, but only the mere neccessity for the upstream with nothing left for server applications. There are some more providers which offer only a small downstream even if there is no technical neccessity to do so, like with DOCSIS.

So, in theory, you should be able to use the full 1000 Mbps downstream, not only 600?

I can imagine two things that may shift the results:

1. With TCP ACKs, you can have pure ACKs and SACKs, so the number of packets used can be severly lower than the number of data packets. That is obviously the case in my test. You did not show the downstream part of your test, you we cannot know if SACK was used, which would be dependend on the client.

2. Regardless of the net data being transferred, pure ACK packets are way shorter than data packets, so they incur a larger overhead, so the net data results may not mirror the real bandwitdhs used.

[OPNenthu]

Here is the full image of that cropped one in my post.  This was before I reduced the download pipe, so at this point I was experiencing stalls.  Note that even setting the pipe a bit lower, for example 850Mbps, did not resolve the stalling.

You cannot view this attachment.

My plan wasn't originally 1000/35.  It was something like 800/35, IIRC, but while on a support call a couple months ago the agent offered me a free increase to "gigabit."  Unfortunately they only increased the downstream. 

I was thinking that the original plan was better balanced overall, though what you are saying is that it doesn't matter.  All I need is that my upstream should be 2-4% of the downstream to maintain stability.

I'm not sure what my numbers reveal but following your formula, it would appear that the theoretical maximum of 4% is being exceeded by at least a factor of 2x.

[OPNenthu]

Unfortunately today I ran into the Waveform test stalls again, so was a little premature.  This happened despite the other tests giving me A-A+ results and happened on both browsers, Firefox and Chromium.

Either I haven't fully resolved some bottleneck, or what @dinguz said about the Waveform servers being overloaded could be true.

I'm thinking it's likely on my side because sometimes the Cloudflare speedtest also stalls mid test, though less frequent than Waveform.  Those are the only two that ever stall.  I never have any issue with speedtest.net, fast.com, and so far none with LibreQoS.  So which do I trust?

I'll keep digging.

Note: I also set a technician appointment to inspect the exterior lines because the cable TV boxes aren't working and they couldn't measure a good signal from their end, but whatever it is doesn't seem to affect the internet.  Maybe there's a fluctuation that is impacting in a way that I can't directly measure.  However, then I would expect to see packet loss on my gateways.

There is some- I have momentary spikes of loss, but those have always been there.

The only "clean" graphs I have seen are the ones with the white theme in the attachments.  Those are from the firewall at my parents' house and they have FTTH, so altogether a different medium.  The spikes are relatively rare there.  I am surprised I caught one at all as usually both graphs are loss-free... could just be an upstream router hiccup as I am setting the monitor IP to a first or second hop on the ISP path.

[Seimus]

In regards of waveform bufferblaost test,

Do you by chance use ZenArmor maybe? I have seen stalls as well but caused by ZA, as ZA started to block some of the the IPs behind which waveform is hosted. They btw moved the hosting, and since that time ZA started to block it.

Regards,
S.

[OPNenthu]

@Seimus No, I only use the firewall with static IP blocklists (Spamhaus, FireHOL, ...) for outbound filtering and DNSBLs.   I'm blocking DoH/DoQ as well.

[OPNenthu]

So, in theory, you should be able to use the full 1000 Mbps downstream, not only 600?

I've taken this advice and increased the d/l pipe again, but also made a few changes to my shaping strategy that seem to have helped.  I'll give it some time before declaring victory but results so far seem positive and I got Waveform working again for the moment.  Browsing and streaming feel more responsive.

My thinking right now is that the separate control pipes with QFQ scheduling were putting too much pressure on my limited upstream.  It also made it difficult to split the bandwidth between the FQ_CoDel and QFQ pipes for the upstream because of the very narrow margins.  I still like the idea of separate control vs. data however.

So I made these changes:

1) Consolidated my pipes to just two with FQ_CoDel (no QFQ)
2) Consolidated my control plane to two queues (up/down), but kept the separate rules to classify ACK/ICMP/DNS.
3) Gave a 5:1 weight advantage to the control queues

You cannot view this attachment.

The weighting helped to smooth out a momentary latency spike when the bufferbloat test trasnitions from D/L to U/L.  The two attached tests illustrate the difference.  (Note: I have active background downloads and video streams during the tests in order to see how it behaves under network load, so latencies are higher than when I test idle.)

I think I'm happy to trade a perfect A+ with lowest possible latencies in order to get an A with more stability.  Maybe this is a fact of life with DOCSIS.

[Seimus]

The weight doesn't do anything when using FQ_CoDel. The FQ in the FQ_CoDel does what it means, fair queuing.

So basically FQ_CoDel, doesn't do any priority queuing nor weighted queuing

In theory even this design should not cause to much harm for control plane, as FQ_CoDel will not let starve any flow under the condition there is not an excessive amount of flows. It can however create extra DROP and TAIL DROP for the control plane e.g create AQM back-pressure.

What is interesting here is that when using FQ_CoDel and not multiples schedulers your tests show good results. So the question is, how and why two different scheduler for two different traffic planes impacted the results? Considering the Control plane if configured properly should not carry over any data plane e.g test traffic for the test.

Regards,
S.

[OPNenthu]

So basically FQ_CoDel, doesn't do any priority queuing nor weighted queuing

Yeah, I remember this from our conversations but I had convinced myself that it fixed that spike.  Most likely those spikes were just transient and it's only coincidence that they disappeared.

So the question is, how and why two different scheduler for two different traffic planes impacted the results?

I went ahead and reset back to the layout with the separate control pipes.  For the moment the control plane is back to QFQ.

If I notice another stall then I can try to find an appropriate control pipe width for the upstream that fixes it.  Having failed that, I can try WFQ instead of QFQ at the expense of some CPU overhead.  Finally, having failed all that, I can change the control pipe schedulers to FQ_CoDel and see if the problem persists.

I'm trying to establish whether the upload control pipe it just too small, or if the scheduler type is the issue.  If I do run into stalls, are there any specific outputs from the terminal that might be useful to pinpoint the issue?

[Seimus]

Overall your tshoot process above described sounds like a solid plan.

In regards of CLI,
you can check

Code Select Expand

ipfw sched show
This will show for any scheduler active flows and Tot_pkt/bytes Pkt/Byte Drp. The more traffic you push the easier is to see it in the CLI outputs.


Honestly still at this point I do not think the Control plane shaping is the problem. Because the control plane classifies, marks, queues and shapes only the control plane for IPv6 > ICPMv6. No other plane or type of traffic should hit this Pipe and Scheduler.

Issue you describe the "Stall" sounds like issue with "new flow start". Considering the "buffer bloat test" traffic should hit the Data Plane (FQ_C Scheduler);

This could be as mentioned
1. Over-tuned Scheduler (see comment from @meyergru)
2. Wrongly tuned limit parameter (which can impact slow start and new flow start)
3. ECN (which can impact slow start)

Regards,
S.

[Seimus]

Interesting tool, although I get a B for video calls each time and I do not understand why.

P.S.: Waveform still works for me.

I see similar results but for Video streaming e.g "download" results is C. For everything else I have A+ and overall far better results than is their requirement. Same goes as well for waveform, A+ 0ms increase.

The funny part is, my own testing methodology to confirm how good the Shaper is tuned is S+ rank. This is tested on real live traffic with factorial of partner yelling at you.

How to use this testing methodology, do all at once:
1. Check if you partner is watching her favorite show on live TV
2. Turn on an Online game
3. Start to Download a game on Steam
4. Run a full system upgrade on Linux
5. Observe

Result outcome:
If, 2. + 3. + 4. do not show any problems = A+
If, 1. + 2. + 3. + 4. do not show any problems = S+
If, 1. shows a problem, pretend you didn't do anything and start again

Regards,
S.

[OPNenthu]

If, 1. shows a problem, pretend you didn't do anything and start again

This is my method too, but now she just blames me automatically even when it's not my fault :)

Regarding the Household test on the LibreQoS site, I asked ChatGPT what the test looks for and it gave an interesting response.  It said that the houshold test falls down quickly when using FQ_CoDel because it cannot distinguish between flows.  All traffic has equal priority so things like gaming, VoIP, etc. can get impacted quickly when there is traffic from multiple clients.

To get a good score there we need CAKE, which can distinguish clients and flows.

As it's not available on FreeBSD, the best we can do is prioritize into queues.  I guess for that to work with FQ_CoDel we would need multiple pipes right?  Or maybe one pipe with no scheduler and instead use CoDel within priority queues?

I would be tempted to try this but I don't know how to match the traffic accurately.  For example, how do we use rules to distinguish video streaming from regular downloads (both using HTTPS)?   Are we supposed to match by destination, e.g. all YouTube.com -> send to high prio queue?

If someone has a guide for that in OPNsense it would be great.  I'm sometimes getting an 'F' on that test.

[Seimus]

This is my method too, but now she just blames me automatically even when it's not my fault :)

Hard life of the homelabber :D

Regarding the Household test on the LibreQoS site, I asked ChatGPT what the test looks for and it gave an interesting response.  It said that the houshold test falls down quickly when using FQ_CoDel because it cannot distinguish between flows.  All traffic has equal priority so things like gaming, VoIP, etc. can get impacted quickly when there is traffic from multiple clients.

This is not true.

FQ_Codel, can and does distinguish packets into different flows. It does uses 5-tuple to create hashes to hash packets into different slots (flows/queues).
https://datatracker.ietf.org/doc/html/rfc8290#section-1.3

These slots (flows) are within the scheduler e.g "schedulers queues". And are separate from the "Shaper Queues".
In FQ_C the default allowed number of Flows is 1000 but you can increase it in order to avoid having different flows being hashed into same slot.

Flows in FQ_C are one of its core components, because FQ_C does per packet per flow "sojourn time" tracking.

FQ_C is set properly should serve all flows equally, while consistently tracking the packet sojourn time in each flow. Thus meaning One single flow should not starve others for BW but as well processing time.

As it's not available on FreeBSD, the best we can do is prioritize into queues.  I guess for that to work with FQ_CoDel we would need multiple pipes right?  Or maybe one pipe with no scheduler and instead use CoDel within priority queues?

Codel is a queue discipline
FQ is a scheduler algorithm
FQ_CoDel is officially an AQM scheduler algorithm where the queue management is done in Flows the moment packet is moved into the scheduler.

If you want to do priority, yes, than you need to divide the traffic into separate pipes each with FQ_C, classifying each service.

Or

One pipe with Weighted scheduler & Queues per traffic classification (classes) set by weights with CoDel turned on each of them. All with MASKs disabled.
This should let you share the BW but dictated by ratio of the weights and have bufferbloat/latency/jitter be managed by CoDel inside each dedicated class e.g Queue. This is because the WFQ or QFQ can read the weights from the downstream connected Queues

If you would use CoDel in the Queues you would still use a Scheduler (default WFQ+). Here the queuing would be done in the Queues.

The FUN part of all of this is you actually do not need a PRIO queue/flow. The PRIO was back in the time a fix for RTP, Real time traffic. Because back in the past there was no AQM/SQM. Without the PRIO the packet could stall in a queue where once full, TAIL drop happened, but the packets in the queue stalled so even if delivered they were already out of sync. So in order to have usable VOIP or VIDEO, PRIO queue was used to manage latency.

AQMs such as FQ_C fixed it as they introduced per packet per flow "sojourn time" tracking.

Here is an overview of the FQ_CoDel algorithm that performs these tasks in parallel:

1. Separate every traffic flow's arriving packets into their own queue.

2. Remove a small batch of packets from a queue, round-robin style, and transmit that batch through the (slow) bottleneck link to the ISP. When each batch has been fully sent, retrieve a batch from the next queue, and so on.

3. Offer back pressure to flows that are sending "more than their share" of data.

This last step is the heart of the FQ_CoDel algorithm. It measures the time that a packet remains in a queue (its "sojourn time"). That's how it determines that a flow is using more than its share. If packets have been in a queue "too long" (that is, if their sojourn times exceed the target setting for longer than the interval), FQ_CoDel begins to mark or drop some of those packets to cause the sender to slow down.

I would be tempted to try this but I don't know how to match the traffic accurately.  For example, how do we use rules to distinguish video streaming from regular downloads (both using HTTPS)?  Are we supposed to match by destination, e.g. all YouTube.com -> send to high prio queue?

Basically by any means, but you are right here. You would have to know the specifications of the service such as, IP, Port, Protocol. And surgically categorize it. 

If someone has a guide for that in OPNsense it would be great.  I'm sometimes getting an 'F' on that test.

I know this was already asked on the forum, and I provided step-by-step, but cant remember which topic it was.

Regards,
S.