IPv6 Control Plane with FQ_CoDel Shaping

[OPNenthu]

EDIT: As explained in the thread below, this is not technically a work-around as I originally thought.  It is an implementation of an IPv6 control plane (a valid technique) for ICMP traffic; an example of Multi-color Shaping.  Please ignore references to "work-around."

------

This is a work-around for those of us wanting to combat bufferbloat with FQ-CoDel and ECN as per the OPNsense guide, but are seeing high packet loss on the IPv6 gateway (specifically on upload) with the shaping applied.  This issue is discussed here and here, as well as in several forum posts.

You cannot view this attachment.

(Note: some have experienced loss of IPv6 connectivity altogether although it's not clear if it's the same underlying cause.  In some cases the ISP may not be supporting ECN, as observed by @meyergru.  This won't help in those situations.)

I took the inspiration to try this from the comments in https://github.com/opnsense/core/issues/6714.  Thanks to GitHub user @aque for the hint.

Starting with the configuration from the OPNsense guide as the basis:

1. Under Firewall->Shaper->Pipes add an additional upload pipe named something like "Upload-Control".  We'll be using it to separate ICMP and ICMPv6 traffic from the CoDel shaper. You can name this more specifically like "Upload-ICMP" but you may wish to use this pipe for additional control protocols (e.g. DHCP, NTP, DNS) in the future so I went with a generic name.

I set the bandwidth for this pipe to 1 Mbit/s in my case, which seems more than enough for my home internet usage (your mileage may vary). So for example if your existing upload pipe was 40 Mbit/s, you'll reduce it to 39 Mbit/s and give the 1 Mbit/s to the new pipe.

Leave everything else default.

I personally did not create a manual queue for this (it's working without one) so I will skip over Firewall->Shaper->Queues.

2. Under Firewall->Shaper->Rules, clone the existing Upload rule and make the following edits:

- Sequence: <upload rule sequence> - 2
- Protocol: icmp
- Target: Upload-Control (the pipe you created in step 1)

Save the rule with a descriptive name like "Upload-Rule-ICMP".  The sequence needs to be at least 1 less than the default Upload rule and you may need to adjust the other rule sequence values accordingly.

3. Repeat step 2 for the ICMPv6 rule:

- Sequence: <upload rule sequence> - 1
- Protocol: ipv6-icmp
- Target: Upload-Control

Save as "Upload-Rule-ICMPv6". 

Make sure "Direction" is "out" for both of these rules (under the advanced settings).

Now when you run a speed test you should no longer see the high packet loss on the IPv6 gateway and you should see the ICMP traffic starting to get tallied under the respective rules in Firewall->Shaper->Status.

You cannot view this attachment.

You cannot view this attachment.

Hope this helps.  Do let me know if I've done something stupid here.  I am not an expert.

(If you're curious about the TCP ACK rules in the screenshot, I followed the advice given by @Seimus in this post.)

[dinguz]

Nice work! I'm wondering though — is this fixing an actual problem in day-to-day use, or is it more about looking good in tests? Would love to hear a bit more about that.

[OPNenthu]

The bufferbloat test result is not meant to give the impression of chasing numbers (apologies if it did).

I won't try and defend the use of shaping for everyone- I think it's a personal choice.  In my case I had to put my ISP gateway into bridge mode in order to run OPNsense and by doing so I've effectively disabled all the nice shaping that the ISP had included on their box.  I pay good money for a "premium" service here that is advertised heavily on TV for its low latency for gaming and video conferencing.  I might as well get what I pay for.

There is a significant difference with and without shaping, yes in terms of the raw numbers, but more importantly in terms of consistency.  With shaping enabled the latency is consistent.  Without it, I've seen it jump around a wide range (low teens to several hundred ms.)

As for routing ICMPv6 around it, purely a work-around.  OPNsense doesn't currently have a way to exclude that traffic from the shaping rule (it was requested in one of the GitHub tickets but doesn't look like it's being worked on).  I can't say whether the packet loss was having a real impact on latency as I was still getting good numbers, but the gateway status going red all the time was uncomfortable.  If it got high enough, I worried that the gateway would go down.

[meyergru]

I thought this was common knowledge:

Bufferbloat plays a role when you have a download or upload running (which might also be someone in your network streaming a video) and getting a higher latency in that case, which could result in lagging online games. It can also cause noticeable interference in audio streams.

In extreme cases, you will notice slow page buildup with complex web pages that consist of dozens or hundreds of ressources, because when your buffers are full and your network stack does not know it, the content will only get transferred on the next retry after packet losses.

This becomes especially noticeable with sites that are far away in terms of turnaround time. To lessen the effects of BDP, you normally would want a buffer size as large as you can get, but this will only go so far as your ISP lets you.

Read more about it here.

@OPNenthu : Nice work! This should probably be added to the traffic shaping guide.

[dinguz]

Apologies for not wording my question more clearly. I'm fully on board with the bufferbloat issue in general; I was just wondering more specifically about the effects of ICMPv6 apparently being squashed by other traffic.

[meyergru]

OPNenthu gave the links to the discussions of issues around this at the start of his post. Basically, using the traffic shaper breaks IPv6 connectivity under high load.

[Seimus]

Nice write up,

When I wrote the buffer-bloat guide I didn't had a possibility to test it on IPv6.
The latency/packet loss can increase as well for IPv4 pings, the reason behind is basically the starvation of BW and Queues. But its not so prominent as its for IPv6.

What you practically did is to give dedicated BW to a specific traffic type e.g BW reservation, in a way we can look at this creating a priority Pipe/Queue or better say a dedicated Pipe/Queue.

When speaking about excluding ICMP from the Queues, well there is maybe a different possibility, instead of matching IP all, to match UDP & TCP all. Because by design ICMP is not associated with TCP nor UDP. Which would result the ICMP not be matched by the default any rules, but without specifiyng a pipe it could start to eat into the whole capacity where there is none. However your approach is better, to give the ICMP, a specific traffic type, a specific dedicated configured chunk of the BW.


There are as well other methods how to mitigate this, like having more specific queues for traffic types, because in a congestion scenario, either Flow queue is TAIL dropping in the FQ_CoDel scheduler or a IP match-AnyPacket queue in Queues is TAIL dropping once the FLOW queue is full. BUT, if a congestion is ongoing, the ICMP would be hit sooner rather than later anyway. That's why I like your approach.


If this is a valid "solution" for IPv6 problems, we can adjust the official buffer-bloat guide mentioning the need to create a specific Pipe for ICMP. Or create a separate page for IPv6 "IPv6 Fighting Bufferbloat with FQ_CoDel"

Regards,
S.


P.S there is always a queue (10002.141074) even if you don't specify it ;). When you don't set a manual queue a dynamic ones are used, 2 by default as specified in the Queue field in the Pipe config.

[OPNenthu]

Thanks all- I appreciate the review/feedback. @Seimus, the explanation about BW starvation as the cause is making a lot of sense now and I can appreciate why @dinguz may not be seeing the issue depending on the available upload bandwidth.

I happen to have remote access to a second physical OPNsense at my parents' house (as well as a mini PC there) that I can use as a control for testing.  The remote instance only has the Bufferbloat fix as per the official guide and does not have the ICMPv6 work-around. The other difference is that my dad's service plan is 300/300 symmetrical compared to mine at 800/40 asymmetrical (and different ISPs).

Here's what I observe.  When I run an online speedtest on the remote network, I see that the gateway is not showing packet loss.  The delay increases slightly, but the loss remains at 0.  This tells me that it will be much harder to observe the issue on that network since it has a sufficiently wide upload pipe.  It's hard to saturate a 300Mbps link in day-to-day browsing.  I'm attaching a screenshot of the remote gateway 'Quality' graph.

On my network however, it's quite easy to start seeing the issue.  All I need to do is be connected to a VPN provider and start a couple video streams.  This creates a sustained load on the WAN upload and because of my smaller overall pipe I see the packet loss creep up.  Online speedtest is the best way to show it though as that puts an immediate heavy load.

[Seimus]

There are two components in networks/paths that directly impact performance e.g user experience;
1. BW capacity
2. Queue size

These two have a common relationship where;

If BW capacity is saturated it will cause back back pressure on the Queues causing them go full
> if a Queue is full depending on the queue management it will perform an action > Dropping, be it TAIL or Early.

How ever there are as well traffic types that can saturate a Queue while BW capacity is not saturated
> if a Queue is full depending on the queue management it will perform an action > Dropping, be it TAIL or Early.

The later is much more harder to Tshoot.
In day to day use from perspective of us Users, Homelabers we mostly experience the 1st scenario. That one matches as well for what you describe above.

TIP: in FQ_Codel you can set the size of the Flow queues, but if set to high too many packets fill the queue and it will cause unnecessary latency. If BW saturation prevails we still maybe TAIL drop from a queue.

------------

Also I think you should not call this a "ICMPv6 work-around".

Because this is by all means how a control plane should have been taken care of.

What I mean by that is, from perspective of a packet, if you are not using a shaper all traffic is looked on as falling into default class any any, one queue, one pipe. When you use QoS/Shaper most of the time basic user needs need only one queue and one pipe. And here comes the problem with control plane and congestion situation.

If we all handle as one BIG Queue and one BIG Pipe, at a certain point what should not fail (control plain) will fail and with it will the network fail.

For example when configuring BGP, and configuring QoS/Shaper we take in mind to separate BGP control plane from other kind of traffic and handle it as a different color (Queue/Pipe). We reserve for it a specific needed BW chunk to guarantee operation and non-disruption for the network during congestion events. By doing so we prevent BGP to go down and be excluded from BW and Queue starvation.

This goes for any control plane.
When we plan QoS/Shapers we need to plan as well to take the control plane in account. Such as ICMPv6 as its necessary for proper functionality of IPv6 which makes it a control plane ;)

[OPNenthu]

Noted- I'll change the description.  How about "IPv6 optimization for FQ_CoDel (anti-Bufferbloat) shaping" ?

Along the lines of a control plane, I am curious:

- Does it make sense to do this for the Download side as well?

- Is there a good way to measure the needed width of the control pipe rather than guessing at 1Mbit/s?  Does OPNsense have built in tools to measure ICMP flows?

(**EDIT: I found some netflow data under Reporting -> Insight, but it's only reporting packet and byte counts.  Not giving me an average rate.  However, the total ICMP v4/v6 count is extremely small relative to my overall traffic (<1% it seems), so probably even 0.5Mbit/s is OK.  I'll stick with 1MBit/s for now.)

- Are there other types of control traffic that make sense to go through this pipe as well?  I alluded earlier to DHCP, NTP, and possibly DNS (although I'm not noticing an issue with these).

[Seimus]

Noted- I'll change the description.  How about "IPv6 optimization for FQ_CoDel (anti-Bufferbloat) shaping" ?

Sure why not, I would call it something like "IPv6 Control Plane with FQ_CoDel Shaping". Or Multi-color Shaping, because that basically what is achieved here.


----------------------------

- Does it make sense to do this for the Download side as well?

Yes it does, if a communication is bidirectional (both ways), this needs to be specified both ways for the shaper.

- Is there a good way to measure the needed width of the control pipe rather than guessing at 1Mbit/s?  Does OPNsense have built in tools to measure ICMP flows?

(**EDIT: I found some netflow data under Reporting -> Insight, but it's only reporting packet and byte counts.  Not giving me an average rate.  However, the total ICMP v4/v6 count is extremely small relative to my overall traffic (<1% it seems), so probably even 0.5Mbit/s is OK.  I'll stick with 1MBit/s for now.)

In-build only via netflow, otherwise you need to check the protocol specification. But considering this is control plane, it should not need much BW.

- Are there other types of control traffic that make sense to go through this pipe as well?  I alluded earlier to DHCP, NTP, and possibly DNS (although I'm not noticing an issue with these).

This is a nice question, but when we are speaking about  control plane to "guarantee operation and non-disruption for the network during congestion events".
We are talking about control plane that has direct impact on the networks stability e.g L3 Protocols.

So if you run for example a dynamic routing protocol towards an external device, you would need it.

DHCP, DNS and NTP are L7 so from purely this view they would not mach into this category. There are situations where you need to have a separate class/Queue+Pipe dedicated BW for these, but you should not mix them with the L3 control plane class/Queue+Pipe. I think its not necessary to do this, FQ_C should be handle them fine. However you can create at least separate queues in the main FQ_C Pipe for at least DNS, this is how I have it setup-ed.

Look at this in a following way. If we have something critical or important , its maybe worth consideration to create for it a separate class/Queue+Pipe, to guarantee a minimum BW for operation purposes;
A. from network view
  > most critical is always something that has direct impact on the network stability > control plane + service plane
B. from client view 
  > important for example DHCP, DNS or > management plane (SSH)
C. from user view    
  > user important applications IPTV, RTP etc. > data plane (user defined apps)

A. needs to be always taken care of, always in its own dedicated way.

B. + C. considering FQ_C into the equation, this can be handled totally fine with it, in certain edge scenarios however there is necessary to separate them. Because FQ_C doesn't do any BW prioritization ~ it shares the BW equally.



Regards,
S.

P.S. Sorry for the lengthy replies, but we are here touching topics that I think are bit beyond simple config and done, but rather need to be understood

[OPNenthu]

Thanks, this is enlightening and good to have these explanations for posterity IMO.

I went ahead and replicated the Control pipes and ICMP rules for Download as well.  I wanted to scratch my curiosity so also went ahead and added two manual queues and rules for DoT to/from Quad9 via the FQ_Codel pipes.  So far everything seems to be working smoothly.  Will keep an eye on it for some time.

Screenshots of the updated solution attached, although these are above and beyond the main topic here.  Just to reiterate for those only needing to solve the IPv6 WAN packet loss with FQ_CoDel, you only need to add the Control pipes & rules in both directions.  Ignore all the ACK/DoT/Quad9 stuff (I'm too lazy to delete them at this point).

(P.S. it was tricky to match DoT by 5-tuple because it is neither a true TCP or UDP protocol according to Wikipedia, and port 853 is not specific enough.  So instead I matched ip/853 to and from the Quad9 public IPs, as I have configured in Unbound).

[Seimus]

(P.S. it was tricky to match DoT by 5-tuple because it is neither a true TCP or UDP protocol according to Wikipedia, and port 853 is not specific enough.  So instead I matched ip/853 to and from the Quad9 public IPs, as I have configured in Unbound).

You could just match the port 53 (DNS) + 853 (DoT), these are reserved ports so no other application should be use them. However if you use DoH, which is over port 443, than you need to be more precise to specify as well Destinations.

-----------------------

I was trying to lookup technical documentation for Control plane QoS used in Enterprise solutions. Looks like the default is always 1% of the used BW in your case 1% of 40Mbit. But this takes in account that there are control planes for multiple protocols.

As you run only IPv6 the 1Mbit is enough, but in case you will need to Shape as well control planes for other protocols (example BGP) its worth to consider to increase the BW of Control plane Pipe. And use weighted queues, as the default scheduler is WFQ, so basically this way you can keep one Pipe for control plane and creates classes/queues per specific protocol to allocate proper BW reservation by the merit queue weight.

-----------------------

As pointed by @meyergru, it would be really beneficial to introduce the need and understanding of Shaping for control plane traffic.
When I will have time, I will create a PR, touching this topic in general with example for IPv6.

Of course @OPNenthu if you want you can do it and share the PR and I can just contribute to it ;)

Regards,
S.

[OPNenthu]

Thanks for confirming the needed BW.  It aligns with my observations from netflow as well.

And use weighted queues, as the default scheduler is WFQ, so basically this way you can keep one Pipe for control plane and creates classes/queues per specific protocol to allocate proper BW reservation by the merit queue weight.

Glad you touched on this.  I was debating whether FIFO might perform better for this purpose, assuming the pipe was only being used for ICMP-type traffic.  I briefly tried it but wasn't noticing any difference, and the default (WFQ) gives us more options like you said.

Of course @OPNenthu if you want you can do it and share the PR and I can just contribute to it ;)

I appreciate it but I'm out of my depth on the topic. Happy to proofread or test.

[OPNenthu]

I just took a look at your bufferbloat submission for reference: https://github.com/opnsense/docs/pull/571

That doesn't seem to too bad to try and follow.  Maybe I can install a reStructuredText editor in VSCode and get some initial content down as a starting point.