[Tutorial] - How to configure fq_codel for comcast to help bufferbloat / QoS

[theogravity]

Hi there,

After seeing a few threads on how to configure fq_codel / fq codel, I eventually figured out the right settings (I wouldn't say perfect) that will get myself an A on the bufferbloat report. This post is being created to for those who do not want to sift through forum threads and have the right info in one place to get this working.

This was written using the v18.1 opnsense firmware.

I am on Comcast with a 280 Mbps download (to 300 Mbps burst) and a 10 Mbps upload (to 12 Mbps burst) for reference.

For the quantum / limit values, I used this as a guide:

https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/

Note: Do NOT check the enable CoDel box at all in any of these steps. Make sure to hit the 'apply' button after you've added in each section to apply settings.

In the Firewall > Traffic Shaper

Create two pipes

Download Pipe:

Code: [Select]

- Bandwidth: 280 Mbit/s
- queue: 2 (I found this was the best value so far after playing around with it)
- Scheduler type: FlowQueue-CoDel
- Enable (FQ-)CoDel ECN
- FQ-CoDel Quantum: 1000
- FQ-CoDel Limit: 1000
- description: I called mine "Download pipe"

For quantum / limit, the rule seems to be 300 per 100 Mbps.

Upload Pipe:

Code: [Select]

- Bandwidth: 11 Mbit/s
- Scheduler type: FlowQueue-CoDel
- Enable (FQ-)CoDel ECN
- description: I called mine "Upload pipe"

(Note: I did not define a quantum / limit here.)

Create two queues

Download queue:

Code: [Select]

- Pipe: Download pipe
- Weight: 100
- Enable (FQ-)CoDel ECN

Upload queue:

Code: [Select]

- Pipe: Upload pipe
- Weight: 100
- Enable (FQ-)CoDel ECN

Create two rules

For the download rule:

Code: [Select]

- Interface should be the WAN interface
- Target: download queue
- Protocol: ip
- Destination: The LAN network address. If you use an address of 192.168.1.x with a 255.255.255.0 subnet, the value will most likely be "192.168.1.0/24"

I use a 172.16.0.x with a 255.255.0.0 subnet, so my value is 172.16.0.0/16

For the Upload rule:

Code: [Select]

- Interface should be the WAN interface
- Target: upload queue
- Protocol: ip
- Source: The LAN network address. If you use an address of 192.168.1.x, the value will most likely be "192.168.1.0/24"

It is important you use the correct network address. The 192.168.1.0/24 value in this context means that "for any IP address under this subnet (anything under 192.168.1.x)...":

- if source, apply the upload queue when the 192.168.1.x IPs are sending data out to WAN
- if destination, apply the download queue when the WAN is sending data to 192.168.1.x addresses

Now restart your router. The settings should take effect. You do not need to restart to modify any values (but don't forget to hit 'apply' after changes) at this point on.

Notes

In the traffic shaper GUI, if you go to status, you will get the WRONG information (I think it's a bug or it's using some incorrect flag to get status). Eg:

it says FIFO instead of FQ_CODEL for the type.

Code: [Select]

Limiters:
10000: 280.000 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 75536 type FIFO flags 0x0 0 buckets 0 active
10001:  11.000 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 75537 type FIFO flags 0x0 0 buckets 0 active


Queues:
q10000  50 sl. 0 flows (1 buckets) sched 10001 weight 100 lmax 0 pri 0 droptail
q10001  50 sl. 0 flows (1 buckets) sched 10000 weight 100 lmax 0 pri 0 droptail

If you want to verify your settings, you need to go into the shell and type:

Code: [Select]

ipfw sched show
And you should get something like this:

Code: [Select]

10000: 280.000 Mbit/s    0 ms burst 0
q10000  50 sl. 0 flows (1 buckets) sched 10001 weight 100 lmax 0 pri 0 droptail
 sched 10000 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1000 limit 1000 flows 1024 ECN
   Children flowsets: 10001
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        1       83  0    0   0
10001:  11.000 Mbit/s    0 ms burst 0
q10001  50 sl. 0 flows (1 buckets) sched 10000 weight 100 lmax 0 pri 0 droptail
 sched 10001 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 600 flows 1024 ECN
   Children flowsets: 10000

Hope this helps!

Using the above settings, you should get the best performance for upload, and near-best perf for downloads, resulting in an A rating.

Feel free to post better values if you have any!

[franco]

Hi theogravity,

Thanks for this!

Moving this to the tutorial section.


Cheers,
Franco

[odites999]

Great work theogravity!! Thanks so much. I've just setup it and it's working great.


Cheers!

[odites999]

I've found another user sharing his experience: https://www.lullabot.com/articles/eliminating-robots-and-voip-glitches-with-active-queue-management. Andrew Berry gives us a similar setup with small diferences. Thanks Andrew!


Cheers!

[donald24]

I would like to add up, that if you use IPv6 destination/source rules won't match an IPv4-rule, you would better be off setting both rules for up/down to any/any and setting only the direction correct in the rule.
This way it controls the full WAN-line.

[senser]

you can simplify the rules. Instead of using ip subnetworks, just select direction: in for download queue and direction: out for upload queue.

[Tattoofreak]

you can simplify the rules. Instead of using ip subnetworks, just select direction: in for download queue and direction: out for upload queue.

Are you sure about that? Isn't it the same as in pfSense's FQ_Codel rules where IN is UP and OUT is DOWN as described in the following videolink? I'm not sure about that so that's why I am asking.

https://youtu.be/iXqExAALzR8?t=402

[N0_Klu3]

@theogravity, do you know how to get this to work if you're in a dual WAN situation?

I tried to add both to WAN1/WAN2 but it just gimped my connections and killed the dual WAN functionality.

[ingvarr]

Hi.
I assume that I am missing some of the basics, but what about passing part of the traffic through VPN?
I have some wireguard interfaces that grab traffic for some of the nodes, and they have their own gateways.
Now, physically that all goes to the same WAN, upon firewall rules with gateway specified.
So, I have:
* WAN1 with shaper rules, gets some traffic.
* WAN_covert_hole87 on Wireguard (physically same WAN1 link), gets some traffic.

Does WAN_covert_hole87 need a separate pair of rules, or shaper applies to anything that goes to the physical interface, no matter virtual gateway ceremonies?

[DiHydro]

I would like to add up, that if you use IPv6 destination/source rules won't match an IPv4-rule, you would better be off setting both rules for up/down to any/any and setting only the direction correct in the rule.
This way it controls the full WAN-line.

I was just looking at making this work for ipv6 today, as my new modem is using ipv6 addresses as of a month or so ago.

[JohnnyGrey]

I truly appreciate this! This is incredible! I'm on Comcast's 1,000/35 plan, and this seems to have helped quite a bit. It seems I needed to lower the quantum and limit to around 2400 instead of 3000. I only did one test each, so some of this may be margin of error.

Here's my results from https://www.waveform.com/tools/bufferbloat:

Before
Unloaded Latency: 13ms
Download Active Latency: +33ms
Upload Active Latency: +6ms
Down: 636.3mbps
Up: 44.9mbps

3000/3000
Unloaded Latency: 11ms
Download Active Latency: +27ms
Upload Active Latency: +1ms
Down: 725.2mbps
Up: 39.4mbps

2175/2175
Unloaded Latency: 12ms
Download Active Latency: +10ms
Upload Active Latency: +4ms
Down: 790.0mbps
Up: 38.8mbps

2400/2400
Unloaded Latency: 12ms
Download Active Latency: +7ms
Upload Active Latency: +4ms
Down: 777.6mbps
Up: 40.1mbps

Hardware:
- Motorola SB8600 (using single gigabit WAN)
- SuperMicro mobo, i3 7300, 16gb DDR4 ECC RAM.
- Using both onboard NICs, one as WAN, one as LAN.
- TP-Link 8-port gigabit switch between this PC and OPNsense

EDIT: Getting better results leaving the quantum and limit blank, and reducing the down pipe to 900mbps.
Bufferbloat Grade: A+
Unloaded Latency: 13ms
Download Active Latency: +4ms
Upload Active Latency: +0ms
Down: 889.5mbps
Up: 38.6mbps

[sophlink]

Hi,

I was struggling a lot too because my Bufferbloat grades were C or D, no matter what I did. I used this thread as the basic configuration and this another forum to set up some CoDel parameters such as target, interval, quantum, etc.:

https://community.ui.com/questions/Best-Practices-for-Smart-Que-tuning-FQ-CoDel-on-and-ER-X/845b3bd4-676c-4b3e-be0e-2fb9abe97415

But mostly, last reply in this thread remind me an important thing I forgot: bandwidth reservation for QoS to work. If you don't do this, you won't see any difference, believe me!
Reserve at least 5-10% of your bandwidth in pipes, as the user from last reply did, i.e. if you have 100 mbps, set the pipe to 90 mbps. I reserved 20% as my connection speed is pretty variable (blame ADSL).

Now my Bufferbloat tests are A+ with network quiet, even doing the test in a WiFi device:

Unloaded: 71 ms
Download Active: 7 ms
Upload Active: 0 ms

and when all devices are using network actively:

Unloaded: 63 ms
Download Active: 25 ms
Upload Active: 8 ms

[joeyboon]

Wanted to thank you! Solved this solved an issue on my 1Gb fiber link , which experienced packet loss when under heavy load.

[Jun Seo-Hyun]

Hi.
I assume that I am missing some of the basics, but what about passing part of the traffic through VPN?
I have some wireguard interfaces that grab traffic for some of the nodes, and they have their own gateways.
Now, physically that all goes to the same WAN, upon firewall rules with gateway specified.
So, I have:
* WAN1 with shaper rules, gets some traffic.
* WAN_covert_hole87 on Wireguard (physically same WAN1 link), gets some traffic.

Does WAN_covert_hole87 need a separate pair of rules, or shaper applies to anything that goes to the physical interface, no matter virtual gateway ceremonies?

Hey, I was wondering if you figured out how to deal with VPN Interfaces in this setup? 
I'm facing the same problem and am not really able to get it work. Set up two additional rules for my VPN Interface but now my speeds are much lower than they should be.