guide to using fq_codel on 17.1

[opnfwb]

Hello and first post for me! Love the software and great work by this community on OPNsense! I was excited to see that fq_codel is available in 17.1 however, I can't figure out how to get this working.

Is there a guide available or a walkthrough on how to use fq_codel in OPNsense?

[opnfwb]

For the information of people that may be looking for the same thing, I was able to get this working successfully on OPNsense 17.1 by searching the forums for the OPNsense 16.7 series that introduced the fq_codel functionality.

I used the below threads to setup a simple fq_codel pipe and get a "knob-less" QoS implemented on OPNsense 17.1. It seems to be working quite well so far.

https://forum.opnsense.org/index.php?topic=4193.0
https://forum.opnsense.org/index.php?topic=3758.0

[guest15389]

Do you happen to notice this issue at all?

https://github.com/opnsense/core/issues/1279

I was trying to setup as well and just couldn't quite get it working.

I wanted to weight some traffic higher and some lower, which I did through all the queues I had setup.

Basically, I have 300/300 pipe for up/down. I setup queues up and down in terms of high/default/low.

I drop all my non priority stuff in a low queue, normal traffic in medium and I have all my gaming/voip/dns/ack traffic in my high queue.

I've tested similar setups in pfSense/Untangle/ipFire/etc to see what really does the best and I've stuck with pfSense so far but was eager to toy around with the fq_codel to see how that works.

i've yet to get the same results and I was trying to figure out if that was more related to the issue above in the setup I was trying to do.

[opnfwb]

I have not experienced the issue outlined in that Github link. However, I am trying to do something very simple with very few queues/pipes. I'm basically just trying to duplicate what IPFire and OpenWRT already do with their fq_codel bandwidth controls. I prefer a BSD based firewall because I think pf is a better technical solution to the problem and I think that the OPNsense community is more active. So I was only trying to duplicate that very simple goal.

I've attached some screenshots of my config. I'm using the DSLreports test to confirm that buffer bloat is being controlled. I am using this to ASSume that the fq_codel is working because I see the same results if I run the same DSLreports test with IPfire and it's own fq_codel enabled. I hope this helps. I can post additional screen grabs of my config if necessary.

[guest15389]

Looking at your status though, the flows are all '0' so I'm not sure it's actually matching any of the rules or working properly.

That's what I was trying to figure out if that means it isn't working / matching any rules or it is.

[guest15389]

Also, I don't think those pipe or queues are setup with FQ_CODEL.

You'd see this in the status:

[opnfwb]

Yes, I was confused by this too (partly why I posted here asking for help). Based on the links above, I just left all the defaults in place and/or left the sections blank. I think that's why some of my status information appears to be so limited.

Attached is a detailed screenshot of how I have one of the downpipes configured. All I can say would be to try it and run some bufferbloat tests and check if you are seeing an improvement. In my case, with these settings, I am seeing an improvement in the bufferbloat tests.

I wish some of this had better documentation as I still find it confusing and I'm just doing trial and error to see if it's working.

[guest15389]

In my testing so far, if you put any Source or Destination in the "Mask", it errors out creating the queues.



If you see the logs, you should see that error rolling through.

I think from my side, it doesn't matter since I have the same upload/download and I'll do some testing tonight to see if it works. I've been playing around on my VM to validate the config issues I was seeing, but it is simple to reproduce.

For Traffic Shaping, create any PIPE and pick a mask, hit apply. You get an error in the logs and it doesn't work if you check the status on the queues.

[opnfwb]

I'm using the latest version 17.1.2. I actually just reloaded it on another box and manually re-applied all the settings based on the links above. Here is what I see in the log file after creating the pipes/queues/rules.

Mar 1 15:20:08   kernel: load_dn_aqm dn_aqm PIE loaded
Mar 1 15:20:08   kernel: load_dn_aqm dn_aqm CODEL loaded
Mar 1 15:20:08   kernel: load_dn_sched dn_sched FQ_PIE loaded
Mar 1 15:20:08   kernel: load_dn_sched dn_sched FQ_CODEL loaded
Mar 1 15:20:08   kernel: load_dn_sched dn_sched PRIO loaded
Mar 1 15:20:08   kernel: load_dn_sched dn_sched WF2Q+ loaded
Mar 1 15:20:08   kernel: load_dn_sched dn_sched RR loaded
Mar 1 15:20:08   kernel: load_dn_sched dn_sched QFQ loaded
Mar 1 15:20:08   kernel: load_dn_sched dn_sched FIFO loaded
Mar 1 15:20:08   kernel: DUMMYNET 0 with IPv6 initialized (100409)
Mar 1 15:20:08   kernel: ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to accept, logging disabled

I'm only selecting fq_codel when I create the pipe and input the bandwidth value in kbits. I don't select any other fq_codel components when creating the queues or the rules. I follow RickNY's instructions to the letter and it seems to work to manage buffer bloat. With these disabled, I get bad buffer bloat. When I follow RickNY's steps I get an A+ rating on the same DSLreports test.

I'm not sure why it isn't working for you. I was able to successfully duplicate these settings on two different boxes. For what it's worth, I'm using newer Intel quad port cards with the igb driver. Not sure if that matters or not.

[guest15389]

I think you are just using codel though as you can see from your status screen and mine, you don't see the AQM Codel bit so the queue isn't setup all the way. I'm guessing the codel queue piece is but using a different  scheduler.

[guest15389]

Did a bit more testing tonight and have a ruleset that is working.

I have 300/300 FIOS Internet Connection so I setup a PIPE with 290 Mbit.



I setup 3 Queues for High/Default/Low:


Here is what one queue looks like:


My rules go from High to Low and I pretty much prioritize TCP ACK, DNS, VOIP and my gaming devices. I nuke my Plex Media Server:





I validated my rules are matching properly and I can see traffic using a shell command:


Speedtest:


I had my status which showed the queues properly:


If I mask the PIPE with a source/destination, it still errors causing the ipfw to be in an inconsistent state because of the flow busy error. I had to reboot to clear that out.

Since my link is the same up/down, I'm not sure that's a problem for me.

[opnfwb]

Your screenshots gave me an idea. I had followed the example from RickNY that I linked in an earlier post in this thread. In that example, I created my queues WITHOUT checking the Enable CoDel box.

I noticed your screenshots had that box checked. I checked that box on both of my queues and I now also show aqm codel listed in the status. It would appear as though checking the Enable CoDel box in the Pipe configuration doesn't have an effect, it has to be selected in the queue configuration.

My connection is asynchronous 120/10, I had to create two pipes (limiters) to segment my queues and rules to respect the bandwidth limits.

Limiters:
10000: 115.000 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 75536 type FIFO flags 0x0 0 buckets 0 active
10001:   9.700 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 75537 type FIFO flags 0x0 0 buckets 0 active


Queues:
q10000  50 sl. 0 flows (1 buckets) sched 10000 weight 100 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms NoECN
q10001  50 sl. 0 flows (1 buckets) sched 10001 weight 100 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms NoECN

[guest15389]

So in the PIPE config, you are not picking a mask.
In the QUEUE config, you are picking a mask.

Is that correct? I was able to remove flowbusy errors by removing a mask in the PIPE and leaving it in the QUEUE setup.

[opnfwb]

In my config, both the PIPE and the QUEUE DO NOT have a mask.

I set the source/destination networks in the rules.

Also, I did more digging in to this and it looks like if we want to have FQ_codel, we can keep the "enable codel" option unchecked in the queues. Check out Figure 11 on page 9 of this PDF: http://caia.swin.edu.au/reports/160708A/CAIA-TR-160708A.pdf

Based on what I'm reading there, it looks like FQ_codel is enabled by simply selecting it as the scheduler and then leaving the "enable codel" box unchecked in the pipe and in the queues.

[guest15389]

Nice.

Yeah, that seems to set the scheduler so it is working.

I made a few more changes I needed an upload and a download pipe since one pipe was maxing at 290 so I couldn't get my upload/download setup.

I removed all the extra boxes and validated the queue scheduler.

Code: [Select]

ipfw sched show
10000: 290.000 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 10000 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 10002 10001 10000
10001: 290.000 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 10001 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 10004 10003

So definitely is working as I would expect with those additional steps