Hi there,
After seeing a few threads on how to configure fq_codel / fq codel, I eventually figured out the right settings (I wouldn't say perfect) that will get myself an A on the bufferbloat report. This post is being created to for those who do not want to sift through forum threads and have the right info in one place to get this working.
This was written using the v18.1 opnsense firmware.
I am on Comcast with a 280 Mbps download (to 300 Mbps burst) and a 10 Mbps upload (to 12 Mbps burst) for reference.
For the quantum / limit values, I used this as a guide:
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
Note: Do NOT check the enable CoDel box at all in any of these steps. Make sure to hit the 'apply' button after you've added in each section to apply settings.
In the Firewall > Traffic Shaper
Create two pipes
Download Pipe:
- Bandwidth: 280 Mbit/s
- queue: 2 (I found this was the best value so far after playing around with it)
- Scheduler type: FlowQueue-CoDel
- Enable (FQ-)CoDel ECN
- FQ-CoDel Quantum: 1000
- FQ-CoDel Limit: 1000
- description: I called mine "Download pipe"
For quantum / limit, the rule seems to be 300 per 100 Mbps.
Upload Pipe:
- Bandwidth: 11 Mbit/s
- Scheduler type: FlowQueue-CoDel
- Enable (FQ-)CoDel ECN
- description: I called mine "Upload pipe"
(Note: I did not define a quantum / limit here.)
Create two queues
Download queue:
- Pipe: Download pipe
- Weight: 100
- Enable (FQ-)CoDel ECN
Upload queue:
- Pipe: Upload pipe
- Weight: 100
- Enable (FQ-)CoDel ECN
Create two rules
For the download rule:
- Interface should be the WAN interface
- Target: download queue
- Protocol: ip
- Destination: The LAN network address. If you use an address of 192.168.1.x with a 255.255.255.0 subnet, the value will most likely be "192.168.1.0/24"
I use a 172.16.0.x with a 255.255.0.0 subnet, so my value is 172.16.0.0/16
For the Upload rule:
- Interface should be the WAN interface
- Target: upload queue
- Protocol: ip
- Source: The LAN network address. If you use an address of 192.168.1.x, the value will most likely be "192.168.1.0/24"
It is important you use the correct network address. The 192.168.1.0/24 value in this context means that "for any IP address under this subnet (anything under 192.168.1.x)...":
- if source, apply the upload queue when the 192.168.1.x IPs are sending data out to WAN
- if destination, apply the download queue when the WAN is sending data to 192.168.1.x addresses
Now restart your router. The settings should take effect. You do not need to restart to modify any values (but don't forget to hit 'apply' after changes) at this point on.
Notes
In the traffic shaper GUI, if you go to status, you will get the WRONG information (I think it's a bug or it's using some incorrect flag to get status). Eg:
it says FIFO instead of FQ_CODEL for the type.
Limiters:
10000: 280.000 Mbit/s 0 ms burst 0
q75536 50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
sched 75536 type FIFO flags 0x0 0 buckets 0 active
10001: 11.000 Mbit/s 0 ms burst 0
q75537 50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
sched 75537 type FIFO flags 0x0 0 buckets 0 active
Queues:
q10000 50 sl. 0 flows (1 buckets) sched 10001 weight 100 lmax 0 pri 0 droptail
q10001 50 sl. 0 flows (1 buckets) sched 10000 weight 100 lmax 0 pri 0 droptail
If you want to verify your settings, you need to go into the shell and type:
ipfw sched show
And you should get something like this:
10000: 280.000 Mbit/s 0 ms burst 0
q10000 50 sl. 0 flows (1 buckets) sched 10001 weight 100 lmax 0 pri 0 droptail
sched 10000 type FQ_CODEL flags 0x0 0 buckets 1 active
FQ_CODEL target 5ms interval 100ms quantum 1000 limit 1000 flows 1024 ECN
Children flowsets: 10001
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 ip 0.0.0.0/0 0.0.0.0/0 1 83 0 0 0
10001: 11.000 Mbit/s 0 ms burst 0
q10001 50 sl. 0 flows (1 buckets) sched 10000 weight 100 lmax 0 pri 0 droptail
sched 10001 type FQ_CODEL flags 0x0 0 buckets 0 active
FQ_CODEL target 5ms interval 100ms quantum 1514 limit 600 flows 1024 ECN
Children flowsets: 10000
Hope this helps!
Using the above settings, you should get the best performance for upload, and near-best perf for downloads, resulting in an A rating.
Feel free to post better values if you have any!
Hi theogravity,
Thanks for this!
Moving this to the tutorial section. :)
Cheers,
Franco
Great work theogravity!! Thanks so much. I've just setup it and it's working great.
Cheers!
I've found another user sharing his experience: https://www.lullabot.com/articles/eliminating-robots-and-voip-glitches-with-active-queue-management (https://www.lullabot.com/articles/eliminating-robots-and-voip-glitches-with-active-queue-management). Andrew Berry gives us a similar setup with small diferences. Thanks Andrew!
Cheers!
I would like to add up, that if you use IPv6 destination/source rules won't match an IPv4-rule, you would better be off setting both rules for up/down to any/any and setting only the direction correct in the rule.
This way it controls the full WAN-line.
you can simplify the rules. Instead of using ip subnetworks, just select direction: in for download queue and direction: out for upload queue.
Quote from: senser on October 31, 2020, 09:50:48 PM
you can simplify the rules. Instead of using ip subnetworks, just select direction: in for download queue and direction: out for upload queue.
Are you sure about that? Isn't it the same as in pfSense's FQ_Codel rules where
IN is
UP and
OUT is
DOWN as described in the following videolink? I'm not sure about that so that's why I am asking.
https://youtu.be/iXqExAALzR8?t=402
@theogravity, do you know how to get this to work if you're in a dual WAN situation?
I tried to add both to WAN1/WAN2 but it just gimped my connections and killed the dual WAN functionality.
Hi.
I assume that I am missing some of the basics, but what about passing part of the traffic through VPN?
I have some wireguard interfaces that grab traffic for some of the nodes, and they have their own gateways.
Now, physically that all goes to the same WAN, upon firewall rules with gateway specified.
So, I have:
* WAN1 with shaper rules, gets some traffic.
* WAN_covert_hole87 on Wireguard (physically same WAN1 link), gets some traffic.
Does WAN_covert_hole87 need a separate pair of rules, or shaper applies to anything that goes to the physical interface, no matter virtual gateway ceremonies?
Quote from: donald24 on August 21, 2019, 08:03:35 PM
I would like to add up, that if you use IPv6 destination/source rules won't match an IPv4-rule, you would better be off setting both rules for up/down to any/any and setting only the direction correct in the rule.
This way it controls the full WAN-line.
I was just looking at making this work for ipv6 today, as my new modem is using ipv6 addresses as of a month or so ago.
I truly appreciate this! This is incredible! I'm on Comcast's 1,000/35 plan, and this seems to have helped quite a bit. It seems I needed to lower the quantum and limit to around 2400 instead of 3000. I only did one test each, so some of this may be margin of error.
Here's my results from https://www.waveform.com/tools/bufferbloat:
Before
Unloaded Latency: 13ms
Download Active Latency: +33ms
Upload Active Latency: +6ms
Down: 636.3mbps
Up: 44.9mbps
3000/3000
Unloaded Latency: 11ms
Download Active Latency: +27ms
Upload Active Latency: +1ms
Down: 725.2mbps
Up: 39.4mbps
2175/2175
Unloaded Latency: 12ms
Download Active Latency: +10ms
Upload Active Latency: +4ms
Down: 790.0mbps
Up: 38.8mbps
2400/2400
Unloaded Latency: 12ms
Download Active Latency: +7ms
Upload Active Latency: +4ms
Down: 777.6mbps
Up: 40.1mbps
Hardware:
- Motorola SB8600 (using single gigabit WAN)
- SuperMicro mobo, i3 7300, 16gb DDR4 ECC RAM.
- Using both onboard NICs, one as WAN, one as LAN.
- TP-Link 8-port gigabit switch between this PC and OPNsense
EDIT: Getting better results leaving the quantum and limit blank, and reducing the down pipe to 900mbps.
Bufferbloat Grade: A+
Unloaded Latency: 13ms
Download Active Latency: +4ms
Upload Active Latency: +0ms
Down: 889.5mbps
Up: 38.6mbps
Hi,
I was struggling a lot too because my Bufferbloat grades were C or D, no matter what I did. I used this thread as the basic configuration and this another forum to set up some CoDel parameters such as target, interval, quantum, etc.:
https://community.ui.com/questions/Best-Practices-for-Smart-Que-tuning-FQ-CoDel-on-and-ER-X/845b3bd4-676c-4b3e-be0e-2fb9abe97415
But mostly, last reply in this thread remind me an important thing I forgot: bandwidth reservation for QoS to work. If you don't do this, you won't see any difference, believe me!
Reserve at least 5-10% of your bandwidth in pipes, as the user from last reply did, i.e. if you have 100 mbps, set the pipe to 90 mbps. I reserved 20% as my connection speed is pretty variable (blame ADSL).
Now my Bufferbloat tests are A+ with network quiet, even doing the test in a WiFi device:
Unloaded: 71 ms
Download Active: 7 ms
Upload Active: 0 ms
and when all devices are using network actively:
Unloaded: 63 ms
Download Active: 25 ms
Upload Active: 8 ms
Wanted to thank you! Solved this solved an issue on my 1Gb fiber link , which experienced packet loss when under heavy load. :)
Quote from: ingvarr on June 28, 2021, 11:56:08 PM
Hi.
I assume that I am missing some of the basics, but what about passing part of the traffic through VPN?
I have some wireguard interfaces that grab traffic for some of the nodes, and they have their own gateways.
Now, physically that all goes to the same WAN, upon firewall rules with gateway specified.
So, I have:
* WAN1 with shaper rules, gets some traffic.
* WAN_covert_hole87 on Wireguard (physically same WAN1 link), gets some traffic.
Does WAN_covert_hole87 need a separate pair of rules, or shaper applies to anything that goes to the physical interface, no matter virtual gateway ceremonies?
Hey, I was wondering if you figured out how to deal with VPN Interfaces in this setup?
I'm facing the same problem and am not really able to get it work. Set up two additional rules for my VPN Interface but now my speeds are much lower than they should be.
From F > A... thanks so much... :D
Removing the upload quantum and using 192.168.1.0/24 rather then any has reduced latency by 90ms. !
From C>A+ all thanks to everyone on this post.
I have 1GB fiber with google Fiber for both upload and download.
Before
Bufferbloat Grade C
Unloaded 18ms
Download Active +17ms
Upload Active +171ms
Putting 900 Mbits/s on download pipe, and 850 Mbit/s on Upload pipe.
After
Bufferbloat Grade A+
Unloaded 18ms
Download Active +1ms
Upload Active +1ms
For some odd reason my upload active goes really high after 900, the sweet spot for me was 850. Hope this helps anyone.
Anybody here with AT&T fiber and using BGW320 ?
I set mine up as ip pass through, OPNsense baremetal latest version.
I am getting D grade all the time now. Before with spectrum non fiber on their gig service I managed to get A+ multiple times.
Update: got A+. Just had to dial in the upload cap
what am i doing wrong!
followed this exactly and discovered that i have no control over upload speed and has added a 300ms upload spike
either opnsense is bugged or this tutorial is wrong ive literally told it to cap upload to 5mbs and its still hitting max line capacity
or is it only capping upload when something is downloading
fixed it
How?
There's a comprehensive guide in pfsense. Just convert it to opnsense. It works perfectly well. Pay attention to the footnote at it has other tips on fqcodel parameters
https://isc.sans.edu/diary/Securing+and+Optimizing+Networks+Using+pfSense+Traffic+Shaper+Limiters+to+Combat+Bufferbloat/27102 (https://isc.sans.edu/diary/Securing+and+Optimizing+Networks+Using+pfSense+Traffic+Shaper+Limiters+to+Combat+Bufferbloat/27102)
Does anyone know the equivalent of "Queue Length" of pfsense in Opnsense?
Quote from: MagikMark on July 21, 2024, 05:12:56 AM
There's a comprehensive guide in pfsense.
Guess what? There is one for OPNsense, too:
https://docs.opnsense.org/manual/how-tos/shaper_bufferbloat.html
To be fair it is pretty new and you have to search for it ;)
Cheers,
Franco
Traffic shaping like depicted in the docs generally works, but there is a caveat: With at least one provider (Deutsche Glasfaser), the IPv4 connection gets completely dropped if traffic shaping is enabled and the connection limits are reached (https://forum.opnsense.org/index.php?topic=32912.msg159554).
Thanks for posting the guide from docs! It was co-created with the help of bufferbload community and discussed with the one of the creators of the algorithm.
Quote from: MagikMark on July 22, 2024, 03:54:43 AM
Does anyone know the equivalent of "Queue Length" of pfsense in Opnsense?
If you use FQ_C, there are 3 queue lengths that can be set up, however one of them does nothing, and second can not be set from GUI cause OPNsense doesnt have the option. In reality you care only about the queue length defined by FQ_C which is
limit.
The queue lengths are:
1. queue in the Pipe - this is not used when you create a Queue by yourself in the Queue TAB. It is only important when you create Pipe with directly attached rules to those Pipe. A dynamic queue will be created this value sets the length of the queue. Can be let on default with FQ_C.
2. queue in the Queue tab - this specifies the queue length for the Queue. However OPNsense doesn't have a setting for it, its possible to specific it via ipfw command. Can be let on default with FQ_C.
3. limit in the FQ_C - this is the queue length that is most important, as it specifies how many packets can be queued/stored by the FQ_C algorithm.
The default size of queue which specifies the queue length for a manually created Queue is 50, which is a default for Ethernet is enough, this queue size is front of the FQ_C. FQ_C handles all flows separately and divides them per the 5-tuple into his dynamic queues it creates per flow.
Regards,
S.
Quote from: meyergru on July 22, 2024, 10:29:52 AM
Traffic shaping like depicted in the docs generally works, but there is a caveat: With at least one provider (Deutsche Glasfaser), the IPv4 connection gets completely dropped if traffic shaping is enabled and the connection limits are reached (https://forum.opnsense.org/index.php?topic=32912.msg159554).
This is interesting, is it applicable to overall traffic shaper or only when you run FQ_C?
Regards,
S.
I never tried anything besides FQ_CODEL, because bufferbloat was my main concern. Alas, I had to stop trying, because the affected devices are remote and I lose contact each time this happens. Thus I ceased my experiments w/r to this.
I also have no idea what is happening there, because the same settings work fine with other providers.
Quote from: meyergru on July 22, 2024, 11:30:00 AM
I never tried anything besides FQ_CODEL, because bufferbloat was my main concern. Alas, I had to stop trying, because the affected devices are remote and I lose contact each time this happens. Thus I ceased my experiments w/r to this.
I also have no idea what is happening there, because the same settings work fine with other providers.
I see,
well the description and behavior in your German thread (I used google translate so maybe I misinterpreted something) sounds to me like issues with slow start or new flow start. For FQ_C bad performance or problems during slow/new start are usually caused by two reasons ECN & limit.
Regards,
S.
Which is why I tried with and without ECN to no avail. But it was worse than just bad performance - the IPv4 connection broke down completely and was rebuilt only after a few minutes of outage. Because of the remote access, I could never see what really happens during that time.
However, I think that this ISP has some kind of misuse protection that gets triggered, because I had similar problems when I failed to filter RFC1918 addresses on the WAN interface. I was quite stunned when I saw some RFC1918 packets going out over my WAN connecttion and found that this can happen when some devices target arbitrary IPs which get directed at the default gateway and which are not defined in any local networks.
Quote from: Seimus on July 22, 2024, 11:50:23 AM
For FQ_C bad performance or problems during slow/new start are usually caused by two reasons ECN & limit.
What is the final suggestion on ECN enable/disable? Resources, manuals, guides and forum posts are inconsistent regarding this. Some say to enable both for download and upload, some to disable it for upload. OPNsense guide isn't quite clear if it should be enabled or disabled for upload.
What does slow start refer to exactly and is there an easy way to test it?
Thank you!
The ECN standard (https://en.wikipedia.org/wiki/Explicit_Congestion_Notification) is pretty new in networking periods of time, so not everybody (i.e. ISP) supports it properly.
Using it can have adverse effects because some router equipment may abort the connection if it sees the previously reserved bits that are now being used for ECN. That is why I disabled it after seeing problems with traffic shaping with ECN enabled, alas to no avail.
So, there is no real suggestion. Theoretically, ECN would allow to signal that a congestion is imminent, so that is a plus. But if it is implemented poorly or not at all on your ISP's side, it is counter-productive.
Basically I agree what @meyergru said.
There are no clear instructions on this, you can enable it and if you don't see any problems on the flows (sessions/connections) keep it enabled.
QuoteWhat does slow start refer to exactly and is there an easy way to test it?
Slow start is exactly what is says "slow start of the flow" "TCP slow start", meaning you download something but it takes a while to reach the throughput it should.
In regards of tests, there are several tools than can be used to test it scientifically. For example crusader. But as well speed test can show it, those one mentioned in the Docs guide, or Iperf3. Or you can see it on real live flows if you see the throughput values.
Regards,
S.
@theogravity Thanks for creating this works nicely!
I was wondering if we could prioritize acks inside the pipe to process this first as this may help with the latency in local network during heavy load
Quote from: MagikMark on December 13, 2024, 12:01:46 PMI was wondering if we could prioritize acks inside the pipe to process this first as this may help with the latency in local network during heavy load
You can definitely create a separate Queue for ACK before it hits the Scheduler which creates internal queues per FLOW. This approach is good because when you have a single created Queue it can only fit 50 packets and than it TAIL drops.
However, you can not prioritize this Queue from the others. The weights in Queues when using FQ_C is not used, and FQ_C internal per flow Queues dont offer any prioritization either.
Regards,
S.
@Seimus
Thanks for the reply can you help with a guide on how I should proceed. So far I experimented with the ff:
Pipes:
1. Queue set to 2. Found out this works the best specially when using VPN from the client side. If I set this to default sometimes speed under VPN from client side do not burst. It's stuck to 10 or 100Mbps. My line is 600Mbps
Queue:
1. Created a priority upload queue with weight of 10 and the rest is set to 1.
Rules:
1. Sequence 1 is for download. Sequence 2 is for upload. Sequence 3 is priority upload.
Result.
Priority Upload is not getting any hit. Changed the rules sequence by putting priority upload to first. I'm now getting hits. However, VPN speeds originating from clients sometimes do not burst.
*VPN is set using a TorGuard app in the client's machine. It's NOT set in OPNsense
You can follow the official docs and create more Queues + attach specific rules to them. Order of rules matters cause its going from top to bottom.
https://docs.opnsense.org/manual/how-tos/shaper_bufferbloat.html
In regards
QuotePipes:
1. Queue set to 2.
The Queue setting in Pipes is used for dynamical created Queues. This is used when you don't create manually the Queues under the Queue tab. FreeBSD & OPNsense give you the possibility to create Pipe + Rule only, however there always needs to be a Queue as Queue is an adapter. When you create the Queues manually, this setting still creates the dynamic Queues but they are not used; because you attach Rules to the specific Queues you manually created.
QuoteQueue:
1. Created a priority upload queue with weight of 10 and the rest is set to 1.
As mentioned the Weight does nothing if FQ_C is used, Weighs are used by WFQ for example but not by FQ_C
QuoteRules:
1. Sequence 1 is for download. Sequence 2 is for upload. Sequence 3 is priority upload.
FQ_C doesn't provide possibility for prioritization as mentioned and why you can read more in here https://forum.opnsense.org/index.php?topic=43856.0
Another thing, I am not sure what do you mean by bursts, burst in networking e.g QoS is a time window that allows exceeding of the configured rate, this however is done by policers, shapers can not do that.
Regards,
S.
@Seimus
Thank you very much to your insights. I really appreciate it.
My new experimental setting:
Pipe:
1. Queue 3
Rules:
1. added tcp ack rule on top of the other rules
Results
It looks like latency went down by 10ms. Will be running this for a couple of days
* The bursting I meant was the speed we are getting from the ISP. I'm on a 600Mbps plan and the speed burst between 200 to 600Mbps
Its better to create manually the Queues attach them to the Pipes, and attach Rules to the manually created Queues. It will give you better control.
Order of configuration
Pipe(Scheduler) > Queue > Rule
Order of packet flow
Packet match > Rule > Queues > (Scheduler)Pipe
Order of packet flow in case of FQ_C
Packet match > Rule > Queues > Scheduler
\Flows (per 5-Tuple) > Pipe
Also
Quote* The bursting I meant was the speed we are getting from the ISP. I'm on a 600Mbps plan and the speed burst between 200 to 600Mbps
This is called throughput not bursting.
Regards,
S.
@Seimus
There's a new Traffic Shaping feature in the Firewall Rule (experimental). What would be a good practice for this?
Currently that feature is experimental so take it with a grain of salt.
Basically what that feature does is to give you a possibility to set a Queue or Pipe directly in the rules of pf (Firewall > Rules) instead of ipfw (Sharper > rules). Which means you can use as well aliases for example and all the goodies that you can normally do in Firewall rules but not in Shaper rules. Or you can do application specific Shaping like for example you create a rule for HTTP/s and use the same rule in the Queue + Pipe. Thus you dont need to create a separate Rules in the Shaper section.
Personally I didn't yet try it out as I didn't had any need for it. But few use cases come to my mind like the one above.
Regards,
S.