Non-magic traffic shaping

[ZorbaTHut]

I've just installed OPNsense to replace an old BSD box. I've got a slightly weird situation here; I have a highly oversubscribed connection with a rather strict hierarchy of upload priorities. The priority list is basically:

* Send a little bit of bandwidth from each class
* Send as much as Class A wants
* If there's any left, send as much as Class B wants
* If there's any left, send as much as Class C wants
* If there's any left, send as much as Class D wants
* If there's any left, send as much as Class E wants

On the BSD box, I could accomplish this with CBQ; I reserved 2% of the connection for each of the low-priority processes (just so the connections didn't drop all the time), but otherwise just let bandwidth borrowing do the trick. Unfortunately I don't see an equivalent to this in OPNsense. Queue weighting is limited to a factor of 100, which, with five separate priority classes, would mean that a full 40% of my bandwidth would be going to the wrong place. And CBQ, if it even exists in the software, isn't exposed.

Is there any way to accomplish what I'm looking for?

[mimugmail]

Sadly, no. There's no real priorisation. Also with weighting you'll have drops on higher ratio queues.

[Davesworld]

There are quite a few ways people do it but I settled on a method which works well so far and does not waste any bandwidth.

I create two pipes, one for in and out, each at just barely under their max capability to cut down on bufferbloat. On these in adavnced mode I choose Flow-Queue-Codel and then tick the  Enable CoDel radio button. Move over to the Queues Tab and I make three pairs of queues, one for in and one for out, first pair is a weight of 100 which I label HighPriorityUp and HighPriorityDown, second pair at a weight of 50 which I call medium priority and lastly a pair that has a weight of 1 for everything else not pointed to a specific queue. In the Queues there once again you can pick Codel, I did not since the pipes already have that checked along with the codel scheduler.

Move to the rules, in my case, I chose to make two rules for each local IP that I want weight of 50 or 100. This is also where you choose which direction it goes, in or out. For my movie and music  streaming devices, such as Roku and devices that stream internet radio either exclusively or along with movies, I set the source to any, the destination to that devices IP and wan as the interface, since you also want them to acknowledge and not get bottled up going outbound such as when you do a heavy upload, I put the source as that IP and the destination to any and choose the outbound direction. The streamers point to the 50 weighted queues in and out respectively. VOIP points to the 100 weighted queues in and out respectively. For 1 weighted in and out queues, the source and destination is any so every device is covered by a rule. Any bandwidth not needed by high and medium priority devices at any given moment can and is used by every device not specifically covered by it's own rule.

I've tried many QOS methods over the years and this is the first time I was able to completely saturate my internet connection with four simultaneous multi GB files at a time and not have streaming radio choke and stop.

CFQ which you are used to is available in the pipe creation as well.

As far as FQ-Codel, whether to use the ECN (Explicit Congestion Notification) or not is debated and based on how fat a pipe you have to deal with, my weasley sub 10mbs connection supposedly is better without but we'll see.

[Davesworld]

Screenies Set 1

[Davesworld]

Screenies Set 2

[mimugmail]

There are quite a few ways people do it but I settled on a method which works well so far and does not waste any bandwidth.

I create two pipes, one for in and out, each at just barely under their max capability to cut down on bufferbloat. On these in adavnced mode I choose Flow-Queue-Codel and then tick the  Enable CoDel radio button. Move over to the Queues Tab and I make three pairs of queues, one for in and one for out, first pair is a weight of 100 which I label HighPriorityUp and HighPriorityDown, second pair at a weight of 50 which I call medium priority and lastly a pair that has a weight of 1 for everything else not pointed to a specific queue. In the Queues there once again you can pick Codel, I did not since the pipes already have that checked along with the codel scheduler.

Are you really sure it just dont work as expected with only FQ_CoDel?
Enabling CoDel on Scheduler FQ_CoDel is useless, I believe it even wont be set by OPN, there's a validator.
Also Queues will only work with WFQ, to be honest I never digged into FQ and Queues, but normally they only work with WFQ (and only make sense there).

If you have time, can you just do your setup (2 pipes), no queues, just FQ_Codel, and recheck? It shoud be the same result.

[Davesworld]

I was wondering whether ticking the radio button would make any difference since I already use that queueing. I have a second firewall on a separate DSL line I can tinker with as well.

[mimugmail]

CoDel on FQ_CoDel will be ignored:

https://github.com/opnsense/core/blob/master/src/opnsense/service/templates/OPNsense/IPFW/ipfw.conf#L46

FQ_CoDel doesn't support user defined queues, so this would also have no effect.

I'm quite sure FQ_CoDel on it's own is so smart to handle all alone

[namezero111111]

WFQ with Weighted queues actually works very well (for us at least).
Use non-linear priorities works well for us.
True, there may still be packetloss, but we have found performance to be comparable to CBQ.

[mimugmail]

WFQ with Weighted queues actually works very well (for us at least).
Use non-linear priorities works well for us.
True, there may still be packetloss, but we have found performance to be comparable to CBQ.

Can you post all your screenshots (advanced) of all pipes, queues and rules?
If you like also as a PM, I'd rewrite them and use for a use case in official documentation.

[namezero111111]

Ill set up an example with a few and pm you when i have results

[Davesworld]

Ill set up an example with a few and pm you when i have results

Aww man! I'm curious about how different people set up traffic shaping and how well it works for them yours especially included. Is there any way you can share it here so not just the OP can benefit? Sorry if I appear too pushy.

[mimugmail]

WFQ with Weighted queues actually works very well (for us at least).
Use non-linear priorities works well for us.
True, there may still be packetloss, but we have found performance to be comparable to CBQ.

Can you post all your screenshots (advanced) of all pipes, queues and rules?
If you like also as a PM, I'd rewrite them and use for a use case in official documentation.

I think I invested about 2 months in sum without a break for this project .. the last thing I want is to keep something secret ... 

[namezero111111]

I think he meant for the raw data to turn into proper documentation :}