Messages

After updating to 25.1.1, there is a new feature in firewall rules (Traffic shaping [experimental]). How should this be used correctly? Are normalization rules as previously discussed still needed or should interface rules be used in favor of?

Currently, the traffic shaper is working (verified via Wireshark) based off the following flow:
Normalization rules on the OPNSense mark DSCP flags (high, medium, low) -> Shaper rules process (high, medium, low) -> Queues (weighted) & Pipes -> Clients

Given the new experimental feature set, is this optimal? How would the Dev team recommend to utilize the new feature? Would using the experimental flow be less resource intensive or more efficient etc?

Again, current flow IS functional- if it ain't broke, don't fix it... However, always looking to learn and improve... Thoughts or guidance appreciated.

Just found this:

Yes, maybe you missintrepreted something.
Just a quick recap:

There are two methods to filter in FreeBSD: ipfw and pf (there are some older ones too).
In the beginning pf was the standard and all GUI stuff is based on this. Sadly pf under FreeBSD isn't very active (compared to OpenBSD) and more development goes into ipfw. But the work to rewrite all the GUI stuff would take too much time and is way too error prone since so many ppl use this in very complex ways. Don't get me wrong, pf is still the way to go, no downsides in security. Current shaping technology is only developed on ipfw so the OPN guys build a way to use both, pf for filtering and ipfw for shaping. You can in theory mark packets with DSCP values via pf (firewall rules), but you can't match them afterwards. May I have to recheck this when I find more time, perhaps I didn't test everything.

So in sum, if you want to speed up DNS, you don't need EF, you can just use the rule as a condition to give DNS more weight or bandwidth. The only way where DSCP really makes sense is in big enterprises where edge switches already mark the packets with DSCP. Then you don't need tons of rules to and link them to queues/pipes. You can just have any/any rules with a gives DSCP match.

It would seem that I'm wanting to blend two incompatible systems, unfortunately my switch cannot tag the DSCP only filter based on them.

**EDIT: After investigating further with WireShark, I was able to see DSCP values of "Class Selector 5" etc at the clients. Marking via normalization appears to be working. Additionally, I adjusted my shaper rules to be any/any and merely match on the DSCP values- the pipes are showing activity and the desktop is achieving A+ on bufferbloat - Shaping based off DSCP appears to be working. Further testing to be had..

After reading (https://forum.opnsense.org/index.php?topic=15652.0) and looking over (https://en.wikipedia.org/wiki/Differentiated_services) I have my shaper rules to match based on DSCP values that react to AQM and set my switch to also act based on DSCP values.

Is the correct way to actually set the DSCP values in OPNSense to use normalization rules (which can be applied to all interfaces) or is it better to create a floating rule or per interface rules?

In Pipe you configure the scheduler, the Weights are configured in the Queues.

Do not enable Codel in the Pipe, cause the Codel in Pipe is only turned on for dynamic queues e.g if you dont use manually created Queues.

So do this:

Pipe:
- Configure BW
- Scheduler WFQ
- Everything else blank

Queues:
- Create Queues how many you need per a specific service
- Set proper Weights, the higher the weight the more BW chunk the classified traffic will get. Image Weight as a ratio of the total BW configured in a Pipe
- Create separate DL and UP Queues
- Enable Codel on the Queues

Rules:
- Create Queues how many you need per a specific Queue to classify the packets into that specific Queue
- Create separate DL and UP Rules to classify the packet to the specific Queues


Go thru > https://docs.opnsense.org/manual/how-tos/shaper_prioritize_using_queues.html

Regards,
S.

This is exactly the current testing config. No go. Latency is much higher and very inconsistent.

excerpt from: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm

"SQM is an integrated system that performs per-packet/per-flow network scheduling, active queue management (AQM), traffic shaping, rate limiting, and QoS prioritization. In comparison, "classic" AQM only manages queue length and "classic" QoS only does prioritization."

excerpt from: https://www.bufferbloat.net/projects/cerowrt/wiki/Smart_Queue_Management/

"Smart Queue Management", or "SQM" is shorthand for an integrated network system that performs better per-packet/per flow network scheduling, active queue length management (AQM), traffic shaping/rate limiting, and QoS (prioritization).

"Classic" QoS does prioritization only.

"Classic" AQM manages queue lengths only.

"Classic" packet scheduling does some form of fair queuing only.

"Classic" traffic shaping and policing sets hard limits on queue lengths and transfer rates

"Classic" rate limiting sets hard limits on network speeds.

In reading: https://gist.github.com/bradoaks/940616

It appears that what I'm wanting to adjust in OPNSense- which is currently not including in the GUI, is the "congestion avoidance algorithms" (HFSC vs HTB etc). Is that correct or have I misunderstood some terminology or definitions here?

First and foremost @Seimus -- Thank you for staying patient with me and not interpreting my questions as argumentative or getting frustrated. I greatly appreciate the explanations and you have helped clarify some of the things I've read- such as Fq_Codel ignoring queue weights. Which also explains why I haven't been able to accomplish the goal.

3. If you care about prioritization, you can go one step back and use WFQ as scheduler and turn on Codel in the Queue. This may not perform as good as does FQ_C, but you will get prioritization based on Weights.

Thanks again for pointing in the right direction and clarifying information in my brain  ::)

Okie Dokie-- At least in initial testing, WFQ does not alleviate the issues at all. Latency spikes through the roof and nearly unusable for real time traffic (even with Codel enabled and weights configured etc).

Since we cannot configure an AQM via Gui- How can this be done via console?? Any resources to point me to?

It seems to me, perhaps this is a feature the Dev team should consider adding.. We can fine tune Fq_Codel, yet have zero options in regard to the mechanisms it relies on... just sayin :)

To further illustrate my point, most of the applications in use (even real time game traffic) are only using between 250-500Kb. However, if anyone starts a show on Netflix or any other streaming service, that traffic simply trashes the throughput of everything else. Given today's culture with so many people working from home etc, I imagine this is a very common request and/or scenario..

Am I missing something silly or is a second device/other shaper needed to accomplish per machine MAC/IP priority?

Thank you for clarification on some of the feature sets.

This does mark the packet with the specific DSCP if I am not wrong, on which the shaper can classify the packet.

However there is no reason to do this as you can classify based on 5-tuple. If you would have another device that would mark the packets before it hits the OPNsense or an application on an Endpoint would do it, you could then use OPNsense to take advantage of rules based on DSCP values.

The issue is on the WAN asymmetrical connection. On the LAN side, everything is connected 1Gbs full duplex and see no issues to/from the media server (JellyFin) etc. I have a Netgear 308gs that is currently configured for port based priority (high, medium, low) but does allow DSCP values.

All the rules in OPNSense are based on 5-tuple info, except given the nature of usage- "any" must be used for either source or destination as it is unknown (no set server, etc) and there seems to be an issue if ports or ips are not specifically declared as seen here: https://forum.opnsense.org/index.php?topic=24756.0

If you have still performance issues with FQ_C while the WAN is in congestion, you can tune it further. Keep in mind one of the most important parameters to have when using FQ_C is to set the BW properly into the shaper > pipe

If you want to do BW prioritization you have to use WFQ not FQ_C.

In all of my testing with QoS setups - WFQ does not mitigate bufferbloat and latency is all over the place for real time traffic. Fq_Codel does, and definitely helps when the house is busy.

However, the issue is actually prioritizing individual devices for throughput priority NOT bandwidth sharing (i.e. all traffic from my desktop-> process first). Which is why I need to flag specific things as voice etc. Bandwidth is not the issue.

For a contrast: DDWRT QoS allows for FQ_Codel (uses HTB or HFSC) to be used directly in conjunction with priority based on traffic type, ip address, mac address etc and can very simply accomplish my goal all on one page. Unless I have misinterpreted what is happening there and it is merely bandwidth sharing as well.

So my original question remains: What is the correct way to do this in OPNSense?

Greetings! Here's what I'm attempting to accomplish:

I want qos/sqm on to fight bufferbloat thus eliminating lag/stuttering/buffering in voip/conference calls for work machines, and for game systems such as Nintendo Switch and PS5. I also want to prioritize work/game machines over IoT devices, smart phones etc.

Here's where I'm at:

Shaper is enabled with fq_codel to combat bufferbloat. Which is working as expected in that I receive an A+ using the waveform test, as well as cloudflare, ookla etc.

Where I'm running into issue is that I cannot get Opnsense to actually prioritize traffic to/from specific machines - above other traffic, i.e. IoT devices. The VLAN for IoT is already flagged for lowest priority at the interfaces->other types->VLAN "PCP = Background(1,lowest)". Config in the shaper, queuing weights are used and in rules apply a DSCP value.

I have also made rules in the firewall->settings->normalization that mimic the shaper rules, i.e. same direction/source/destination, dscp value etc. to raise work machines to C5 and lower IoT to default (0x00).

My question is this:

What is the correct way to do this? Do normalization rules override shaper rules? Do you need both? Do DSCP values also need to be set in the interface (LAN/WAN/Floating) rules?

I'm on a asymmetrical 600/20 coax line, however, heaven forbid someone starts a Netflix stream on an Ipad - calls / games latency is all over the place.. any help, tips or advice is appreciated  8)

If traffic shaper is on.. should DSCP priority be flagged using normalization rules or the shaper rules.. or both?

Does one override the other? Would having it set in both places cause issues (assuming flag and direction on interface is the same)?

The source port is randomly selected by the client, leave as any. Theoretically, the redirect rule "should" catch everything. I, however, also have block rules for port 53, 853, 5353 and 9953 - just in case - Zero Trust. Do be sure the redirect rule is above the block rules.

I also block DNS on the WAN if source is !WAN address on port 853 (also block all other 53,853,5353,9953). This effectively limits all DNS to my selected DNS over TLS servers.

The fq_codel length error message originates if the queue is full past it's limit. After reading extensively about codel configuration and all the conflicting information available, I've determined on my Asymmetrical coax connection (600/20) that the queue length ONLY needed to be lengthened on the Download queue to stop the messages spamming the logs (since the upload max is so much lower, codel may be tuned differently).

I have tested and retested many many times using:
https://www.waveform.com/tools/bufferbloat
https://fast.com/
https://www.speedtest.net/
https://speed.cloudflare.com/

I always receive an A+ rating with fq_codel and shaping ON. Without, anywhere from B to D pending congestion and activity.

All configurations to fq_codel take place on the pipes. Same as you witnessed- I have to severely lower my Down speed.

Download = 360Mb (can fluctuate, but X<400 most stable), Queue = 2, Type = fq_codel, Target = 12, Interval = 100, ECN = ON, Quantum = 300, Limit = 4096, Flows = 8192

Upload = 18Mb, Queue = 2, Type = fq_codel, Target = 12, Interval = 80, ECN = OFF, Quantum = 300, Limit = 1024, Flows = 8192

I use weighted queues to prioritize my desktop over other hosts, and rules to prioritize DNS and TCP ACKs into those weighted queues.

Hope that helps! Happy Computing!

Further resources:
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
https://docs.opnsense.org/manual/how-tos/shaper_bufferbloat.html
https://www.lullabot.com/articles/eliminating-robots-and-voip-glitches-with-active-queue-management
https://isc.sans.edu/diary/Securing+and+Optimizing+Networks+Using+pfSense+Traffic+Shaper+Limiters+to+Combat+Bufferbloat/27102
https://github.com/opnsense/docs/blob/master/source/manual/how-tos/shaper_bufferbloat.rst

EDIT: Be advised- these settings are EXTREMELY specific to each use case (router hardware, network conditions/configuration, etc) and may not yield the same results.

Edited my original post to remove the inaccuracy. Dug up the old Unix networking book to clarify in my own brain:
"DNS hostname lookups are typically performed over UDP, but DNS also uses TCP for some operations."

Further in the hardening section, I was able to locate:
"DNS communicates over both UDP and TCP. Because UDP is a quick, packet-based protocol that allows for limited data transfer, it is typically used for the actual process of hostname resolution. TCP, meanwhile, is most commonly used for transactions that require large, reliable, and sustained data transfer- that is, zone transfers. However, individual queries can be made over TCP as well."

I must have confused the information in the zone transfer section as it goes onto discuss blocking TCP on 53 specifically in business settings and does warn: "in rare cases, this may block DNS queries, which are also permitted to use TCP. So use this approach with caution."

Thanks again for pointing me in the right direction @doktornotor and apologies for any confusion caused.

"Traditional" DNS queries only use UDP but are limited, AFAIK. eDNS and DNSSec are extensions to the DNS system - also AFAIK. However, after reading further about DNS extensions @ https://en.wikipedia.org/wiki/Domain_Name_System and https://en.wikipedia.org/wiki/Extension_Mechanisms_for_DNS It appears that these extensions run on TCP across port 53, thus I decided to enable the rules for both protocols (TCP/UDP) across port 53, as per your suggestion.

If I am incorrect please point me to resources. I am all in for continued learning and ongoing education. Hopefully, with this change we can see further performance improvements! To clarify, I changed the protocols in my firewall allow rules and also the NAT rules to be sure that traffic may flow as intended.

Thanks for the tip! @ doktornotor

Hey All  ;D,

Hopefully, this can be helpful as I saw several posts above in regards to NTP and PiHoles specifically, both of which I have running on my network and across a VLAN for IOT devices. I have been running OPNSense with a PiHole for several years- I've been able to NAT things to it without issue.

Physical setup is pretty simple - Modem -> OPNSense -> Managed Switch -> PiHole -> End Users

First suggestion: If you don't already, start using Aliases! They make the rule tables much easier to read and understand.

Let's address the PiHole or local DNS server other than OPNSense itself. The reason it breaks when this (the generic tutorial rule) is applied to the PiHole address is that the NAT rule will also catch the requests coming out of the PiHole, UNLESS you also change the source piece of the rule. We actually need to use both inverse matches (the !) on source and destination for this to work properly. See attachment 1. It reads as follows;

On LAN interface - if the source address is NOT the PiHole on any port (end devices randomize source ports) AND the destination address is NOT the PiHole on port 53, then NAT the request to the PiHole on port 53.

What this double negative rule does, is allow the PiHole to send traffic to the router/firewall whilst catching every other request.

For the VLAN or IOT interface - the source inverse is not needed as the PiHole resides on a different subnet, so you can simply say- If the destination is NOT the PiHole, NAT it to the PiHole. See attachment 2. I have a firewall rule in place that allows/passes DNS traffic to the PiHole (can be seen in attachment 3).

Furthermore, I block all other DNS requests and keep the IOT devices from talking to each other. See attachment 3. You can now extrapolate this idea across however many VLANs or subnets are in use.

Next, let's address the NTP service. I use the OPNSense machine to host NTP, which makes the NAT rules quite simple. The service is also applied on the VLAN/IOT interface. Unfortunately, within the OPNSense UI we can't select multiple destinations, so separate NAT rules are required for each interface (no biggie, just clone it and modify the interface). Additionally, within the UI, I could not select from known addresses/interfaces for the NAT IP as I could for any other NAT rule (appears to be a UI bug). I opted not using an alias here as the addresses are those of the interfaces themselves i.e. X.X.X.1 - NTP only uses UDP on port 123. See attachment 4.

The NTP rule(s) read as follows:
On selected Interface- any source on UDP that the destination is NOT the interface address on port 123, then NAT to the interface address on port 123.

Finally, don't forget to adjust the rule order across all interfaces to get traffic flowing as you intend. Remember, they process top down. Hopefully, this can save someone from banging their head too many times or giving up on the PiHole altogether! Happy computing!