tcp (ACK) Rules In Traffic Shaper Not Working

[ky41083]

Using iperf3 to test traffic shaper rules. Everything is working, except my one tcp (ACK packets only) rule. It matches absolutely every iperf3 TCP stream, if it's positioned at the top of the rule set.

If I reposition said ACK rule to the bottom of the rule set, everything works as it should (ie correct traffic into the correct queues).

I have to imagine this is a bug, as you would normally want to catch ACK's and prioritize them towards the top of a rule set. Even if you didn't, it still shouldn't be matching all TCP traffic.

This should be very easy to reproduce...

Simply make a rule bound to WAN, tcp (ACK packets only), any:any any:any, direction out, place at or near the top of your rule set. It will match all outbound WAN traffic, regardless of the ACK flag being set or not.

Any help and/or sanity checks are appreciated!

[ky41083]

More digging...

If I change the protocol of the rule, from tcp (ACK packets only), to tcp (non-ACK packets), the rule then matches absolutely no traffic at all, including ACK packets.

So, the two options are NOT reversed or anything like that on the back end, they are both simply handled wrong (misconfigured) by the back end code.

To me, this seems rather important, as being able to get both full upload and download bandwidth at the same time, relies heavily on being able to prioritize ACK packets. As is, OPNsense cannot do this.

On top of that, anyone with a previously working QoS config using ACK rules, isn't working even remotely as they intended.

[ky41083]

TLDR Version:

tcp (ACK packets only) rules: Match all traffic

tcp (non-ACK packets) rules: Match no traffic

Please fix. Thank you!

[Ciprian]

Seeing what you are describing, here I am, stating that I didn't manage Traffic Shaper to work. It is activated, my intention being on sharing the bandwidth evenly between users, but no matter the settings, I didn't get it to work as intended.

More than this, I somehow concluded that it makes no difference if it's enabled, disabled, or set-up in a particular way otherwise.

I gave up for now, but it might be the same underling cause you have described? Maybe!...

[ky41083]

Unfortunately for you, I do not think the two are related. If I (re)move the tcp ACK rules from my rule set, everything works as expected.

Getting QoS to balance is somewhat difficult to grasp at first. Try reading https://docs.opnsense.org/manual/how-tos/shaper.html again, case #2 applies directly to you. If you still can't get it working, start a new thread, many users here (including myself) have gotten this to work, and I'm sure they'd be happy to help.

[ky41083]

I am still seeing this issue. No solutions. All other rules working as expected, (still) does not appear to be a configuration issue. Will post the raw ipfw rules tomorrow.

[ky41083]

Does nobody seriously care, that the traffic shaper as is, cannot process ACK packets?

Side note, this specific firewall is currently unavailable, completely separate issue. Will post raw rules as soon as I get it back.

[iMx]

Does nobody seriously care, that the traffic shaper as is, cannot process ACK packets?

Side note, this specific firewall is currently unavailable, completely separate issue. Will post raw rules as soon as I get it back.

Your ipfw rules?  Some rules? No rules? 

I'm trying to troubleshoot something similar, I don't actually seem to see ANY of my traffic shaping rules in 'ipfw -a list', presumably I should?

[ky41083]

The box is down right now at a remote location, faulty hardware. I should be driving there this week to swap it out with another box, keeping the same storage device & config (the SSD is the only new component in it).

I was pulling the ipfw rules out of a debug related text file. Do a

Code: [Select]

find / | grep -i ipfw from an SSH session, you will see the text file listed somewhere in /tmp or /var /usr/local/etc/ipfw.rules. Either

Code: [Select]

vi or

Code: [Select]

cat that text file, and you will get back a dump of your shaper rules as they are fed into ipfw.

[iMx]

It's usually more efficient to use a complete find statement, rather than grepping the entire file system:

Code: [Select]

find ./ -name "*ipfw*"

Rules are located in /usr/local

Code: [Select]

/usr/local/etc/ipfw.rules

[ky41083]

Honestly, personal preference. Either way, find churns through the entire filesystem. Neither way is more efficient, just depends on how you like your output.

[ky41083]

Shaper rules, finally:

Code: [Select]

add 60001 pipe 10001 ip from any to any src-port  any dst-port any in  via fxp0
add 60002 pipe 10004 ip from any to any src-port  any dst-port any in  via ovpns1
add 60003 pipe 10003 ip from any to any src-port  any dst-port any out  via ovpns1
add 60004 queue 10001 ip from any to any src-port  any dst-port 67-68 out  via fxp0
add 60005 queue 10001 ip from any to any src-port  any dst-port 53 out  via fxp0
add 60006 queue 10001 ip from any to any src-port  any dst-port 123 out  via fxp0
add 60007 queue 10001 icmp from any to any src-port  any dst-port any out  via fxp0
add 60008 queue 10001 ip from any to any src-port  any dst-port 5201 out  via fxp0
add 60009 queue 10003 udp from any to any src-port  1194 dst-port any out  via fxp0
add 60010 queue 10002 ip from any to any src-port  any dst-port any out  via fxp0
add 60011 queue 10000 tcp from any to any src-port  any dst-port any out  tcpflags ack  via fxp0

As of now, the rule using the "all ACK's" preset, is at the bottom of the stack, because it matches all traffic. If anybody can spot why (I can't), hopefully we can get this fixed, as well as the "any but ACK" preset.

Thanks in advance for any help with these two bugs.

[camouflageX]

Hello ky41083,

I think I am facing the same problem as you did. Were you able to fix it? This is what I am dealing with:

I tried to set up the traffic shaper for IPsec using the Weighted Fair Queueing and different queues. It looks like this:

Limiters:
10000:   3.500 Mbit/s    0 ms burst 0
q141072  50 sl. 0 flows (1 buckets) sched 75536 weight 0 lmax 0 pri 0 droptail
 sched 75536 type FIFO flags 0x0 0 buckets 0 active
10001:   3.500 Mbit/s    0 ms burst 0
q141073  50 sl. 0 flows (1 buckets) sched 75537 weight 0 lmax 0 pri 0 droptail
 sched 75537 type FIFO flags 0x0 0 buckets 0 active

Queues:
q10004  50 sl. 0 flows (1 buckets) sched 10000 weight 10 lmax 0 pri 0 droptail
q10005  50 sl. 0 flows (1 buckets) sched 10001 weight 10 lmax 0 pri 0 droptail
q10002  50 sl. 0 flows (1 buckets) sched 10001 weight 30 lmax 0 pri 0 droptail
q10003  50 sl. 0 flows (1 buckets) sched 10001 weight 60 lmax 0 pri 0 droptail
q10000  50 sl. 0 flows (1 buckets) sched 10000 weight 30 lmax 0 pri 0 droptail
q10001  50 sl. 0 flows (1 buckets) sched 10000 weight 60 lmax 0 pri 0 droptail

IPFW rules:
60001  4620680  1247706962 queue 10004 tcp from 10.2.34.0/24 to any tcpflags ack via enc0
60002  4750788  2458322764 queue 10005 tcp from any to 10.2.34.0/24 tcpflags ack via enc0
60003  5152944   684232045 queue 10000 ip from 10.2.34.99 to any via enc0
60004  5095463   672980906 queue 10002 ip from any to 10.2.34.99 via enc0
60005  1580289    85477155 queue 10001 ip from any to 192.168.241.0/26 via enc0
60006  2659841  2520202989 queue 10003 ip from 192.168.241.0/26 to any via enc0

When I try to test the rules with iperf3, I noticed that all of its traffic would go to the "ACK packets only" rules:

root@gateway:~ # ipfw flowset show
q10004  50 sl. 1 flows (1 buckets) sched 10000 weight 10 lmax 0 pri 0 droptail
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     2605  3237662 39 37555  18
q10005  50 sl. 1 flows (1 buckets) sched 10001 weight 10 lmax 0 pri 0 droptail
  0 ip           0.0.0.0/0             0.0.0.0/0        2      104  0    0   0
q10002  50 sl. 1 flows (1 buckets) sched 10001 weight 30 lmax 0 pri 0 droptail
  0 ip           0.0.0.0/0             0.0.0.0/0        2      290  0    0   0
q10003  50 sl. 0 flows (1 buckets) sched 10001 weight 60 lmax 0 pri 0 droptail
q10000  50 sl. 1 flows (1 buckets) sched 10000 weight 30 lmax 0 pri 0 droptail
  0 ip           0.0.0.0/0             0.0.0.0/0        1      200  0    0   0
q10001  50 sl. 0 flows (1 buckets) sched 10000 weight 60 lmax 0 pri 0 droptail

Any ideas would be helpful. Did I do something wrong? Is it a bug?

[ky41083]

It is definitely a bug. I'm not sure where the bug lies, in OPNsense's implementation, or upstream. But it is most definitely a bug. A quick packet capture confirms that.

I was hoping a dev would see this, and either weigh in, or create an issue. As you can see, that never happened. I played with it a bit, tried adding / modifying rules manually via SSH, I never got it to work like it should. Ended up just not using ACK / non-ACK presets.

I never did get around to opening an issue on GitHub either, I don't have time to work through to resolution at the moment, so feel free to do that