Traffic shaper limit not stable throughput

[FlangeMonkey]

Hi Guys,

I've been playing with the Traffic Shaper, having come for pfSense, but gave up on the shapers, I thought I'd give it another look with the change to dummynet/ipfw.

I have a simple Pipe with bandwidth set to 200Mbit and a rule for one host.  When I test this limit and look at some traffic charts on opnsense, it's extremely choppy between 110Mbit and 150Mbit.  I'd expect it to hit 200Mbit and stay there.  I have tested at different bandwidths and it does the same, never keeping at a stable throughput.

Any ideas?  Is it my understanding.

I am running on an ESXi Virtual Machine, performance and memory look ok and they are E1000 NICs.

Thanks,

[mimugmail]

So you have to rules, in and Out direction?

You reach 200 Mbit without Shaper?

[FlangeMonkey]

If I hit advanced, the default direction is both and I have tried playing with this setting and it makes no difference.

without the shaper, yes I can reach over 220Mbit, which is usually consistent.

[mimugmail]

I just tried it in my 10G lab via an IPSEC tunnel, set pipe to 300mbit, no matter which scheduler or codel, no queues, 2 rules, source/dest any and direction in and out. Via iperf I get constant 290Mbit which is ok for shaping.

[FlangeMonkey]

I might need to test with iperf and lab it.  Are your Traffic Graph's flat?

Could any buffers or anything that I don't know about course this?  It appears to be whenever it hits a wall, at 230Mbit I got constant (but that about my max), 225Mbit it's like the rocky mountains and about 30 to 75Mbits short.

[FlangeMonkey]

I have found what looks like a Bug in my testing, where do I report this?

The GUI isn't deleting pipes (not tested queues yet), if I delete one from the GUI and run 'ipfw pipe show' the pipe still exists.

Not found the course of my issues yet, so any additional input would be appreciated,

Thanks,

[mimugmail]

There's a reset button on the right (where apply is on the left)

[franco]

Some pipes can get stuck / locked -- I don't remember the details, something about "flowset busy" -- so they are created anew while not clearing the old ones by default.


Cheers,
Franco

[guest15389]

If something gets stuck, it's usually related to playing around with the rules over and over.

If you type a dmesg, you'll see:

Code: [Select]

config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
If you reboot and check again, you'll just see everything is normal:

Code: [Select]

DUMMYNET 0 with IPv6 initialized (100409)
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_sched dn_sched FQ_PIE loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_aqm dn_aqm PIE loaded
I usually just check my rules and queues are matching properly by doing an ipfw command:

Code: [Select]

root@phoenix:~ # ipfw -a list
00100     0        0 allow pfsync from any to any
00110     0        0 allow carp from any to any
00120     0        0 allow ip from any to any layer2 mac-type 0x0806,0x8035
00130     0        0 allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140     0        0 allow ip from any to any layer2 mac-type 0x8863,0x8864
00150     0        0 deny ip from any to any layer2 not mac-type 0x0800,0x86dd
00200     0        0 skipto 60000 ip6 from ::1 to any
00201   380   476662 skipto 60000 ip4 from 127.0.0.0/8 to any
00202     0        0 skipto 60000 ip6 from any to ::1
00203     0        0 skipto 60000 ip4 from any to 127.0.0.0/8
01002   300    31261 skipto 60000 udp from any to 192.168.1.1 dst-port 53 keep-state :default
01002  1074    85576 skipto 60000 ip from any to { 255.255.255.255 or 192.168.1.1 } in
01002  1385   570807 skipto 60000 ip from { 255.255.255.255 or 192.168.1.1 } to any out
01002     0        0 skipto 60000 icmp from { 255.255.255.255 or 192.168.1.1 } to any out icmptypes 0
01002     0        0 skipto 60000 icmp from any to { 255.255.255.255 or 192.168.1.1 } in icmptypes 8
06000 17242  9884472 skipto 60000 tcp from any to any out
06199 21177 11088391 skipto 60000 ip from any to any
30000     0        0 count ip from any to any
60000     0        0 return ip from any to any
60001     0        0 queue 10005 ip from any to 192.168.1.30 src-port 563 in via igb0
60002   154    18765 queue 10005 ip from any to 192.168.1.31 in via igb0
60003     0        0 queue 10002 ip from 192.168.1.30 to any dst-port 563 out via igb0
60004   128    12241 queue 10002 ip from 192.168.1.31 to any out via igb0
60005   120     8682 queue 10003 icmp from any to any in via igb0
60006   133     9410 queue 10000 icmp from any to any out via igb0
60007   879   385479 queue 10003 ip from any to any src-port 53 in via igb0
60008   978    76621 queue 10000 ip from any to any dst-port 53 out via igb0
60009   117    44051 queue 10003 ip from any to 192.168.1.50 in via igb0
60010     0        0 queue 10003 ip from any to 192.168.1.51 in via igb0
60011     9     1089 queue 10003 ip from any to 192.168.1.55 in via igb0
60012    16     1225 queue 10003 ip from any to 192.168.1.90 in via igb0
60013   169    88662 queue 10000 ip from 192.168.1.50 to any out via igb0
60014     0        0 queue 10000 ip from 192.168.1.51 to any out via igb0
60015    18     1557 queue 10000 ip from 192.168.1.55 to any out via igb0
60016    23     1764 queue 10000 ip from 192.168.1.90 to any out via igb0
60017  9313  8720129 queue 10004 ip from any to any in via igb0
60018  8127  1332238 queue 10001 ip from any to any out via igb0
65533 21374 11435256 allow ip from any to any
65534     0        0 deny ip from any to any
65535     0        0 allow ip from any to any
You can see the last part, which is the queues incrementing properly on the rules matches. My basic config is 2 pipes (upload/download) and 3 queues for high/medium/low traffic. I match in and out traffic on the wan interface respectively to make sure everything gets in the right queue. At the end, I have a default catch all in and out rule that puts everything in my default queue. I can easily push my gig FIOS in both directions at almost max capacity with this.

[FlangeMonkey]

While testing I did perform a reboot, and the pipes were still present via ipfw but not in the GUI.  I will double check and look at your mentioned log messages and report back.

I originally built a couple of queues for testing but noticed it wasn't taking effect, I think this is because it was too high or slightly over my links true bandwidth.  For troubleshooting, I took it down to one pipe, no queues and one rule for download and started to test at step one, this is where I'm having issues.  For some unknown reason it's not stable, if I drop the bandwidth on the pipe to say 200Mbit, I only get about 150Mbit and its spikie.

Any idea's why would be a great help.

[FlangeMonkey]

I ended up testing on pfSense and had the same issue.  Someone posted that it could be down to queue size, which it was however on pfSense I can take this up as high as I link, on OPNsense it's limited to 100, is there any reason why?  I'd like to bump this up higher.

Thanks,

[mimugmail]

Can you a screenshot of pipe and rules with advanced mode enabled?