OPNsense Forum

Archive => 18.1 Legacy Series => Topic started by: FlangeMonkey on January 29, 2018, 07:50:48 pm

Title: Traffic shaper limit not stable throughput
Post by: FlangeMonkey on January 29, 2018, 07:50:48 pm: Hi Guys,

I've been playing with the Traffic Shaper, having come for pfSense, but gave up on the shapers, I thought I'd give it another look with the change to dummynet/ipfw.

I have a simple Pipe with bandwidth set to 200Mbit and a rule for one host. When I test this limit and look at some traffic charts on opnsense, it's extremely choppy between 110Mbit and 150Mbit. I'd expect it to hit 200Mbit and stay there. I have tested at different bandwidths and it does the same, never keeping at a stable throughput.

Any ideas? Is it my understanding.

I am running on an ESXi Virtual Machine, performance and memory look ok and they are E1000 NICs.

Thanks,
Title: Re: Traffic shaper limit not stable throughput
Post by: mimugmail on January 29, 2018, 07:59:10 pm: So you have to rules, in and Out direction?

You reach 200 Mbit without Shaper?
Title: Re: Traffic shaper limit not stable throughput
Post by: FlangeMonkey on January 29, 2018, 08:05:33 pm: If I hit advanced, the default direction is both and I have tried playing with this setting and it makes no difference.

without the shaper, yes I can reach over 220Mbit, which is usually consistent.
Title: Re: Traffic shaper limit not stable throughput
Post by: mimugmail on January 29, 2018, 08:21:54 pm: I just tried it in my 10G lab via an IPSEC tunnel, set pipe to 300mbit, no matter which scheduler or codel, no queues, 2 rules, source/dest any and direction in and out. Via iperf I get constant 290Mbit which is ok for shaping.
Title: Re: Traffic shaper limit not stable throughput
Post by: FlangeMonkey on January 29, 2018, 08:38:44 pm: I might need to test with iperf and lab it. Are your Traffic Graph's flat?

Could any buffers or anything that I don't know about course this? It appears to be whenever it hits a wall, at 230Mbit I got constant (but that about my max), 225Mbit it's like the rocky mountains and about 30 to 75Mbits short.
Title: Re: Traffic shaper limit not stable throughput
Post by: FlangeMonkey on January 30, 2018, 01:17:38 pm: I have found what looks like a Bug in my testing, where do I report this?

The GUI isn't deleting pipes (not tested queues yet), if I delete one from the GUI and run 'ipfw pipe show' the pipe still exists.

Not found the course of my issues yet, so any additional input would be appreciated,

Thanks,
Title: Re: Traffic shaper limit not stable throughput
Post by: mimugmail on January 30, 2018, 01:20:58 pm: There's a reset button on the right (where apply is on the left)
Title: Re: Traffic shaper limit not stable throughput
Post by: franco on January 30, 2018, 01:39:03 pm: Some pipes can get stuck / locked -- I don't remember the details, something about "flowset busy" -- so they are created anew while not clearing the old ones by default.

Cheers,
Franco
Title: Re: Traffic shaper limit not stable throughput
Post by: guest15389 on January 30, 2018, 01:57:17 pm: If something gets stuck, it's usually related to playing around with the rules over and over.

If you type a dmesg, you'll see:

Code: [Select]
config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy!
If you reboot and check again, you'll just see everything is normal:

Code: [Select]
DUMMYNET 0 with IPv6 initialized (100409) load_dn_sched dn_sched FIFO loaded load_dn_sched dn_sched QFQ loaded load_dn_sched dn_sched RR loaded load_dn_sched dn_sched WF2Q+ loaded load_dn_sched dn_sched PRIO loaded load_dn_sched dn_sched FQ_CODEL loaded load_dn_sched dn_sched FQ_PIE loaded load_dn_aqm dn_aqm CODEL loaded load_dn_aqm dn_aqm PIE loaded
I usually just check my rules and queues are matching properly by doing an ipfw command:
Code: [Select]
root@phoenix:~ # ipfw -a list 00100 0 0 allow pfsync from any to any 00110 0 0 allow carp from any to any 00120 0 0 allow ip from any to any layer2 mac-type 0x0806,0x8035 00130 0 0 allow ip from any to any layer2 mac-type 0x888e,0x88c7 00140 0 0 allow ip from any to any layer2 mac-type 0x8863,0x8864 00150 0 0 deny ip from any to any layer2 not mac-type 0x0800,0x86dd 00200 0 0 skipto 60000 ip6 from ::1 to any 00201 380 476662 skipto 60000 ip4 from 127.0.0.0/8 to any 00202 0 0 skipto 60000 ip6 from any to ::1 00203 0 0 skipto 60000 ip4 from any to 127.0.0.0/8 01002 300 31261 skipto 60000 udp from any to 192.168.1.1 dst-port 53 keep-state :default 01002 1074 85576 skipto 60000 ip from any to { 255.255.255.255 or 192.168.1.1 } in 01002 1385 570807 skipto 60000 ip from { 255.255.255.255 or 192.168.1.1 } to any out 01002 0 0 skipto 60000 icmp from { 255.255.255.255 or 192.168.1.1 } to any out icmptypes 0 01002 0 0 skipto 60000 icmp from any to { 255.255.255.255 or 192.168.1.1 } in icmptypes 8 06000 17242 9884472 skipto 60000 tcp from any to any out 06199 21177 11088391 skipto 60000 ip from any to any 30000 0 0 count ip from any to any 60000 0 0 return ip from any to any 60001 0 0 queue 10005 ip from any to 192.168.1.30 src-port 563 in via igb0 60002 154 18765 queue 10005 ip from any to 192.168.1.31 in via igb0 60003 0 0 queue 10002 ip from 192.168.1.30 to any dst-port 563 out via igb0 60004 128 12241 queue 10002 ip from 192.168.1.31 to any out via igb0 60005 120 8682 queue 10003 icmp from any to any in via igb0 60006 133 9410 queue 10000 icmp from any to any out via igb0 60007 879 385479 queue 10003 ip from any to any src-port 53 in via igb0 60008 978 76621 queue 10000 ip from any to any dst-port 53 out via igb0 60009 117 44051 queue 10003 ip from any to 192.168.1.50 in via igb0 60010 0 0 queue 10003 ip from any to 192.168.1.51 in via igb0 60011 9 1089 queue 10003 ip from any to 192.168.1.55 in via igb0 60012 16 1225 queue 10003 ip from any to 192.168.1.90 in via igb0 60013 169 88662 queue 10000 ip from 192.168.1.50 to any out via igb0 60014 0 0 queue 10000 ip from 192.168.1.51 to any out via igb0 60015 18 1557 queue 10000 ip from 192.168.1.55 to any out via igb0 60016 23 1764 queue 10000 ip from 192.168.1.90 to any out via igb0 60017 9313 8720129 queue 10004 ip from any to any in via igb0 60018 8127 1332238 queue 10001 ip from any to any out via igb0 65533 21374 11435256 allow ip from any to any 65534 0 0 deny ip from any to any 65535 0 0 allow ip from any to any
You can see the last part, which is the queues incrementing properly on the rules matches. My basic config is 2 pipes (upload/download) and 3 queues for high/medium/low traffic. I match in and out traffic on the wan interface respectively to make sure everything gets in the right queue. At the end, I have a default catch all in and out rule that puts everything in my default queue. I can easily push my gig FIOS in both directions at almost max capacity with this.
Title: Re: Traffic shaper limit not stable throughput
Post by: FlangeMonkey on January 30, 2018, 06:09:15 pm: While testing I did perform a reboot, and the pipes were still present via ipfw but not in the GUI. I will double check and look at your mentioned log messages and report back.

I originally built a couple of queues for testing but noticed it wasn't taking effect, I think this is because it was too high or slightly over my links true bandwidth. For troubleshooting, I took it down to one pipe, no queues and one rule for download and started to test at step one, this is where I'm having issues. For some unknown reason it's not stable, if I drop the bandwidth on the pipe to say 200Mbit, I only get about 150Mbit and its spikie.

Any idea's why would be a great help.
Title: Re: Traffic shaper limit not stable throughput
Post by: FlangeMonkey on February 01, 2018, 11:13:53 pm: I ended up testing on pfSense and had the same issue. Someone posted that it could be down to queue size, which it was however on pfSense I can take this up as high as I link, on OPNsense it's limited to 100, is there any reason why? I'd like to bump this up higher.

Thanks,
Title: Re: Traffic shaper limit not stable throughput
Post by: mimugmail on February 02, 2018, 06:00:22 am: Can you a screenshot of pipe and rules with advanced mode enabled?