Traffic Shaper not working as expected - not close to configured pipe speed

[didibo]

I'm running 18.1.5.

I've been having problems getting the traffic shaper to work. As an example, I have two simple pipes - one set to 400Mbps which I apply to download traffic on my subnet via rules, and another pipe set to 20Mbps which I apply to upload traffic on my subnet (not using queues in this example).

The interfaces on the device a gigabit - and I test using an iperf client and server on either side. The upload pipe works as expected, no problems there. However, the download pipe I only get around 250Mbit/s (when its set to 400Mbps) - no matter what settings I try for the pipe. In some circumstances, shortly after resetting the rules I see 395Mbit/s for about 4-5 seconds, and then it settles back down to around 250Mbits/s again. I've tried no mask, source, destination, codel, different scheduler types - just can't seem to get past the 250Mbit/s.

If I set the download pipe to 800Mbit/s I then get around 600Mbit/s of traffic through the interface. With no other traffic on the network, I'm struggling to see why I don't get the full speed? With no traffic shaping enabled I get around 940Mbit/s.

Any ideas why this is happening? Ultimately I'd like to start using queues to prioritise traffic but I'm just trying to get the basic pipes working for the moment - I just can't get close to the configured speed. I could fudge it by upping the bandwidth configured, but that makes no sense to me.

[mimugmail]

You always write queue but you mean pipe, correct?
Your download rule is direction "in" on interface "wan"?
What scheduler do you use?

[didibo]

Oops, yes. I've corrected the original post.

I've tried all the schedulers on the pipe - currently its set to weighted fair queueing.

The rules are set on the WAN interface - one rule for traffic sourced from my network directing to the upload pipe, one rule for traffic destined for my network - directing to the download pipe. Both rules have the direction setting set to default, i.e. both.

The download pipe works as I would expect up to around 200Mbit/s - if I go higher on the pipe bandwidth is where I see this type of behaviour e.g. below the download pipe is set to 400Mbit/s:

[  5]   0.00-1.04   sec  47.5 MBytes   383 Mbits/sec             
[  5]   1.04-2.05   sec  47.5 MBytes   394 Mbits/sec             
[  5]   2.05-3.01   sec  45.0 MBytes   393 Mbits/sec             
[  5]   3.01-4.05   sec  48.8 MBytes   394 Mbits/sec             
[  5]   4.05-5.06   sec  46.2 MBytes   385 Mbits/sec             
[  5]   5.06-6.08   sec  30.0 MBytes   246 Mbits/sec             
[  5]   6.08-7.06   sec  30.0 MBytes   255 Mbits/sec             
[  5]   7.06-8.06   sec  30.0 MBytes   254 Mbits/sec             
[  5]   8.06-9.06   sec  30.0 MBytes   249 Mbits/sec             
[  5]   9.06-10.03  sec  30.0 MBytes   259 Mbits/sec             
[  5]  10.03-11.08  sec  31.2 MBytes   250 Mbits/sec             
[  5]  11.08-12.05  sec  30.0 MBytes   261 Mbits/sec             
[  5]  12.05-13.02  sec  30.0 MBytes   259 Mbits/sec             
[  5]  13.02-14.04  sec  30.0 MBytes   248 Mbits/sec             
[  5]  14.04-15.06  sec  31.2 MBytes   256 Mbits/sec

Upload pipe works as expected (example below it was set to 23 Mbit/s):
[  5]   0.00-1.10   sec  2.00 MBytes  15.2 Mbits/sec             
[  5]   1.10-2.11   sec  2.50 MBytes  20.9 Mbits/sec             
[  5]   2.11-3.07   sec  2.50 MBytes  21.8 Mbits/sec             
[  5]   3.07-4.09   sec  2.62 MBytes  21.4 Mbits/sec             
[  5]   4.09-5.07   sec  2.50 MBytes  21.4 Mbits/sec             
[  5]   5.07-6.05   sec  2.50 MBytes  21.4 Mbits/sec             
[  5]   6.05-7.08   sec  2.62 MBytes  21.4 Mbits/sec

[mimugmail]

Please also set proper direction for both rules and both Interface wan

[didibo]

I've just done that - it still falls back to the around 250Mbit/sec throughput level:

[  5]   0.00-1.01   sec  42.5 MBytes   354 Mbits/sec             
[  5]   1.01-2.05   sec  46.2 MBytes   372 Mbits/sec             
[  5]   2.05-3.03   sec  43.8 MBytes   373 Mbits/sec             
[  5]   3.03-4.04   sec  45.0 MBytes   376 Mbits/sec             
[  5]   4.04-5.03   sec  45.0 MBytes   381 Mbits/sec             
[  5]   5.03-6.02   sec  46.2 MBytes   391 Mbits/sec             
[  5]   6.02-7.03   sec  47.5 MBytes   395 Mbits/sec             
[  5]   7.03-8.08   sec  43.8 MBytes   350 Mbits/sec             
[  5]   8.08-9.06   sec  31.2 MBytes   268 Mbits/sec             
[  5]   9.06-10.06  sec  31.2 MBytes   261 Mbits/sec             
[  5]  10.06-11.02  sec  30.0 MBytes   262 Mbits/sec             
[  5]  11.02-12.04  sec  32.5 MBytes   267 Mbits/sec             
[  5]  12.04-13.08  sec  32.5 MBytes   264 Mbits/sec             
[  5]  13.08-14.07  sec  31.2 MBytes   263 Mbits/sec             
[  5]  14.07-15.04  sec  30.0 MBytes   260 Mbits/sec             
[  5]  15.04-16.07  sec  31.2 MBytes   256 Mbits/sec             
[  5]  16.07-17.05  sec  31.2 MBytes   267 Mbits/sec             
[  5]  17.05-18.07  sec  31.2 MBytes   257 Mbits/sec             

[mimugmail]

Restarted the machine yet? Iperf with 10 parallel streams?

[guest15389]

Are you CPU bound on your router?

It's probably easier to post all your config to see where the problem might be.

[mimugmail]

Rules only matching Just IP, not tcp or udp at first

[didibo]

Nope, I'm not CPU bound. The router is a Core i7 with 4 x 2.70GHz cores. CPU is rarely above 1-2%.

As I said, it can support faster bandwidth but with the traffic shaper set it doesn't get up to configured pipe bandwidth.

FYI - here's the iperf results when the download pipe is set to 900Mbit/sec:

[  5]   0.00-1.00   sec  88.8 MBytes   743 Mbits/sec             
[  5]   1.00-2.01   sec  80.0 MBytes   669 Mbits/sec             
[  5]   2.01-3.03   sec  82.5 MBytes   674 Mbits/sec             
[  5]   3.03-4.02   sec  81.2 MBytes   688 Mbits/sec             
[  5]   4.02-5.02   sec  80.0 MBytes   672 Mbits/sec             
[  5]   5.02-6.02   sec  83.8 MBytes   702 Mbits/sec             
[  5]   6.02-7.02   sec  80.0 MBytes   673 Mbits/sec             
[  5]   7.02-8.03   sec  80.0 MBytes   666 Mbits/sec             
[  5]   8.03-9.01   sec  81.2 MBytes   692 Mbits/sec             
[  5]   9.01-10.02  sec  80.0 MBytes   669 Mbits/sec       

And here is the iperf results with the pipe's disabled:

[  5]   0.00-1.02   sec   102 MBytes   846 Mbits/sec             
[  5]   1.02-2.01   sec   106 MBytes   893 Mbits/sec             
[  5]   2.01-3.01   sec   110 MBytes   923 Mbits/sec             
[  5]   3.01-4.02   sec   112 MBytes   939 Mbits/sec             
[  5]   4.02-5.01   sec   111 MBytes   938 Mbits/sec             
[  5]   5.01-6.02   sec   112 MBytes   936 Mbits/sec             
[  5]   6.02-7.02   sec   111 MBytes   938 Mbits/sec             
[  5]   7.02-8.01   sec   111 MBytes   937 Mbits/sec             
[  5]   8.01-9.02   sec   112 MBytes   938 Mbits/sec     

I don't think it is a hardware capacity issue. If the router can do 600-700Mbps when the pipe is set to 900Mbps - it should easily be able to do 400Mbps. The problem seems to be this slow down - which in my mind, shouldn't be happening. As I said, it only seems to happen on pipes when the configured bandwidth exceeds around 200Mbps.

Here's the traffic shaper config:

   <TrafficShaper version="1.0.1">
      <pipes>
        <pipe uuid="41386202-308a-4557-b22d-5571e95e1d95">
          <number>10000</number>
          <enabled>1</enabled>
          <bandwidth>403</bandwidth>
          <bandwidthMetric>Mbit</bandwidthMetric>
          <queue/>
          <mask>none</mask>
          <scheduler/>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <fqcodel_quantum/>
          <fqcodel_limit/>
          <fqcodel_flows/>
          <origin>TrafficShaper</origin>
          <description>down-pipe</description>
        </pipe>
        <pipe uuid="774dca0d-c50e-4ba3-a48f-fa2fecc385a1">
          <number>10001</number>
          <enabled>1</enabled>
          <bandwidth>23</bandwidth>
          <bandwidthMetric>Mbit</bandwidthMetric>
          <queue/>
          <mask>none</mask>
          <scheduler/>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <fqcodel_quantum/>
          <fqcodel_limit/>
          <fqcodel_flows/>
          <origin>TrafficShaper</origin>
          <description>up-pipe</description>
        </pipe>
      </pipes>
      <queues/>
      <rules>
        <rule uuid="38eea6a7-e1a1-4581-8179-2993ad000f88">
          <sequence>1</sequence>
          <interface>wan</interface>
          <interface2/>
          <proto>ip</proto>
          <source>any</source>
          <source_not>0</source_not>
          <src_port>any</src_port>
          <destination>192.168.1.0/24</destination>
          <destination_not>0</destination_not>
          <dst_port>any</dst_port>
          <direction>in</direction>
          <target>41386202-308a-4557-b22d-5571e95e1d95</target>
          <description/>
          <origin>TrafficShaper</origin>
        </rule>
        <rule uuid="6fbe7e2d-dea7-4eb6-acd4-51e25de2af1f">
          <sequence>2</sequence>
          <interface>wan</interface>
          <interface2/>
          <proto>ip</proto>
          <source>192.168.1.0/24</source>
          <source_not>0</source_not>
          <src_port>any</src_port>
          <destination>any</destination>
          <destination_not>0</destination_not>
          <dst_port>any</dst_port>
          <direction>out</direction>
          <target>774dca0d-c50e-4ba3-a48f-fa2fecc385a1</target>
          <description/>
          <origin>TrafficShaper</origin>
        </rule>
      </rules>
    </TrafficShaper>

[guest15389]

You can log in a shell and check the CPU as if it's slowing down, you could be hitting a capacity type item.

After you've made a lot of changes, I've noticed you need to reboot the box at times too as the settings get stuck.

Are you seeing the rules properly match as well and your packet counts going up?

Code: [Select]

ipfw -a list
60001    10729575      722118824 queue 10000 ip from 192.168.1.50 to any out via igb0
60002           0              0 queue 10000 ip from 192.168.1.51 to any out via igb0
60003           0              0 queue 10000 ip from 192.168.1.55 to any out via igb0
60004      334030       27960028 queue 10000 ip from 192.168.1.90 to any out via igb0
60005    41245052    50198644141 queue 10003 ip from any to 192.168.1.50 in via igb0
60006           0              0 queue 10003 ip from any to 192.168.1.51 in via igb0
60007           0              0 queue 10003 ip from any to 192.168.1.55 in via igb0
60008      214741       17230565 queue 10003 ip from any to 192.168.1.90 in via igb0
60009   304341484   123338730748 queue 10002 ip from 192.168.1.31 to any out via igb0
60010    15648706      859429595 queue 10002 ip from 192.168.1.30 to any dst-port 563 out via igb0
60011           0              0 queue 10005 ip from any to 192.168.1.31 in via igb0
60012    57975694    86445478932 queue 10005 ip from any to 192.168.1.30 src-port 563 in via igb0
60013  3428992560  3029988556752 queue 10001 ip from any to any out via igb0
60014  6129870360  7607157122646 queue 10004 ip from any to any in via igb0

[didibo]

Not CPU bound. Logged into the shell and ran the same iperf tests. CPU utilisation does not go above 1-2% (CPU is 98-99% idle).

On the ipfw I can see the packets counts increasing on the 'pipe' entries:

60001   627060   940224830 pipe 10000 ip from any to 192.168.1.0/24 in via xn0
60002   137984     6413798 pipe 10001 ip from 192.168.1.0/24 to any out via xn0

[guest15389]

Are you running any other services like intrusion detection or anything else?

When I test my link on a    Intel(R) Core(TM) i7-4500U CPU @ 1.80GHz (4 cores), I can see when I peg my link (gigabit FIOS) and hold at ~900Mb/s, I get like 14-15% usage on the box. ~400, I can see 5-7% when the test is going on.

[didibo]

Am running other services but not IDS. This isnt a capacity issue.
See the iperf results above. When pipe is configured for 900Mbps it chugs along at 600-700Mbps. When the pipe is configured for 400Mbps it only provides around 250Mbps sustained. The issue is the drop in the rate after a few seconds which then doesnt recover with little other traffic on the network. It should be hitting or close to the pipe configured bandwidth, but it isnt.

[mimugmail]

Can you try just fifo and reboot the machine?

[didibo]

Ok, I set both pipes to FIFO, rebooted. Same results - pipe is configured for 403Mbps:

[  5]   0.00-1.05   sec  47.5 MBytes   381 Mbits/sec             
[  5]   1.05-2.01   sec  45.0 MBytes   393 Mbits/sec             
[  5]   2.01-3.05   sec  48.8 MBytes   394 Mbits/sec             
[  5]   3.05-4.03   sec  46.2 MBytes   393 Mbits/sec             
[  5]   4.03-5.05   sec  47.5 MBytes   394 Mbits/sec             
[  5]   5.05-6.03   sec  46.2 MBytes   394 Mbits/sec             
[  5]   6.03-7.05   sec  45.0 MBytes   370 Mbits/sec             
[  5]   7.05-8.07   sec  30.0 MBytes   247 Mbits/sec             
[  5]   8.07-9.00   sec  28.8 MBytes   259 Mbits/sec             
[  5]   9.00-10.08  sec  32.5 MBytes   253 Mbits/sec             
[  5]  10.08-11.08  sec  30.0 MBytes   252 Mbits/sec             
[  5]  11.08-12.04  sec  28.8 MBytes   252 Mbits/sec             
[  5]  12.04-13.03  sec  30.0 MBytes   254 Mbits/sec             
[  5]  13.03-14.06  sec  31.2 MBytes   255 Mbits/sec             
[  5]  14.06-15.06  sec  30.0 MBytes   250 Mbits/sec             
[  5]  15.06-16.07  sec  31.2 MBytes   261 Mbits/sec             
[  5]  16.07-17.05  sec  30.0 MBytes   257 Mbits/sec             
[  5]  17.05-18.09  sec  31.2 MBytes   252 Mbits/sec             
[  5]  18.09-19.08  sec  31.2 MBytes   266 Mbits/sec