Traffic Shaper not working as expected - not close to configured pipe speed

[didibo]

I've hit on a configuration which now works - after much tweaking and reading this post - https://forum.opnsense.org/index.php?topic=7423.0

The main difference seems to be introducing a queue (as opposed to going directly to the pipe in the rules), plus switching from a WFQ or FIFO scheduler type fo FlowQeue-CoDel.

This is now working as expected on my server:

[SUM]   0.00-30.06  sec  1.35 GBytes   387 Mbits/sec              sender
[SUM]   0.00-30.06  sec  1.35 GBytes   387 Mbits/sec              receiver

The above is using 10 parallel streams. I'm posting my traffic shaper config below in case anyone else may run into the same problem. Thanks all for your assistance.

<TrafficShaper version="1.0.1">
      <pipes>
        <pipe uuid="852389a7-b347-46f5-b037-98c2d3af03fd">
          <number>10000</number>
          <enabled>1</enabled>
          <bandwidth>403</bandwidth>
          <bandwidthMetric>Mbit</bandwidthMetric>
          <queue>10</queue>
          <mask>none</mask>
          <scheduler>fq_codel</scheduler>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <fqcodel_quantum>1000</fqcodel_quantum>
          <fqcodel_limit>1000</fqcodel_limit>
          <fqcodel_flows/>
          <origin>TrafficShaper</origin>
          <delay/>
          <description>down-pipe</description>
        </pipe>
        <pipe uuid="58d6c82d-bde9-4853-8fb0-d8941f38582b">
          <number>10001</number>
          <enabled>1</enabled>
          <bandwidth>23</bandwidth>
          <bandwidthMetric>Mbit</bandwidthMetric>
          <queue/>
          <mask>none</mask>
          <scheduler>fq_codel</scheduler>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <fqcodel_quantum/>
          <fqcodel_limit/>
          <fqcodel_flows/>
          <origin>TrafficShaper</origin>
          <delay/>
          <description>upload-pipe</description>
        </pipe>
      </pipes>
      <queues>
        <queue uuid="cb212c8a-d208-4692-8f98-41f3dc1d1aea">
          <number>10000</number>
          <enabled>1</enabled>
          <pipe>852389a7-b347-46f5-b037-98c2d3af03fd</pipe>
          <weight>100</weight>
          <mask>none</mask>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <description>main-down-q</description>
          <origin>TrafficShaper</origin>
        </queue>
        <queue uuid="e19bbd16-bb1a-4932-8eea-5814f9f70abd">
          <number>10001</number>
          <enabled>1</enabled>
          <pipe>58d6c82d-bde9-4853-8fb0-d8941f38582b</pipe>
          <weight>100</weight>
          <mask>none</mask>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <description>main-up-q</description>
          <origin>TrafficShaper</origin>
        </queue>
      </queues>
      <rules>
        <rule uuid="afa8b077-f5c9-4f40-ad1d-3c1d05fd8395">
          <sequence>9</sequence>
          <interface>wan</interface>
          <interface2/>
          <proto>ip</proto>
          <source>any</source>
          <source_not>0</source_not>
          <src_port>any</src_port>
          <destination>192.168.1.0/24</destination>
          <destination_not>0</destination_not>
          <dst_port>any</dst_port>
          <direction/>
          <target>cb212c8a-d208-4692-8f98-41f3dc1d1aea</target>
          <description/>
          <origin>TrafficShaper</origin>
        </rule>
        <rule uuid="22e3d03f-83ab-4c12-9e1a-99ed75e8de58">
          <sequence>10</sequence>
          <interface>wan</interface>
          <interface2/>
          <proto>ip</proto>
          <source>192.168.1.0/24</source>
          <source_not>0</source_not>
          <src_port>any</src_port>
          <destination>any</destination>
          <destination_not>0</destination_not>
          <dst_port>any</dst_port>
          <direction/>
          <target>e19bbd16-bb1a-4932-8eea-5814f9f70abd</target>
          <description/>
          <origin>TrafficShaper</origin>
        </rule>
      </rules>
    </TrafficShaper>

[Ciprian]

LE: everything you describe happens for me also, regarding the smaller throughput when TS is used.

And regarding the last solution you found, are you sure this setup has any effect, but a bufferbloat improvement? (Since the speed, like it doesn't - for me, at least, it doesn't!)

Either I'm missing something, or TS really is not working as it should - Eg: status wouldn't show any flows if any other scheduler than WFQ is selected in pipes. Also, status always displays FIFO as the scheduler in pipes, no matter what other scheduler you actually use.

And if I test the speed simultaneously with somebody else we both get the maximum speed configured in pipes, meaning enough simultaneous users can reach the absolute available maximum bandwidth of the WAN, no matter what limit you choose for the pipe(s). Like the pipes limit the speed for a flow, not for all the flows.

Reaching an end with this, I start to seriously think to a dedicated different QoS solution...

[Ciprian]

And, if the limit in pipes is the limit of WAN BW (minus 0 - 10%) then everyone's speed is of no logic: I just tested and I got ~80/20 and the other one got ~20/80 http://www.speedtest.net/result/7243848295 (my results, his are just about the same, and transposed, my download is his upload, and so).

Of course, it's not the first time I encounter this unexpected and undesired results, I just tested once again.