OPNsense Forum

Archive => 20.7 Legacy Series => Topic started by: nuwe70 on October 17, 2020, 06:34:15 pm

Title: [SOLVED] Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on October 17, 2020, 06:34:15 pm
Hello,
I have discovered a strange problem using the Weighted Fair Queueing scheduler. The greater the bandwidth of the pipe, the less the weight matters. This behaviour starts about at a pipe bandwidth of 50 Mbit/s.

The following measurements show the problem.
Pipe Bandwidth    Host1 Bandwidth (Weight 100)    Host2 Bandwidth (Weight 1)
10 Mbit/s9,56 Mbit/s100 Kbit/s
20 Mbit/s19,1 Mbit/s185 Kbit/s
30 Mbit/s28,7 Mbit/s290 Kbit/s
40 Mbit/s38,3 Mbit/s380 Kbit/s
50 Mbit/s47,7 Mbit/s740 Kbit/s
60 Mbit/s55,3 Mbit/s2,46 Mbit/s
70 Mbit/s55,8 Mbit/s12,1 Mbit/s
80 Mbit/s55,9 Mbit/s22,2 Mbit/s
90 Mbit/s55,8 Mbit/s31,5 Mbit/s
100 Mbit/s55,8 Mbit/s41,1 Mbit/s

The Bandwidth tests were performed with iperf3. The test setup consists of one pipe, two queues and two roles.

Code: [Select]
Bandwidth: 100  (adjusted for each test as seen above)
Metric: Mbit/s
Mask: none
Description: PipeDown

Code: [Select]
Pipe: PipeDown
Weight: 100
Mask: none
Description: QueueDownHigh

Code: [Select]
Pipe: PipeDown
Weight: 1
Mask: none
Description: QueueDownLow

Rule "ShapeDownHigh" assigns traffic from Host1 to the queue "QueueDownHigh".
Rule "ShapeDownLow" assigns traffic from Host2 to the queue "QueueDownLow".

Does anyone have an idea what the problem might be? Does a similar setup work for anyone?
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: mimugmail on October 17, 2020, 06:54:54 pm
Inzeresting, I try to reproduce.

What happens when you switch to QFQ instead WFQ
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on October 17, 2020, 06:58:24 pm
Thank you for your response! But unfortunately no difference.
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: mimugmail on October 17, 2020, 10:03:17 pm
Any chance to test with 20.1?
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: hkp on October 18, 2020, 02:30:24 pm
Are you sure host1 is able to saturate the pipe? What happens when you remove the second queue? Does host1 deliver 100MBits?
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on October 18, 2020, 03:21:21 pm
@hkp Sure. Then I get around 870 Mbit/s.

@mimugmail Yes. I have just downloaded OPNsense 20.1 and I will set up the test environment shortly.
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on October 18, 2020, 05:28:02 pm
Unfortunately exactly the same problem exists in OPNsense 20.1.
Edit: And also in OPNsense 19.1.4  :-\
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: mimugmail on October 18, 2020, 07:35:58 pm
This is good, not another problem due to 20.7 .. I'll test next week
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on October 27, 2020, 09:16:13 pm
Are there any news here? Can anyone try to reproduce this problem?
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on December 03, 2020, 06:29:58 pm
push  ;)
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on April 26, 2021, 09:31:14 pm
I think it's a bug in ipfw/dummynet. The same behaviour appears in the Limiter of pfSense.
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: mimugmail on April 27, 2021, 10:19:49 pm
Quote from: nuwe70 on April 26, 2021, 09:31:14 pm
I think it's a bug in ipfw/dummynet. The same behaviour appears in the Limiter of pfSense.

Do you have a link for this?
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on April 28, 2021, 12:35:04 pm
Quote from: mimugmail on April 27, 2021, 10:19:49 pm
Do you have a link for this?

I read that the framework behind the pfSense Limiter is what OPNsense uses as traffic shaper technology. So I have set up a new pfSense VM and tested it myself. The result is the same as described in the first post. Bandwidth without Limiter enabled 2.30 Gbits/sec.
The config screenshots are attached. Additionally, two rules that redirect traffic from two different hosts to queueLow and queueHigh.
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: mimugmail on April 28, 2021, 01:55:49 pm
Do you know if this is tracked somewhere at pfsense (forum or redmine).
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on April 28, 2021, 02:39:35 pm
No, unfortunately I have not found anything there.  :-\
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: mimugmail on April 28, 2021, 04:54:10 pm
Maybe we nee queue buffers! Check this article:

https://klarasystems.com/articles/dummynet-the-better-way-of-building-freebsd-networks/
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on April 28, 2021, 06:12:15 pm
I have read the article and tested with higher queue buffer. There is no difference.
If I understand it correctly, the queue buffer in the article was necessary because a single host was not able to max out the bandwidth of the pipe. This is not a problem for me.
If two hosts using the same pipe but with different queues and weights, the weight will be ignored more and more if the pipe bandwidth is higher than about 50 MBit/s. For example with a pipe bandwith of 100 MBit/s, both hosts using the same bandwidth even though one host has a weight of 1 and the other host a weight of 100. The weights are correct for a lower pipe bandwidth of, for example, 10 Mbit/s.
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: mimugmail on April 28, 2021, 07:15:43 pm
Hm, seems I need to rebuild my lab to check this.
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on April 28, 2021, 08:37:12 pm
That would be great, thanks in advance.
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: mimugmail on April 29, 2021, 09:22:13 am
Ok, this is my lab now. Can you please as detailed as possible describe what should be tested with IP's/names from the diagram:

                  +--------+                         +--------+               
                  | FW-A   |        10.255.255.0     |  FW-B  |               
                  |        |-------------------------|        |               
                  +--------+                         +--------+               
                 |         |                           |       |               
                 |         |                           |       |               
                 |         |                           |       |               
 192.168.10.0    |         |                           |       |   192.168.11.0
                 |         |                           |       |               
                 |         |                           |       |               
                 |         |                           |       |               
                 |         |                           |                       
         +---------+        +----------+       +-----------+    +-----------+ 
         |         |        |          |       |           |    |           | 
         | Deb-A1  |        |Deb-A2    |       | Deb-B1    |    | Deb-B2    | 
         +---------+        +----------+       +-----------+    +-----------+ 
           .201                .202                .201            .202       
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on April 29, 2021, 10:55:10 am
Set up FW-A as follows.

Firewall -> Shaper -> Pipes -> New Pipe
Bandwidth: 10 Mbit/s
Description: Pipe

Firewall -> Shaper -> Queues -> New Queue
Pipe: Pipe
Weight: 100
Description: QueueHigh

Firewall -> Shaper -> Queues -> New Queue
Pipe: Pipe
Weight: 1
Description: QueueLow

Firewall -> Shaper -> Rules -> New Rule
Sequence: 1
Interface: WAN
Source: 192.168.10.201
Destination: any
Target: QueueLow
Description: RuleLow

Firewall -> Shaper -> Rules -> New Rule
Sequence: 2
Interface: WAN
Source: 192.168.10.202
Destination: any
Target: QueueHigh
Description: RuleHigh


Set up iperf3 server
Run two iperf3 servers on two different ports on the WAN side, for example on 10.255.255.10 with
iperf3 -s -p 5000
iperf3 -s -p 5001


Test 1
Check if a single host (for example 192.168.10.201) is limited to 10 Mbit/s with
iperf3 -c 10.255.255.10 -p 5000

If so, run iperf clients on both hosts in parallel:
192.168.10.201:
    iperf3 -c 10.255.255.10 -p 5000 -t 60
192.168.10.202:
    iperf3 -c 10.255.255.10 -p 5001 -t 60

Check if 192.168.10.201 if using about 1% of the bandwidth and 192.168.10.202 about 99% of the bandwidth.

Test 2
Edit Pipe to
Bandwidth: 100 Mbit/s

Do parallel iperf test again and now 192.168.10.201 and 192.168.10.202 using almost the same bandwidth. But this is not expected!
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: mimugmail on May 05, 2021, 01:49:25 pm
The lab is/was on a VPS hoster .. the values were too rotative. I need to set one up with real hardware.
Maybe on Friday when I'm in office
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on May 17, 2021, 11:23:30 am
Thanks for your help! Let me know if I can help in any way.
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on June 14, 2021, 01:01:44 pm
Have you had time to reproduce this behavior on real hardware?
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: mimugmail on June 14, 2021, 06:59:54 pm
No, currently I have a lab with two OPN directly connected and in each side one client. Should this also bei possible with multiple streams?
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on June 14, 2021, 08:04:14 pm
Yes, I think so. Then you have to change the rules to match the iperf server ports instead of the entire host.

Edit:
I just tested it. Based on post #20 you have to change the following.

Firewall -> Shaper -> Rules -> New Rule
Sequence: 1
Interface: WAN
Source: 192.168.10.201
Destination: any
Dst-port: 5000
Target: QueueLow
Description: RuleLow

Firewall -> Shaper -> Rules -> New Rule
Sequence: 2
Interface: WAN
Source: 192.168.10.201
Destination: any
Dst-port: 5001
Target: QueueHigh
Description: RuleHigh

And run the following command twice on host 192.168.10.201.
Code: [Select]
iperf3 -c 10.255.255.10 -p 5000 -t 60
Title: Re: Traffic Shaper: Weights over 50 MBit/s faulty
Post by: nuwe70 on September 29, 2021, 10:52:09 pm
Holy shit! :D For more than a year I have this problem. Now I figured out what causes this behavior.

I always used virtual machines to run OPNsense. The problem only exists in virtual machines, not on real hardware. The reason is that the OS is using a different value for the kernel parameter "kern.hz" in virtual environments. This parameter sets the kernel interval timer rate and affects for example dummynet or ZFS. The default value in a VM is 100 but on real hardware 1000.

So the solution is to set the kernel parameter to a higher value, for example 1000. The higher the bandwidth, the higher the value must be.
Go to System -> Settings -> Tunables and add a new entry
Tunable: kern.hz
Description: Set the kernel interval timer rate
Value: 1000

More information here: https://groups.google.com/g/mailing.freebsd.ipfw/c/oVbFsI3JqfM