Messages

Holy shit! :D For more than a year I have this problem. Now I figured out what causes this behavior.

I always used virtual machines to run OPNsense. The problem only exists in virtual machines, not on real hardware. The reason is that the OS is using a different value for the kernel parameter "kern.hz" in virtual environments. This parameter sets the kernel interval timer rate and affects for example dummynet or ZFS. The default value in a VM is 100 but on real hardware 1000.

So the solution is to set the kernel parameter to a higher value, for example 1000. The higher the bandwidth, the higher the value must be.
Go to System -> Settings -> Tunables and add a new entry
Tunable: kern.hz
Description: Set the kernel interval timer rate
Value: 1000

More information here: https://groups.google.com/g/mailing.freebsd.ipfw/c/oVbFsI3JqfM

[Firewall -> Shaper -> Rules -> New Rule]

Yes, I think so. Then you have to change the rules to match the iperf server ports instead of the entire host.

Edit:
I just tested it. Based on post #20 you have to change the following.

Firewall -> Shaper -> Rules -> New Rule
Sequence: 1
Interface: WAN
Source: 192.168.10.201
Destination: any
Dst-port: 5000
Target: QueueLow
Description: RuleLow

Firewall -> Shaper -> Rules -> New Rule
Sequence: 2
Interface: WAN
Source: 192.168.10.201
Destination: any
Dst-port: 5001
Target: QueueHigh
Description: RuleHigh

And run the following command twice on host 192.168.10.201.

Code Select Expand

iperf3 -c 10.255.255.10 -p 5000 -t 60

Have you had time to reproduce this behavior on real hardware?

Thanks for your help! Let me know if I can help in any way.

[Firewall -> Shaper -> Pipes -> New Pipe]

Set up FW-A as follows.

Firewall -> Shaper -> Pipes -> New Pipe
Bandwidth: 10 Mbit/s
Description: Pipe

Firewall -> Shaper -> Queues -> New Queue
Pipe: Pipe
Weight: 100
Description: QueueHigh

Firewall -> Shaper -> Queues -> New Queue
Pipe: Pipe
Weight: 1
Description: QueueLow

Firewall -> Shaper -> Rules -> New Rule
Sequence: 1
Interface: WAN
Source: 192.168.10.201
Destination: any
Target: QueueLow
Description: RuleLow

Firewall -> Shaper -> Rules -> New Rule
Sequence: 2
Interface: WAN
Source: 192.168.10.202
Destination: any
Target: QueueHigh
Description: RuleHigh


Set up iperf3 server
Run two iperf3 servers on two different ports on the WAN side, for example on 10.255.255.10 with
iperf3 -s -p 5000
iperf3 -s -p 5001


Test 1
Check if a single host (for example 192.168.10.201) is limited to 10 Mbit/s with
iperf3 -c 10.255.255.10 -p 5000

If so, run iperf clients on both hosts in parallel:
192.168.10.201:
    iperf3 -c 10.255.255.10 -p 5000 -t 60
192.168.10.202:
    iperf3 -c 10.255.255.10 -p 5001 -t 60

Check if 192.168.10.201 if using about 1% of the bandwidth and 192.168.10.202 about 99% of the bandwidth.

Test 2
Edit Pipe to
Bandwidth: 100 Mbit/s

Do parallel iperf test again and now 192.168.10.201 and 192.168.10.202 using almost the same bandwidth. But this is not expected!

That would be great, thanks in advance.

I have read the article and tested with higher queue buffer. There is no difference.
If I understand it correctly, the queue buffer in the article was necessary because a single host was not able to max out the bandwidth of the pipe. This is not a problem for me.
If two hosts using the same pipe but with different queues and weights, the weight will be ignored more and more if the pipe bandwidth is higher than about 50 MBit/s. For example with a pipe bandwith of 100 MBit/s, both hosts using the same bandwidth even though one host has a weight of 1 and the other host a weight of 100. The weights are correct for a lower pipe bandwidth of, for example, 10 Mbit/s.

No, unfortunately I have not found anything there.  :-\

Do you have a link for this?

I read that the framework behind the pfSense Limiter is what OPNsense uses as traffic shaper technology. So I have set up a new pfSense VM and tested it myself. The result is the same as described in the first post. Bandwidth without Limiter enabled 2.30 Gbits/sec.
The config screenshots are attached. Additionally, two rules that redirect traffic from two different hosts to queueLow and queueHigh.

I think it's a bug in ipfw/dummynet. The same behaviour appears in the Limiter of pfSense.

I can confirm the problem for example with this torrent https://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.iso.torrent and Transmission.
It would be great if the entire torrent traffic would be blocked with these rules. I have also tested Suricata, there it works as expected.

I have the same problem. The widget does not work on Firefox but Chrome works fine. Also the traffic graph page works fine.

Firefox Console (two warnings, no errors):

Code Select Expand


This page uses the non standard property "zoom". Consider using calc() in the relevant property values, or using "transform" along with "transform-origin: 0 0".
(index.php)

Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help http://xhr.spec.whatwg.org/
(jquery-3.5.1.min.js:2:81766)


Chrome Console warning:

Code Select Expand


[Deprecation] Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.
(jquery-3.5.1.min.js:2)


Without the traffic shaper I have a constant bandwith of 2,2 GBit/s.

An overview screenshot of the Pipes, queues and rules can be found in the attachment. The advanced settings have not been changed.
Edit view of each pipe, queue and rule here: https://imgur.com/a/uQU5lZB
Client 1 IP: 172.16.1.101
Client 2 IP: 172.16.1.102

Edit: In this setup the rules matching the upload traffic not the download traffic. So the queue and rule descriptions are not correct. However, with the source and destination swapped in the rules, you can see exactly the same behavior in download traffic.

I think it's from upstream or some timing issue since it only happens above 70 or 100 mbit

But what does that mean for my test setup? It contains 4 VMs and does not rely on the connection to my ISP. 1 OPNsense router, 1 VM that simulates the www using two iperf servers and 2 clients that conneting to this iperf servers. These two clients have different weights (1 and 100).

@Voodoo
Do you mean this thread? https://forum.opnsense.org/index.php?topic=19660
Unfortunately I still have this problem. I'm also pretty sure this must be a bug.