Traffic Shaper not working as expected - not close to configured pipe speed

[guest15389]

Can you try just fifo and reboot the machine?

Ok. I'm out then if you don't want to give more info. Good luck.

[didibo]

I posted the results of the FIFO and reboot as requested.

[mimugmail]

Will try to reproduce tomorrow ...

[opnfwb]

Can you post full specs of the device? I'm specifically interested in the NIC chipset and the interface (PCIe, PCI, etc.).

[didibo]

OPNsense is running in a XenCenter 7.2 hypervisor with os-xen 1.1 plugin installed.

relevant dmesg output from opnsense below:

xn0: Ethernet address: 12:c8:56:fe:5f:6c
xn1: <Virtual Network Interface> at device/vif/1xn0:  on xenbusb_front0
backend features: feature-sg feature-gso-tcp4
xn1: Ethernet address: da:6c:43:19:2d:a5
xn2: <Virtual Network Interface> at device/vif/2xn1:  on xenbusb_front0
backend features: feature-sg feature-gso-tcp4
xn2: Ethernet address: f2:d5:27:71:e7:19
xenbusb_back0: <Xen Backend Devices> on xenstore0
xenballoon0: <Xen Balloon Device> on xenstore0
xn2: xctrl0: backend features:<Xen Control Device> feature-sg on xenstore0

[opnfwb]

I would guess this is some kind of NIC tuning issue on Xen. Assuming that the underlying physical hardware is good quality (Intel NICs), I would try disabling flow control on the Xen NICs assigned to OPNsense. Also check Interfaces/Overview and see if you're getting errors/drops on any of the assigned virtual NICs.

I did some googling around and found quite a few references with Xenserver and FreeBSD NICs being slow. So this seems somewhat common to need some tweaking to run at full speed on Xen. This googling also found an old thread on OPNsense where Franco had posted this:
nettool -K <vif name> tx off
nettool -K <xen bridge> tx off

I would try disabling both TX/RX on all vNICs and see if you can re-run your tests with any improvement.

[didibo]

Thanks opnfwb - I don't think this is the issue (btw I have flow control etc. off on the NICs).

The NICs and the platform quite happily can go faster (as per my posts earlier), i.e. with the pipe's disabled I get close to full line speed:

[  5]   6.02-7.02   sec   111 MBytes   938 Mbits/sec             
[  5]   7.02-8.01   sec   111 MBytes   937 Mbits/sec             
[  5]   8.01-9.02   sec   112 MBytes   938 Mbits/sec     

If I set the pipe bandwidth to 900Mbit/sec I then see the drop:

[  5]   7.02-8.03   sec  80.0 MBytes   666 Mbits/sec             
[  5]   8.03-9.01   sec  81.2 MBytes   692 Mbits/sec             
[  5]   9.01-10.02  sec  80.0 MBytes   669 Mbits/sec       

and with the pipe bandwidth set to 400Mbit/sec I see the drop:

[  5]   0.00-1.05   sec  47.5 MBytes   381 Mbits/sec             
[  5]   1.05-2.01   sec  45.0 MBytes   393 Mbits/sec             
[  5]   2.01-3.05   sec  48.8 MBytes   394 Mbits/sec             
[  5]   3.05-4.03   sec  46.2 MBytes   393 Mbits/sec             
[  5]   4.03-5.05   sec  47.5 MBytes   394 Mbits/sec             
[  5]   5.05-6.03   sec  46.2 MBytes   394 Mbits/sec             
[  5]   6.03-7.05   sec  45.0 MBytes   370 Mbits/sec             
[  5]   7.05-8.07   sec  30.0 MBytes   247 Mbits/sec             
[  5]   8.07-9.00   sec  28.8 MBytes   259 Mbits/sec             
[  5]   9.00-10.08  sec  32.5 MBytes   253 Mbits/sec             
[  5]  10.08-11.08  sec  30.0 MBytes   252 Mbits/sec             
[  5]  11.08-12.04  sec  28.8 MBytes   252 Mbits/sec             
[  5]  12.04-13.03  sec  30.0 MBytes   254 Mbits/sec             
[  5]  13.03-14.06  sec  31.2 MBytes   255 Mbits/sec             

so the platform and the NICs can go much faster. The issue is why is there a drop? e.g. pipe set to 400 only getting 250.

Also, the pipe works as expected at speeds less than 200 Mbit/sec - I get the pipe speed. As soon as you go higher, this drop off behaviour seems to be happening with no other traffic across the opnsense router. If I want a sustained 400Mbit/sec I could set the pipe to 600Mbit/sec and I would get it, but this doesn't make any sense.

[didibo]

After some more testing, I noticed that setting the number of dynamic queues to 100 in the pipe improves matters. Still not quite up to configured pipe bandwidth (400 Mbits/sec):

[  5]   0.00-1.05   sec  38.8 MBytes   309 Mbits/sec             
[  5]   1.05-2.03   sec  37.5 MBytes   322 Mbits/sec             
[  5]   2.03-3.06   sec  40.0 MBytes   327 Mbits/sec             
[  5]   3.06-4.05   sec  38.8 MBytes   327 Mbits/sec             
[  5]   4.05-5.06   sec  40.0 MBytes   332 Mbits/sec             
[  5]   5.06-6.06   sec  38.8 MBytes   326 Mbits/sec             
[  5]   6.06-7.01   sec  37.5 MBytes   330 Mbits/sec             
[  5]   7.01-8.04   sec  40.0 MBytes   327 Mbits/sec             
[  5]   8.04-9.04   sec  40.0 MBytes   334 Mbits/sec             
[  5]   9.04-10.03  sec  38.8 MBytes   329 Mbits/sec 

and interestingly, no drop off. Is this something to do with the way dynamic queues are being handled? Is there a way to set a value higher than 100 for the pipe?

[mimugmail]

First of all, regarding iperf, only look at the sum value. It's impossible to shape a single stream within a second.
It tested your setup and it runs fine:


PIPE-UP
16Mbit
FQ_CoDEL

SRC 10.0.1.0/24
Direction OUT
WAN Interface

iperf3 -p 5000 -f m -V -c 10.0.2.10 -t 30 -P 10
[SUM]   0.00-30.00  sec  56.3 MBytes  15.7 Mbits/sec  1695             sender
[SUM]   0.00-30.00  sec  55.3 MBytes  15.5 Mbits/sec                  receiver


---


PIPE-DOWN
403Mbit
FQ_CoDEL

DST 10.0.1.0/24
Direction In
WAN Interface

iperf3 -p 5000 -f m -V -c 10.0.2.10 -t 30 -P 10 -R
[SUM]   0.00-30.00  sec  1.36 GBytes   390 Mbits/sec  28747             sender
[SUM]   0.00-30.00  sec  1.36 GBytes   390 Mbits/sec                  receiver

[didibo]

Thanks for looking into this mimugmail - appreciate it.

I've repeated the test using the set up you provided, plus the iperf3 commands as you had in your test set up.

PIPE-UP
16Mbit
FQ_CoDEL

src 192.168.1.0/24
direction OUT
WAN interface

iperf3 -p 5201 -f -m -V -c 192.168.0.11 -t 30 -P 10
[SUM]   0.00-31.31  sec  53.2 MBytes  14.3 Mbits/sec              sender
[SUM]   0.00-31.31  sec  47.6 MBytes  12.8 Mbits/sec              receiver

---

PIPE-DOWN
403Mbit
FQ_CoDEL

DST 192.168.1.0/24
direction IN
WAN interface

iperf3 -p 5201 -f -m -V -c 192.168.0.11 -t 30 -P 10 -R
[SUM]   0.00-30.71  sec   169 MBytes  46.1 Mbits/sec              sender
[SUM]   0.00-30.71  sec   168 MBytes  45.8 Mbits/sec              receiver

So I'm a bit stumped as to why we're seeing a difference. I'll keep trying a few different configurations. I may try a clean install on opnsense (don't think it will make a difference but will give it a go).

[john9527]

Just a wild guess, but I recently solved a VPN performance problem that was being caused by power saving options.   I was running with PowerD disabled, and it apparently limited the processor.   Changing to PowerD enabled with either Hiadaptive or Maximum mode fixed the problem.

[mimugmail]

Clean install, setup LAN/WAN, no firewall rules, no IPS etc. and then test with the shaper.

[didibo]

Exactly the same results with a clean install, nothing turned on (new empty config). Have also tried both with NAT and without. I've set up a separate VM to test with now as well - so will keep fiddling to see what I can find out.

[mimugmail]

Which driver do you use?

[didibo]

It's a Xen virtual machine - dmesg just reports this:

xn1: <Virtual Network Interface> at device/vif/1xn0:  on xenbusb_front0

pciconf -lv doesn't list the network interfaces. How can I find out?