Puzzling assymmetric network speeds

[holunde]

I have OPNSense installed on a Netgate SG-4860 unit.
There is nothing special about the installation, except that I have the wireguard kernel-module installed.
There are no problems with stability or anything like that and the unit can route at my full internet-speed, which is 1Gbit/s, so in quiet times I see up and download speeds using fx speedtest.net at bit over 900 mbits/s in both directions meassured using my home computer.
But I experienced assymmetric speeds connecting to my workplace using using a site to site Wireguard-connection.
Not that it is a problem because the speed is fine, but it puzzles me, what the cause is. Here are the wireguard speeds meassured against between my home-pc and my workstation at work(!) using iperf3. Both pc's run Linux Mint.

First from the home-pc(192.168.254.6) towards the work-pc(10.0.5.1)

ho@hohome:~$ iperf3 -c 10.0.5.1
Connecting to host 10.0.5.1, port 5201
[  5] local 192.168.254.6 port 59292 connected to 10.0.5.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  65.5 MBytes   549 Mbits/sec   82   1.61 MBytes       
[  5]   1.00-2.00   sec  68.8 MBytes   577 Mbits/sec    0   1.76 MBytes       
[  5]   2.00-3.00   sec  67.5 MBytes   566 Mbits/sec    0   1.89 MBytes       
[  5]   3.00-4.00   sec  68.8 MBytes   577 Mbits/sec    0   1.99 MBytes       
[  5]   4.00-5.00   sec  67.5 MBytes   566 Mbits/sec    0   2.06 MBytes       

Now from the work-pc towards the home-pc

ho@hohome:~$ iperf3 -c 10.0.5.1 -R
Connecting to host 10.0.5.1, port 5201
Reverse mode, remote host 10.0.5.1 is sending
[  5] local 192.168.254.6 port 59302 connected to 10.0.5.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  79.2 MBytes   664 Mbits/sec                 
[  5]   1.00-2.00   sec  87.7 MBytes   735 Mbits/sec                 
[  5]   2.00-3.00   sec  88.6 MBytes   744 Mbits/sec                 
[  5]   3.00-4.00   sec  91.0 MBytes   764 Mbits/sec                 
[  5]   4.00-5.00   sec  79.2 MBytes   664 Mbits/sec                 

In the second mesurement the speed varies quite a lot, but it is definitively faster
So I got curious and mesured the speed from the home-pc directly against the lan-port on the OPNSense-router, the SG-4860, using iperf3 from the console on the router(192.168.254.1)

First from the home-pc towards the OPNSense-lan-port

ho@hohome:~$ iperf3 -c 192.168.254.1
Connecting to host 192.168.254.1, port 5201
[  5] local 192.168.254.6 port 42940 connected to 192.168.254.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  81.0 MBytes   679 Mbits/sec    0    484 KBytes       
[  5]   1.00-2.00   sec  77.6 MBytes   651 Mbits/sec    0    484 KBytes       
[  5]   2.00-3.00   sec  78.7 MBytes   660 Mbits/sec    0    484 KBytes       
[  5]   3.00-4.00   sec  79.6 MBytes   668 Mbits/sec    0    484 KBytes       
[  5]   4.00-5.00   sec  78.5 MBytes   658 Mbits/sec    0    484 KBytes       

And now the other way

ho@hohome:~$ iperf3 -c 192.168.254.1 -R
Connecting to host 192.168.254.1, port 5201
Reverse mode, remote host 192.168.254.1 is sending
[  5] local 192.168.254.6 port 42944 connected to 192.168.254.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   112 MBytes   940 Mbits/sec                 
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec                 
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   4.00-5.00   sec   112 MBytes   942 Mbits/sec                 

I've repeated these tests and the behaviour seems absolutely consistent.
The pattern seems to be that traffic going IN to the lan-port of the router is slower than in the other direction.
I'm new to OPNsense, having beeen a pfSense user for years until the latest Wireguard-scandal, so there might be
something that I'm not aware of in the configuration of the OPNSense-system.
But then again, it is odd that there seems to be no assymmetric behaviour when testing with speedtest.net from the home-pc, at least not anything significant.

Does anyone have an idea of what is going on here?

[opnfwb]

Generally hosting iperf on the router itself doesn't give the best results as the router is designed to transfer packets between interfaces. When the iperf tests are running, are you seeing any bottlenecks within top? Run 'top -aSCHIP' while also running an extended iperf3 test (5 minutes or so) and see if you can notice anything getting stuck on a single core.

Another thing I would check would be flow control and Energy Efficient Ethernet (EEE). These are tunable setting for Intel NICs. You can set these tunables to disable FC and EEE for igb chipsets. Obviously if you have more than 4 interfaces, just continue to add to the interface numbers to set this on additional interfaces.

Code Select Expand

dev.igb.0.fc 0
dev.igb.1.fc 0
dev.igb.2.fc 0
dev.igb.3.fc 0
dev.igb.0.eee_disabled 1
dev.igb.1.eee_disabled 1
dev.igb.2.eee_disabled 1
dev.igb.3.eee_disabled 1

I'm saying this last because I am sure you already checked this. But I'm assuming this bottleneck couldn't be related to the work ISP connection having a slower upload speed due to bandwidth sharing with many devices in the workplace?

[holunde]

Hi opnfwb

Thanks for your answer
Yes, I have already experimented with some of the low-level settings and right now I have

dev.igb.X.fc=0
dev.igb.X.eee_disabled=1

set for all 6 ports on the device.
But in my experiments I have never seen any of them have any effect.
And this also goes for some other more powerfull routers I run OPNSense on. This model actually, https://teklager.se/en/products/routers/tlsense-i7-7500U.
But what DOES absolutely have an immediate effect on network performance on these Intel-based routers is to enable PowerD and set the power-mode to maximum in System->Settings->Miscellaneous->Power savings.
Without this the SG-4860 will not route at gigabit-speed.
There are settings like this that is being discussed from time to time, but much of it seems like guess-work. It would be nice with some authoritative guide on this for Intel chip-sets for example.
I've done a lot of tests and there is no doubt. For some reason this device has a slow-down when it comes to the situations, I have described.

[opnfwb]

You make a great point with PowerD. I've seen a few posts here suggest that the "Hi-Adaptive" setting is the most preferred. However, I think that was in the context of the smaller and less powerful PC Engines APU series.

The fact that the unit can route at 1Gbit without the VPN seems to indicate some kind of bottleneck but without loading it up and checking the usage it will be hard to guess what else it could be. It may even be some kind of MTU mismatch that's causing one end to suffer when pushing packets through the VPN tunnel?