Hello Everyone,
For the last few days, I'm seeing very slow traffic when going though the firewall, an OPNSense (v.20.1.8 ) hosted on OVH's Public Cloud and filtering/routing traffic between private networks. I didn't changed anything on OPNSense configuration.
CPU is idle most of the time and there is plenty of free RAM.
- Load average : 0.27, 0.20, 0.17
- Memory usage : 12 % ( 960/7963 MB )
I read a lot of post regarding performance, so I can already tell you that I disabled proxy and IPS.
Also ran iperf3 test across the network :
From Client (OpenVPN client) to Server :
Accepted connection from x.x.0.2, port 56960
[ 5] local x.x.12.51 port 5201 connected to x.x.0.2 port 56962
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 9.83 MBytes 82.4 Mbits/sec
[ 5] 1.00-2.00 sec 10.7 MBytes 89.5 Mbits/sec
[ 5] 2.00-3.00 sec 10.6 MBytes 89.0 Mbits/sec
[ 5] 3.00-4.00 sec 10.3 MBytes 86.2 Mbits/sec
[ 5] 4.00-5.00 sec 11.1 MBytes 92.7 Mbits/sec
[ 5] 5.00-6.00 sec 10.8 MBytes 90.9 Mbits/sec
[ 5] 6.00-7.00 sec 8.11 MBytes 68.0 Mbits/sec
[ 5] 7.00-8.00 sec 10.9 MBytes 91.8 Mbits/sec
[ 5] 8.00-9.00 sec 10.3 MBytes 86.3 Mbits/sec
[ 5] 9.00-10.00 sec 10.6 MBytes 88.7 Mbits/sec
[ 5] 10.00-10.04 sec 445 KBytes 84.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.04 sec 104 MBytes 86.6 Mbits/sec sender
[ 5] 0.00-10.04 sec 104 MBytes 86.6 Mbits/sec receiver
All is fine.
But from Server to Client (-R option on iperf) :
Accepted connection from x.x.0.2, port 57062
[ 5] local x.x.12.51 port 5201 connected to x.x.0.2 port 57063
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 5] 0.00-1.00 sec 34.4 KBytes 282 Kbits/sec 10 2.65 KBytes
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 1 2.65 KBytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 2.65 KBytes
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 2.65 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 2.65 KBytes
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 2.65 KBytes
[ 5] 6.00-7.00 sec 26.5 KBytes 217 Kbits/sec 11 2.65 KBytes
[ 5] 7.00-8.00 sec 79.4 KBytes 651 Kbits/sec 17 2.65 KBytes
[ 5] 8.00-9.00 sec 71.5 KBytes 585 Kbits/sec 16 2.65 KBytes
[ 5] 9.00-10.00 sec 82.0 KBytes 672 Kbits/sec 14 2.65 KBytes
[ 5] 10.00-10.05 sec 0.00 Bytes 0.00 bits/sec 0 3.97 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-10.05 sec 294 KBytes 240 Kbits/sec 70 sender
[ 5] 0.00-10.05 sec 262 KBytes 214 Kbits/sec receiver
I also tried between two hosts on same vlan : 100Mbit/s in both direction.
And between two hosts in different VLAN (routing through OPNSense), traffic is slow (few Mbps instead of 100Mbps), but better than through VPN :
Accepted connection from x.x.11.51, port 37098
[ 5] local x.x.12.51 port 5201 connected to x.x.11.51 port 37100
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 187 KBytes 1.53 Mbits/sec
[ 5] 1.00-2.00 sec 543 KBytes 4.45 Mbits/sec
[ 5] 2.00-3.00 sec 608 KBytes 4.98 Mbits/sec
[ 5] 3.00-4.00 sec 655 KBytes 5.36 Mbits/sec
[ 5] 4.00-5.00 sec 450 KBytes 3.68 Mbits/sec
[ 5] 5.00-6.00 sec 793 KBytes 6.50 Mbits/sec
[ 5] 6.00-7.00 sec 768 KBytes 6.29 Mbits/sec
[ 5] 7.00-8.00 sec 601 KBytes 4.92 Mbits/sec
[ 5] 8.00-9.00 sec 492 KBytes 4.03 Mbits/sec
[ 5] 9.00-10.00 sec 638 KBytes 5.22 Mbits/sec
[ 5] 10.00-10.04 sec 22.6 KBytes 4.65 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-10.04 sec 5.75 MBytes 4.81 Mbits/sec 749 sender
[ 5] 0.00-10.04 sec 5.62 MBytes 4.70 Mbits/sec receiver
Using top, I can see :
CPU: 0.0% user, 0.0% nice, 0.2% system, 1.3% interrupt, 98.5% idleInterrupt is going to 2-3% at max. Is it a problem?
My feeling is that packet processing take more time than it should, which reduce the bandwidth. How can I debug that and guess why it changed like that.
Thank you for your help,
K4ngoo
Found the issue: "Hardware checksum offload" was not deactivated.
Strange thing, the setting was not even present in the configuration. I'm guessing that this setting appear in a recent update and was not set to default, which is activated.
Anyway, my question remains: how would you trace a packet though the OPNSense box to debug this kind of behavior?