Slow traffic going through the firewall

[k4ngoo]

Hello Everyone,
For the last few days, I'm seeing very slow traffic when going though the firewall, an OPNSense (v.20.1.8 ) hosted on OVH's Public Cloud and filtering/routing traffic between private networks. I didn't changed anything on OPNSense configuration.

CPU is idle most of the time and there is plenty of free RAM.

• Load average : 0.27, 0.20, 0.17
• Memory usage : 12 % ( 960/7963 MB )

I read a lot of post regarding performance, so I can already tell you that I disabled proxy and IPS.
Also ran iperf3 test across the network :
From Client (OpenVPN client) to Server :

Code: [Select]

Accepted connection from x.x.0.2, port 56960
[  5] local x.x.12.51 port 5201 connected to x.x.0.2 port 56962
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  9.83 MBytes  82.4 Mbits/sec
[  5]   1.00-2.00   sec  10.7 MBytes  89.5 Mbits/sec
[  5]   2.00-3.00   sec  10.6 MBytes  89.0 Mbits/sec
[  5]   3.00-4.00   sec  10.3 MBytes  86.2 Mbits/sec
[  5]   4.00-5.00   sec  11.1 MBytes  92.7 Mbits/sec
[  5]   5.00-6.00   sec  10.8 MBytes  90.9 Mbits/sec
[  5]   6.00-7.00   sec  8.11 MBytes  68.0 Mbits/sec
[  5]   7.00-8.00   sec  10.9 MBytes  91.8 Mbits/sec
[  5]   8.00-9.00   sec  10.3 MBytes  86.3 Mbits/sec
[  5]   9.00-10.00  sec  10.6 MBytes  88.7 Mbits/sec
[  5]  10.00-10.04  sec   445 KBytes  84.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.04  sec   104 MBytes  86.6 Mbits/sec                  sender
[  5]   0.00-10.04  sec   104 MBytes  86.6 Mbits/sec                  receiver
All is fine.
But from Server to Client (-R option on iperf) :

Code: [Select]

Accepted connection from x.x.0.2, port 57062
[  5] local x.x.12.51 port 5201 connected to x.x.0.2 port 57063
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  5]   0.00-1.00   sec  34.4 KBytes   282 Kbits/sec   10   2.65 KBytes
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   2.65 KBytes
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   2.65 KBytes
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   2.65 KBytes
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   2.65 KBytes
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   2.65 KBytes
[  5]   6.00-7.00   sec  26.5 KBytes   217 Kbits/sec   11   2.65 KBytes
[  5]   7.00-8.00   sec  79.4 KBytes   651 Kbits/sec   17   2.65 KBytes
[  5]   8.00-9.00   sec  71.5 KBytes   585 Kbits/sec   16   2.65 KBytes
[  5]   9.00-10.00  sec  82.0 KBytes   672 Kbits/sec   14   2.65 KBytes
[  5]  10.00-10.05  sec  0.00 Bytes  0.00 bits/sec    0   3.97 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.05  sec   294 KBytes   240 Kbits/sec   70             sender
[  5]   0.00-10.05  sec   262 KBytes   214 Kbits/sec                  receiver

I also tried between two hosts on same vlan : 100Mbit/s in both direction.

And between two hosts in different VLAN (routing through OPNSense), traffic is slow (few Mbps instead of 100Mbps), but better than through VPN :

Code: [Select]

Accepted connection from x.x.11.51, port 37098
[  5] local x.x.12.51 port 5201 connected to x.x.11.51 port 37100
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   187 KBytes  1.53 Mbits/sec
[  5]   1.00-2.00   sec   543 KBytes  4.45 Mbits/sec
[  5]   2.00-3.00   sec   608 KBytes  4.98 Mbits/sec
[  5]   3.00-4.00   sec   655 KBytes  5.36 Mbits/sec
[  5]   4.00-5.00   sec   450 KBytes  3.68 Mbits/sec
[  5]   5.00-6.00   sec   793 KBytes  6.50 Mbits/sec
[  5]   6.00-7.00   sec   768 KBytes  6.29 Mbits/sec
[  5]   7.00-8.00   sec   601 KBytes  4.92 Mbits/sec
[  5]   8.00-9.00   sec   492 KBytes  4.03 Mbits/sec
[  5]   9.00-10.00  sec   638 KBytes  5.22 Mbits/sec
[  5]  10.00-10.04  sec  22.6 KBytes  4.65 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.04  sec  5.75 MBytes  4.81 Mbits/sec  749             sender
[  5]   0.00-10.04  sec  5.62 MBytes  4.70 Mbits/sec                  receiver

Using top, I can see :

Code: [Select]

CPU:  0.0% user,  0.0% nice,  0.2% system,  1.3% interrupt, 98.5% idleInterrupt is going to 2-3% at max. Is it a problem?

My feeling is that packet processing take more time than it should, which reduce the bandwidth. How can I debug that and guess why it changed like that.

Thank you for your help,
K4ngoo

[k4ngoo]

Found the issue: "Hardware checksum offload" was not deactivated.

Strange thing, the setting was not even present in the configuration. I'm guessing that this setting appear in a recent update and was not set to default, which is activated.

Anyway, my question remains: how would you trace a packet though the OPNSense box to debug this kind of behavior?