Very Slow IPSEC bandwith

[Nicolassimond]

Hello,

We have two OPNsense DEC3840 running the business edition
Here is the information on both of them:
OPNsense 21.4.3-amd64
FreeBSD 12.1-RELEASE-p19-HBSD
OpenSSL 1.1.1k 25 Mar 2021
AES-SNI enabled

We have an IPSec tunnel with the following settings:
PH1 : 128 bit AES-GCM with 128 bit ICV + SHA256 + DH Group 28
PH2 : aes128gcm16 + + 28 (Brainpool EC 256 bits)

I have tested different combination with and without hash and everything, and it doesn't seem to impact IPSEC performance.

We have 1Gbps professional connection on both side, and we only get 100mbps throughput on IPSEC (tested on smb copy, iperf).

Any idea of what is blocking IPSEC performance? The cpu usage doesn't move.


Thanks

[mimugmail]

Interfaces LAN MSS set to 1300

[linuxmail]

hi,

any news here ? We have the same problem .. with DEC3850 .. and we have around ~2,5 - 3Gb/s. Also for the network, which goes  **not** over the tunnel. What I found:

https://www.mayrhofer.eu.org/post/firewall-throughput-opnsense-openwrt/

When IPsec is active - even if the relevant traffic is not part of the IPsec policy - throughput is decreased by nearly 1/3. This seems like a real performance issue / bug in the FreeBSD/HardenedBSD kernel. I will need to try with VTI based IPsec routing to see if the in-kernel policy matching is a problem.

If we don't go over the applicance / OpenSense .. we hit the 10Gb/s limit.

[franco]

That reminds me of https://github.com/opnsense/src/commit/542970fa2d3fb4

But 20% from 2,5-3 is not 10 though the real question is how realistic that assumption is considering IPsec is running in the first place and may be a feature from the firewall is used. Or I'm reading this wrong...

The main question though.. which version? Anything below 22.1 will likely be the same

Looking at the OP saying 0,1 GB/s I'm not sure what we are comparing here...


Cheers,
Franco

[mimugmail]

hi,

any news here ? We have the same problem .. with DEC3850 .. and we have around ~2,5 - 3Gb/s. Also for the network, which goes  **not** over the tunnel. What I found:

https://www.mayrhofer.eu.org/post/firewall-throughput-opnsense-openwrt/

When IPsec is active - even if the relevant traffic is not part of the IPsec policy - throughput is decreased by nearly 1/3. This seems like a real performance issue / bug in the FreeBSD/HardenedBSD kernel. I will need to try with VTI based IPsec routing to see if the in-kernel policy matching is a problem.

If we don't go over the applicance / OpenSense .. we hit the 10Gb/s limit.

In your previous cross-posting you tested multiple scenarios where proxmox to proxmox with different vlan was slow. Does it go back to normal when you disable IPsec? This is not clear from your multiple posts. (better to open a new one btw.)

[linuxmail]

Hi,

Our OpnSense (DEC-3850) is at the moment: 21.10

But 20% from 2,5-3 is not 10 though the real question is how realistic that assumption is considering IPsec is running in the first place and may be a feature from the firewall is used. Or I'm reading this wrong...
Cheers,
Franco

because of a maintenance from our datacenter provider, we where able to shutdown IPSec VPN and tested quickly again with iperf3 from one VLAN to another VLAN, so it goes over the OpnSense appliance. We had ~1Gb/s more throughput. So instead of ~3Gb/s we had nearly ~4Gb/s.

I'm not sure, if we can reach max ~5Gb/s in theory, because the traffic has to go twice over the same OpnSense Interface (which is a LACP 2x10Gb/s). But .. one important thing came into my mind: Before we switched to the DEC-3850, we had a real server (Supermicro x11SSH-LF) and reached ~5Gb/s. But anyway ...

The other mention you found from me (just found that thread after this one and  I've found out, that traffic from one VLAN to an other one is also pretty slow ): It is the same setting, sorry if it was not clear. I've tested every combination, in the moment, OpnSense jumps in .. the throughput breaks down. The question for me is: is that expected, that the "speed" goes under 50%, from what is in theory possible.


cu denny