IPSEC site2site (new connection mode) extremly slow

[mts]

Hello guys,

I have a policy-based IPSEC-tunnel between two networks which is extremly slow.
Site A) has a 100/100Mbit sync fiber-line
Site B) has a 400/50Mbit async line

I'm currently reaching 4!!! Mbit via my VPN-tunnel.
It is only related to IPSEC-VPN. When I use a openVPN roadwarrior connections everything is fine.
The log files do not show any errors or warnings.

My settings are quite basic:
Settings:
Proposals: aes256-sha512-ecp521 [DH21, NIST EC]
Version: IKEv2
MOBIKE: enabled
DPD delay: 10s
Pools: nothing

Authentication: Public key

Children:
Mode: Tunnel
Policies: enabled
Start action: Start
Stop action: Start
DPD action: Start
ESP-proposals: default
Rekay time: 36600


I also tried to enable IPsec normalization with max MSS of 1350 (as I read this on several forums).
The CPU is more or less non-stop idle.

Statistic IPsec-Interface (enc0):
mtu: 1536
received-errors: 0
dropped-packets: 0
send-errors: 0
collisions: 0

Actually I have no idea why it is that slow.

[Monviech (Cedrik)]

Could you try the same with aes256-sha256-dh14 ? Maybe the CPU is too slow for your encryption settings?

[mts]

Sorry, I forgot to mention:
The CPU is more or les idle all the time.

The "slow" opnsense on site B) is a Celeron J4125 Quad Core while the "strong" opnsense on site A) is much larger...


> To be sure I changed to aes256-sha256-modp2048[dh14] > no difference.

[Monviech (Cedrik)]

How do you test this? Did you use iperf3?

[mts]

Unfortunately I did not use iperf3 as I'm not sure how to use it.
My ipsec-VPN is policy-based, so the is no interface to bind on.

How do I use iperf without an ethernet-interface?

[Monviech (Cedrik)]

Well the policy connects at least two networks that are on these firewalls.

So on one firewall you do "iperf3 -s"

and on the other you do "iperf3 -c 192.168.1.1"   (the IP is an interface that the policy allows connection to on your other firewall, so for example the interface IP of your LAN NET)

Make sure your firewall allows tcp 5201. (Best allow it in floating rules temporarily on the interfaces IPsec/LAN)

[mts]

I have to do a one-eighty...

I used iperf now it it shows absolutely no problem... I have all the bandwidth that is available.
I don't know where my problem is.

I'm using remote-desktop from a Mac on site B) to a windows machine on site A).
It is very laggy and slow. Unter "Connection information" it shows a bandwidth of around 4Mbit.
As soon as I open a openVPN roadwarrior-connection from my Mac on site B) it shows 30Mbit.

a) My Mac does something strange
or
b) opnsense somehow slows down data while routing.

[Patrick M. Hausen]

MTU/MSS mismatch?

[mts]

I don't see the problem.
perf3 from firewall to firewall shows no problem and I have the full bandwidth, but as soon as I want to use any service like file-transfer, remote-desktop, ... the connection is EXTREMLY slow.

Any hint where I could take a look?

[Patrick M. Hausen]

Try to set an MSS size of e.g. 1300 just for test purposes on the LAN (or differently named "private") interfaces on both sides. If that fixes the problem, investigate what the proper MSS size needs to be.

[mts]

Actually that did the trick.
I've set it to 1300 (which is in my opinion pretty small) and it works perfectly.

Shoudn't I limit this normalization to traffic between those networks?

[Patrick M. Hausen]

So now you need to find the WAN MTU of the two systems (probably either 1500 or 1492 if PPPoE is involved) and subtract 60 for the outer IP and ESP headers. The real overhead can be as small as 50 but depends on the payload packet size, so 60 is a good value that will always work.

EDIT: depending on your cryptographic alhorithms the overhead can be even larger. So you should probably test it with ping like so: https://yurisk.info/2022/08/30/network-mtu-maximum-size-testing-with-ping-on-linux-and-windows/

Or find out why PMTUD does not work automatically for you. Are both VPN gateways OPNsense? Are you limiting ICMP in any way?