[SOLVED] slow IPsec performance

[fraenki]

Turns out it was some sort of regression in the early 17.7.x series.
After upgrading to 17.7.7 the issue disappeared.


- Frank

[franco]

I find that hard to believe. What version was this tested against? 17.7 exactly?

[fraenki]

I find that hard to believe. What version was this tested against? 17.7 exactly?

Me too, but I couldn't find a better explanation. :)
The boxes were running the ancient 17.7.1_2  before upgrading to 17.7.7.

I've already diffed the config.xml backups, but there was no difference.


Regards
- Frank

[ivoruetsche]

Hi all

In the last days, we want to exchange our Cisco ASA's with opnsense boxes, but we run in this issue. Our old configuration is a ASA5520 <-> ASA5540 with IPSec tunnel, the new are two opnsense boxes with these CPU's "AMD GX-412TC SOC (4 cores)" <-> "Intel(R) Xeon(R) CPU E3-1280 V2 @ 3.60GHz (8 cores)", but i don't think this is the problem, also with an IPSec tunnel.

On the location A is a VM who get a nightly backup from a VM in the location "B". Both are RHEL 7 VM's and we just switch the gateway from the ASAs to the opnSense Boxes. Routers (Cisco and Draytek), Lines, Switches on both locations are untouched.

The VM in the location B have also a public IP on the opnsense FW. So if we try to download a file from the web server or to a scp via the IPSec tunnel, we just get s hand full of KB/s (scp says "--stalled--"), via the public IP full speed. At the beginning the transfer is fast for a short moment, then it comes down very fast. But not all the time sometimes, it goes a bit longer fast before "stalled".

The tunnel on the ASA side was IPSecV1, on opnsense IPSecV2, both ESP etc.

gruss ivo

[mimugmail]

Hi all

In the last days, we want to exchange our Cisco ASA's with opnsense boxes, but we run in this issue. Our old configuration is a ASA5520 <-> ASA5540 with IPSec tunnel, the new are two opnsense boxes with these CPU's "AMD GX-412TC SOC (4 cores)" <-> "Intel(R) Xeon(R) CPU E3-1280 V2 @ 3.60GHz (8 cores)", but i don't think this is the problem, also with an IPSec tunnel.

On the location A is a VM who get a nightly backup from a VM in the location "B". Both are RHEL 7 VM's and we just switch the gateway from the ASAs to the opnSense Boxes. Routers (Cisco and Draytek), Lines, Switches on both locations are untouched.

The VM in the location B have also a public IP on the opnsense FW. So if we try to download a file from the web server or to a scp via the IPSec tunnel, we just get s hand full of KB/s (scp says "--stalled--"), via the public IP full speed. At the beginning the transfer is fast for a short moment, then it comes down very fast. But not all the time sometimes, it goes a bit longer fast before "stalled".

The tunnel on the ASA side was IPSecV1, on opnsense IPSecV2, both ESP etc.

gruss ivo

This sounds like a classic MTU issue. I wrote some hints regarding MTU in this thread, just try them.

[ivoruetsche]

Salü mimugmail

Yes, it was also my first intend and I tried some modifications, but I never touched the MSS field. Thank you to give me the kick to go through the thread again. It's solved now, here is how:

I try with ping, what is the biggest size where I got a reply:

[root@linux01 ~]# ping -s 1395 -M do 198.18.8.48
PING 198.18.8.48 (198.18.8.48) 1395(1423) bytes of data.
^C
--- 198.18.8.48 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4000ms

[root@linux01 ~]#

[root@linux01 ~]# ping -s 1394 -M do 198.18.8.48
PING 198.18.8.48 (198.18.8.48) 1394(1422) bytes of data.
1402 bytes from 198.18.8.48: icmp_seq=1 ttl=62 time=15.2 ms
1402 bytes from 198.18.8.48: icmp_seq=2 ttl=62 time=15.0 ms
1402 bytes from 198.18.8.48: icmp_seq=3 ttl=62 time=14.3 ms
^C
--- 198.18.8.48 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.316/14.856/15.203/0.386 ms
[root@linux01 ~]#

I set the MSS value "1422" from the "good" ping on the LAN interface on the opnsense and the performance was fine. I try to set it only on location A and it works, but this location is connected via DSL.

I also had a look on the Cisco ASA, there was no settings for MSS, so the default value should be active:
"By default the ASA sets the TCP MSS option in the SYN packets to 1380." (from Cisco).

gruss ivo

[mimugmail]

You can double check the behavior with a simple tcpdump on the OPNsense on the LAN interface.
Do a telnet to an open port behind the ASA network and you'll see the MSS size in the 3way handshake.

You client sets mss 1460 (LAN)
06:11:46.357659 IP 192.168.19.30.55704 > 192.168.169.5.445: Flags [SEW], seq 972300290, win 29200, options [mss 1460,sackOK,TS val 151542601 ecr 0,nop,wscale 5], length 0

The server replies with 1460.
06:11:46.357910 IP 192.168.169.5.445 > 192.168.19.30.55704: Flags [S.E], seq 62474480, ack 972300291, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 3002803536 ecr 151542601], length 0

If the ASA modifies this value you'll see a decremented number.

[Buran Ded]

Cisco ASA 55XX by default always transparently fix MSS to 1380 (norm MSS=1460)
on all transiting TCP connections on all interfaces and all RA/S2S VPNs.

This default setting is not visible in config.

But visible via ASDM:

[Buran Ded]

I have some experiments with OpenVPN,
and i see something like ASA:
OpenVPN also defaultly transparently fixing MSS of all transiting TCP-traffic,
but fixing to MSS=1410.

[Buran Ded]

Today i see by own eyes: opnsense reduces MSS by 100 (to 1360).

Traffic capture on webserser side:

22:14:40.305764 30:e4:db:xx:xx:xx > 52:54:00:00:xx:xx, ethertype IPv4 (0x0800), length 66: opn-s-ip.16825 > web-server-ip.80: Flags [ S ], seq 4280183093, win 8192, options [mss 1360,nop,wscale 2,nop,nop,sackOK], length 0

22:14:40.305855 52:54:00:00:xx:xx > 30:e4:db:xx:xx:xx, ethertype IPv4 (0x0800), length 66: web-server-ip.80 > opn-s-ip.16825: Flags [ S. ], seq 3520754639, ack 4280183094, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0

[ivoruetsche]

Hi all

We have also some DSL <-> Fiber VPN's (both side connected by pppoe via a bridge/converter to the provider) and we need to reduce the MSS to 1380 on the LAN side. Without, RDP is useless.

gruss ivo