IPsec and max MSS questions

[clownschiff]

Hi all,

I have performance problems with RDP with some of our virtualized OPNsenses. These firewalls are hosted in a virtual environment and the clients are all connected via IPsec Site2Site tunnels.

[Client]----[Firewall]--------IPsec--------[OPNsense]----[RDP-Server]

The hoster suggested to set the MSS to 1300 for IPsec connections. Which I did in Firewall -> Settings -> Normalization -> Max mss 1300 for the IPsec interface. To test if this setting works, I tried to ping over the tunnel with a payload bigger than 1300 and the "Don't Fragment" flag.

Code Select Expand

ping -f -l 1472 4.3.2.1

I can ping with a size up to 1472 over the tunnel, which should not be possible right? Or do I have to set this on the LAN interface, too? I'm also puzzled how this is possible at all, if the hoster says that 1300 is their max.

[meyergru]

Be careful with ping flags as they are different on different OSes: "-f" means flood ping on FreeBSD, also "-l" is not packet size, but preload.

With FreeBSD, you need "-D" if you want to disable fragmentation and "-s" for size.

[clownschiff]

Thank you for your reply!

I forgot to mention that I used the standard Windows ping for my tests. So -f should be Don't fragment und -l should be size.

Is there any other method I can test if just setting the MSS on the IPsec interface is sufficient?

[meyergru]

The protocol is notorious for all kinds of performance problems, especially over WAN connections. I have two registry files here to make it work without freezes by having it use TCP only:

Code Select Expand


Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services\Client]
"fClientDisableUDP"=dword:00000001


Code Select Expand


Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp]
"SelectTransport"=dword:00000001


When RDP is done via the TCP protocol, re-fragmentation, if needed, should be handled by OpnSense via MSS clamping if the MSS between LAN and WAN differ. This will have a slight impact, but hardly perceivable.

[clownschiff]

Thank you for the registry keys! We just started to investigate the problems, but this could be very helpful going forward.