SMB copy breaks IPSec

[vOoPtNa]

Hey Guys,

I want to report some strange behavior which I've seen on different OPNSense-Boxes in the past weeks/month:

Sometimes IPSec-tunnels just simply get stuck somehow....
"stuck" means: tunnel is still established but no packets flow through the IPSec-interface. I still see ESP-packets on the WAN-interface of the partner-box, but nothing is visible in the IPSec-interface. Absolutley nothing in the IPSec-Log.

Currently I'm troubleshooting an IPSec-tunnel between two OPNSense-boxes(both 20.7.4, one physical one virtual):
Tunnel goes up normally without any errormessages in the logs.
Now the strangest thing in this issue: As soon as i start to copy a file(approx. 100MB) from SiteA to SiteB via SMB the tunnel get "stuck". The only solution is to restart the tunnel.

Some other services like RPC aren't working as well without setting MSS to something lower then 1360.

I tried everything came to my mind:
- changed tunnel from policy-based to route-based
- changed MTU/MSS on all involved interfaces - higher<->lower, doesn't make any difference
- tried to fix via normalization-rules
- changed from IKEv1 to v2, completley recreated the tunnels
- setup a completley new virtual opnsense
- ....

I'm out of ideas at the moment...

it must be related to packet-sizes/fragmentation, but i can't find a solution... and i absolutly have no explaination for a tunnel breaking down due to SMB-copy....
at the moment it is working fine with an OpenVPN-S2S-Tunnel, but i really want to find a fix for this IPSec-issues.

I'm working with OPNSense about 3 years now, managing dozends of boxes with dozends of tunnels. There were problems with IPSec since the beginning of my OPNSense-journey (mostly packet-size related). Some tunnels just work, others don't do. I couldn't figure out the differences. Many times it was just fixed by lowering the MTU to 1450. (especially with PPPoE involved - anyone can explain why?)

Last week I had a similar-problem with a few tunnels on anther 20.7.4 box. Everytime i restarted the IPSec-service 1-3 ICMP-Pings got trough the tunnel. Then "stuck" as described  above.
Could be the same root-cause, but i didn't troubleshooting that in deep, as i fixed this via a second opnsense, which manages the IPSec-Tunnels now.

Over the last month, especially since 20.1, I think the MTU-problems over IPSec are more freqently.

If anyone has an idea how to troubleshoot this further, please share your thoughts/experieneces.

Thanks in advance!

[pupadmin]

Hi vOoPtNa,

is this still an issue with 21.1?
I had a lot of trouble with IPsec-tunnels due to MTU and MSS, and the best solution I came up with was using a routed tunnel with setting MTU=MSS=1350 on the VTI interfaces to actually get the tunnel fragment the packets *before* they enter the tunnel (aka prefragmentation).
This needs some additional pf-rules (on IPsec and the VTI interfaces) to let the fragments and reassembled packets pass (reassembled packets do not seem to carry over the pass flag), but now the traffic is mostly stable (well, there's still a checksum issue, but this is something else...)