OPNSense 21.1.8 IPSEC tunnel to Cisco ASA Rsyncing data stalls after 20 minutes

[teka011]

Hello,

• Former setup ASA5506-x to ASA 5505 IPSec tunnel
• Synology NAS on each site.
• I never had issues rsyncing huge amounts of data both ways maxing out the internet egress throughput at 50mbps.
  

I replaced ASA 5506-X on site A with an OPNSense VM on libvirt. The whole setup works fine except for the IPSec tunnel.

Phase 1 and 2 OK.

When I'm rsyncing data in between the Syno NAS, the traffic maxes out the internet upload throughput for 10-20 minutes and then the trafic stalls and drops to low kb/sec. Then RSync fails / stops. I can still reach the Syno NAS on Site B without issues. I need to restart the RSYNC and it lasts again 10-20 min and stalls. Same for Hyper Backup, backing up large amount of data.

I tried syncing just a dozen of 2GB files. RSYNC still stalls and fails after 10-20 minutes.

Site A: Normalization is active. I tried to set the MSS to 1380 (ipv4) on the WAN interface to use the fW scrubber

Site B: the MSS is configured to 1380 bytes (default) on WAN

I also changed different MTU on the WAN interfaces to give it as much as 120 bytes of headers for IPSec. Below 1480 MTU on OPNsense WAN interface, rsync outputs at 450kb/sec.

I tried to disable AES-NI. Same behaviour.

I tried again my 5506-x <> 5505 today and the Syno NAS RSynced without any issues 1tb at a constant nominal 50mbps when it would fail with my OPNSense setup.

I run Opnsense 21.1.8 amd64.

Any idea ?

[mimugmail]

On ASA do you have in timeouts of P1 and P2 also bytecounters or just seconds?
Seems it tries to reestablish after 200MB and fails for whatever reason (logs please)

[teka011]

I also tried a live 21.7 OPNSense - super small and clean config.
I configured only WAN IP, LAN IP and IPSEC site A to site B.
I ran again a RSYNC from NAS site A to site NAS site B.

11 minutes and RSYNC stalled before timing out although the tunnel remained UP and NAS site B is always reachable.

I will bring in some logs of OPNSense 21.1.8 and ASA 5505. I will try beforehand ASA 5506-X on site B (instead of 5505 (not supported anymore).

I'll post all logs.

[mimugmail]

On ASA, check the tunnel phase2 if there is a byte value and zero it out...

[teka011]

I did replace the ASA 5505 for ASA5506-X. Same behaviour occurs.

ASA logs attached.
opnsense ipsec logs attached too.

I did comment the logs ### and you will see the output when it stalls.

I'm also checking to zero out the bytevalue in P2...

[teka011]

@mimugmail, on ASA only seconds for timeouts.

I tried IKEV1. I ran into the same issue.

Disabling reauth and rekey on phase 1 seems to have fixed the issue.

The default values might have been different from Cisco ASAs. I'm not sure why it would cause the Rsync to stall.

I'm letting the ASA Side rekey/reauth.

If you have any comments, let me know.