OpenVPN(?) freezes system under load

[Natureshadow]

Hi,

we recently added an OpenVPN service to our firewall setup.

Base setup: Two firewalls, one on hardware, one virtualised, as primary/secondary with some interfaces in CARP failover.

The setup was running very stable, until we added an OpenVPN service. Ever since we configured and started OpenVPN on both firewalls, the systems irregularly freeze (both the hardware and the virtualised firewall). It seems to happen when the OpenVPN connection has some traffice for some time.

The OpenVPN setup is a bit obscure, but not obscure enough to justify a system freeze. We only use OpenVPN to open a TAP device, which is bridged to a VLAN interface on the firewall. OpenVPN des not handle addresses, it merely provides the layer 2 connectivity. The OpenVPN service is synced to both firewalls (and works on both firewalls).

Now, when the primary firewall is active, and the (one) OpenVPN client is connected, everything still works fine for some time, but once there is a bit more traffic on the link, at some point the firewall completely freezes (web interface unreachable, all network services unreachable, stops sending CARP packets, local console is unresponsive, failover to secondary firewall happens). It does, however, still reply to ICMP echo requests.

Once penVPN connectivity is back, it takes some time, until there is enough traffic, then the secondary firewall also freezes.

We tried to observe memory consumption and the like, but could not find anything out of the ordinary, not even shortly before a freeze. Also, no logs contain anything helpful, just the normal behaviour until it just stops. We tried toggling hardware crypto offload (normally disabled), but it does not improve the situation.

Any ideas on how to proceed with debugging, or even what could cause such behaviour?

Both ifrewalls run an up-to-date OPNsense 20.1.

Thanks,
Nik