Wireguard Client Dies Seemingly Randomly

[zemsten]

Alright, let me preface this with the fact that I've had a wireguard tunnel up and working to NAT my traffic outbound for a long while now. All of it sudden it has become very inconsistent with no observable changes on my end. I'm on the latest version of OPNSense.

I have a fairly complex network by homelab standards. There are several VLANs on my parent LAN interface, as well as a few separate hardware interfaces, all talking to an old protectli vault running OPNSense. My WAN is a single interface feeding upstream to a separate cellular modem running OpenWRT. Unfortunately there's an extra NAT layer due to that, but that's a different discussion...
I have a wireguard client setup with Mullvad. The config has always worked great. The only thing I changed after setting it up initially was disabling route pulls and configuring a manual gateway, so it wouldn't shove itself in the firewall's routing table as a default route.
I have policy routes setup so that most of my traffic ends up NATing out the wireguard tunnel, with the exception of one entire VLAN, and a couple select hosts elsewhere that just NAT out the modem's gateway. Notably, the firewall itself does not NAT out the wireguard tunnel.

This has always worked until recently.

I'm now having problems where the wireguard gateway will jump to 100% packet loss seemingly randomly. The tunnel dies, and so does the connectivity for those policy routed hosts. If I deactivate and reactivate wireguard, either in the web GUI or with a `wg-quick down wg0 && wg-quick up wg0`, it never gets another handshake. I have to reboot the firewall entirely to get wireguard to come back. I have my endpoint config pointing to an IP address rather than a hostname, so I know it's not DNS. I can still talk to the internet and even that wireguard endpoint specifically from the firewall's CLI, so I know reaching it isn't a problem. Why does wireguard just die though?

A bit more background, although I don't believe it's too important. I did switch to the wireguard kernel module when it became available. I know that it was for testing and potentially unstable. It worked great for weeks, maybe even over a month. When these failures started happening, I uninstalled the kernel module and went back to wireguard-go, just to be in a "fully supported" state so nothing could be blamed on beta testing. The exact same thing happens.

I've gone through what feels like umpteen logs, and I cannot find anything related to the cause. I see the events logged in various places when the wireguard tunnel goes down, but I don't see anything right before that is causing it.

Now I am here, frustrated and desparately asking for the help of this community, as this is almost completely breaking the usability of my network at random intervals.

[Greelan]

Do you have a keepalive set (eg 25 sec) on the Endpoint config on OPNsense?

[zemsten]

I do, it was 30 seconds. I tried changing to 5, and I see the same thing unfortunately.

[zemsten]

I do, it was 30 seconds. I tried changing to 5, and I see the same thing unfortunately.

I'll also mention that at this point I have unset the keepalive, and it still dies.

I'm back on the kernel driver, as I saw mention somewhere of setting a kernel tunable for debugging, net.wg.debug, but I don't seem to have that sysctl OID, even after verifying the kernel has loaded if_wg.ko.

This problem is driving me nuts. Nothing gets logged. I'm on the verge of switching back to openvpn, which would really be a blow to overhead.

[zemsten]

Well, no progress on my front. My ISP is a MVNO, so it's also possible that some traffic is getting blocked/filtered, although it seems unlikely based on previous success.

I need stability more than I need wireguard, so unfortunately I just decided to ditch the setup in lieu of OpenVPN...

Really hope this piques someone's interest though in the search for a similar problem. I'd love to know the solution.