How to Gracefully Recover from Wireguard Tunnel Stall

[statoshi]

I've been running OPNSense 23.1 for the past month with a selective routing setup so that traffic from most of my home network devices gets sent through a wireguard endpoint I control on a VPS. This has been working great, except about once a week the tunnel will stall and the gateway for my tunnel will go down, thus triggering my kill switch.

When this occurs I haven't been able to find a specific error. What I can see is that my tunnel client in OPNSense stops performing handshakes. If I stop and restart the tunnel, it re-establishes the connection fine. If I reboot OPNSense it also recovers fine. Has anyone else experienced this problem and, if so, is there a cron job / setting / automation I can configure that will detect such a stall and kick the wireguard service?

[ChirpyTurnip]

So this happened to me last night - two Wireguard tunnels - both stopped within seconds of each other. I suspect it's just WG grinding to a halt - there were no errors or warnings....one moment it was working, the next I get an alert from monit that the remote ping to one of the end points is down. While looking at it the second connection went down.


I restarted the WireGuard service and everything came back as expected. I suspect we can use monit to trigger a wireguard restart, the trick however is to get it to restart only when the problem is on our end - if the tunnel goes down because a remote end is down we don't need/want to restart everything on our side (thereby breaking working connections, and possibly starting a restart loop as the tunnel would stay down until the remote end it back up).


In my case I'd trigger on *both* remote tunnels being down at the same time - which is really unlikely to be coincidental....the only issue is that I'm not nearly smart enough to configure that without guidance. And Wireguard is not easy to work with, so there's no easy option to set links to be monitored for restart....so I think scripts needed....which is a pain if you're a GUI kinda user....  :-(


But at least you know you're not alone!

[chemlud]

https://forum.opnsense.org/index.php?topic=32232.0

maybe? Should be part of the WG opnsense documentation imho... :-)

[ChirpyTurnip]

Well now that's interesting! I've turned that on and we'll see how we go. Once again I'm saddened that there is a nice (I assume) solution and yet it's not plumbed into the GUI for the service. I posted on the poor UI design previously (and a few people took exception to the fact that I though that usability and UX should be a key consideration) so this is another example where the goodness is there but is obfuscated and hidden.


For example with ACME certificates you can tick the box to keep your certs updated and a CRON job is added - no effort needed! But with WG the same could be achieved - a simple tick box that says "Attempt to restart stalled tunnels automatically" - this would be a worthy feature that looks great to an admin (who doesn't like a tick box that implies it fixes everything) but in reality it only really turns on a CRON job. I find it hard to believe that the UI for OPNsense is so bad that something like this would require more effort than it is worth....


Everything just seems unnecessarily hard.


I've now completed my OPNsense migrations, but I'm not going to lie, I sorely miss pfSense, the UI was *****SO****** much better to use. I just don't like their dodgy practices, I've got no confidence their free edition will stay free, and the CE version didn't include the drivers I needed for my newest hardware....but.....sigh......their sweet sweet UI is missed.....  :-(

[chemlud]

I still use both senses, but the opnsense GUI is much more systematic and logical. Just some time needed to adapt... ;-)