Wireguard vs. DNS resolution (racing each other at boot time)

[oc]

Hello All,

First post here. I've been using OPN for about six months now, coming from various enterprise firewalls, and love it. Thank you Devs and community!

My setup is pretty complex, with multiple gateways pointing to various wireguard tunnels, routes, VLANs, policy-based routing, inbound wireguard server for mobile devices, ZA, crowdsec - and all works, and (fingers crossed) so far I've been spared of issues when updating/patching it.

However, one recurring issue I've experienced is that when I reboot the firewall (typically after an update), the following sequence happens:

1. watching it coming up in the console, wireguard takes a very long time starting (minutes) - but eventually it does

2. upon finally starting, S2S wireguard links are working (there are networks in a couple of other cities going through this OPN), but client VPNs (ex. mobile devices that always go through the OPN for protection when not in the home network) are connecting but can't resolve (so they can ping both internal as well as Internet IPs using the OPN public IP, but they cannot forward lookup external names)

3. the client issue reported at #2 is resolved immediately by restarting the inbound wireguard 'server' service (and I can do that even from afar, initially connecting to the tunnel and then to the OPNs internal management IP). And once the service restarts, all's good until the next reboot (and in fact, during some reboots wireguard got lucky and started after DNS so all was OK - but it's impredictable).

I have researched this online, and it appears to be because wireguard attempts to start before DNS (Unbound, I guess) is fully up and running, which leaves wireguard in an "in-betweeen" state (on top of delaying the startup by a lot of time, which is also very inconvenient).

The solutions suggested online:

a) do not use dynamic IPs names in tunnel names (aliases). While having static IPs is always desirable for VPN tunnel gateways, it's not always possible, as some ISPs aren't offering them (or at least not a the same service tier). So this is not really an option for me.

b) script it so that the wireguard service startup is delayed by a while, thereby giving the DNS resolver enough time to properly come up. This then is also supposed to expedite wireguard startup times. And while this is a reasonable solution, I didn't want to do that on my end, but rather pick the Dev's brains as to a more permanent workaround (for instance, there could be a condition testing if external DNS resolution works and only move on to other services relying on DNS, such as wireguard, after it's confirmed that it does). Other scripts baked into OPNsense are great (such as "Renewing DNS for Wireguard on stale connections" and "Reset WAN interface every day at 3am PT"), so perhaps there can be something to resolve this issue as well.

It's quite likely that a lot of users have to rely on dynamic IP name resolution for external VPN gateways, and it's a pity to let this one unaddressed, considering that the 'watchdog' function works brilliantly.

Again - thank you Devs and to the community. You are truly inspiring.

[meyergru]

The "Renew DNS for Wireguard on stale connections" cron job had a problem with that specific DNS outage on startup, but it has been fixed recently, now taking care of it correctly.

This fix is in the CE version, but can manually be applied to the BE as well.

[oc]

Thank you, meyergru.

The "Renew DNS for Wireguard on stale connections" cron job actually works for me - I've had the opportunity to test it by forcing a dynamic IP change on the other end, and the tunnel restored quickly thereafter. And btw, I'm on the CE and up to date (well, 26.7.3_8).

My issue is the slow boot time, and the fact that the inbound WG server doesn't resolve until a service restart. As stated initially, S2S WG works well even before the restart, it's only affecting the client-inbound WG connections.

IMO, it doesn't appear to be related to the "Renew DNS for Wireguard on stale connections" cron job, but rather to the boot sequence itself.

[meyergru]

For S2S connections, the cron job helps. For C2S, there should be nothing keeping your Wireguard server from starting, because the peer does not have to have an endpoint, so there is no DNS to be resolved that could keep the instance from coming up?

[oc]

My understanding is that the "Renew DNS for Wireguard on stale connections" script helps when one of the S2S endpoints with a dynamic IP resolved by means of an alias is changing its IP *while the tunnel is up* (it has nothing to do with a reboot). On other devices wireguard is notorious to fail to re-establish a connection if the dynamic IP at the other end is changing while the tunnel is established (and manual intervention is required - disabling and re-enabling the tunnel brings it back right away). For instance, on Windows I've written a PowerShell script that I ran every 5 minutes via Scheduled Tasks - if the remote gateway was unavailable over ICMP it restarted the tunnel, which reliably re-established the connection. For instance, on GL.iNET devices the watchguard feature is well implemented, it never led to any issues.

But here we're not talking about that watchdog feature, but instead by the OPNsense wireguard server being unable to resolve its clients' DNS queries after rebooting the firewall, and until the service is restarted manually (I say again, S2S connections work well, it's only about clients which connect but cannot resolve). It has nothing to do with client-side dynamic IPs. According to what I found online, it's because the OPNsense wg 'server' service (instance) attempts to run before the DNS resolver is fully up - and the suggestion solution is scripting the wg server to start after a certain delay (they are proposing 30 seconds, which I can live). But the reason I've written here is because I'd like a Devs-endorsed workaround, rather than something I've found online - especially since it's likely that it would help many others as well.