Wireguard vs. DNS resolution (racing each other at boot time)

[meyergru]

Your latest tcpdump provides an important clue: Even in the broken state, DNS queries from 10.127.127.2 reach 10.127.127.1, and valid DNS responses are sent back through wg4.

This suggests that Unbound is working. The problem is more likely related to packet delivery, firewall rules, or routing/gateway selection during startup.

Since your setup involves multiple gateways and policy-based routing (PBR), I particularly suspect a routing or firewall state issue during boot, possibly involving incorrect gateway selection or asymmetric routing.

Could you perform the following tests before restarting WireGuard, while simultaneously running `tcpdump -ni wg4 'host 10.127.127.2'` on OPNsense?

1. Explicit DNS lookup: Query `10.127.127.1` directly from the affected client (e.g. `nslookup example.com 10.127.127.1`). If this fails despite DNS responses appearing in the capture, the problem is likely on the return path to the client or in the client's DNS stack. If it succeeds, DNS transport itself is working.

2. Direct IP connection: Try accessing a known reachable Internet server by IP address, bypassing DNS entirely. If this fails while DNS works, suspect forwarding, NAT, firewall rules, or gateway selection. The capture will show whether the connection attempt reaches wg4.

3. Connection by hostname: Access the same server by hostname. If direct IP access works but hostname access fails, investigate the client's DNS configuration or resolver behavior rather than general IP routing.

Given your multiple gateways and PBR rules, I would also compare the firewall states and gateway selection for the affected client traffic before and after restarting WireGuard.

These tests should distinguish DNS transport issues from forwarding/routing problems and incorrect client-side name resolution.

I would avoid introducing startup delays until we have identified the failing component.

[oc]

Hallo meyergru - just wanted to follow up on this one as it's been a while. Perhaps my solution will help someone who experienced the same issue (slow boot due to Wireguard being bogged down by DNS resolution when setting up outbound wireguard connections to URLs rather than IPs - and when that happened, inbound Wireguard clients dialing into the opnsense had issues with DNS resolution). This behavior has been reported by others, and various LLMs agreed that it's a known issue, offering just some clunky workarounds (e.g., scripted delayed wireguard). I decided not to go that way.

What I ended up doing was using an external Wireguard router (a Flint 2) which takes care of all outbound Wireguard connections (segregated by VLANs and different igb NIC connections). I have also left the inbound Wireguard instances to opensense, from both another remote Flint 2 carrying quite a bit of traffic, and various mobile devices (such as phones tunneling all traffic through opnsense/zenarmour) - and all works well with just inbound Wireguard. The opnsense is booting fast and external Wireguard clients have no issue connecting to it and getting good DNS resolution.

I still have a (much smaller) issue, also documented - dpinger is giving up if thinking a gateway is down at boot time, and fails to add the respective (already defined) static route to the remote network connected via Wireguard to opnsense's routing table. When that happens, I cannot reach the affected remote network, and have to restart the gateway in question (but once I do that, traffic resumes right away - and I can do it either locally or connected to the opnsense over VPN so no super-big biggie, but it's still annoying in a setup that attempts to be as enterprise-y as possible). Online research suggests this behavior is freebsd-specific. In my case this happens due to a chicken-and-egg situation between opnsense and the Flint2 - because the Flint2 is going online through the opnsense - so obviously while the opnsense is rebooting there's no connectivity - and if the opnsense checks the gateways going through the Flint2 while booting - sometimes they are unavailable and thus the vicious circle. But, like I said, this only happens occasionally, and the fix is one click away, and can be done remotely. But while it works it's a quite cool setup, because the Flint2 does both inbound/outbound VPN, with various levels of Kill switches, and its own traffic and internet access is protected by opnsense/zenarmor/crowdsec.

Thank you meyergru for the steady assistance. I really appreciate it!

Have a great weekend, opnsense community!
OC