I'm really scratching my head on this one.
Previously I had Wireguard VPN:
- Working very well on my iPhone
- Working very well on my Ubuntu Tablet (not much used though)
- Working VERY DREADFULLY on my Android Phone - even though e.g. Home Assistant Notifications would be sent to the Android Phone, I could never read the Details / open them
Yesterday I had a complete Internet Outage during the upgrade of my OpenWRT Upstream Router. Still unsure what exactly caused it, but basically the OpenWRT Router would NOT route anymore. I later discovered that like half the OpenWRT Services responsible for Routing, DNS, etc got disabled for some weird Reason. I re-enabled, restarted them and rebooted the Router, and that seems to be fixed now.
HOWEVER, in the Attempt to get some Internet back and runnnig yesterday, I connected my Secondary Fiber Connection (which I wanted to setup for a long Time) to the OPNSense Router.
Since then, with Multi-WAN Enabled, and assuming that the Settings are correct in terms of which Gateway to use in which Case, Wireguard VPN is COMPLETELY BROKEN for all Platforms.
No Handshake takes place (or no successfull handshake), I sometimes receive some HomeAssistant Notifications on my iPhone, so not sure what's going on there.
In the Firewall Logs I can see 1 x IN + 1 x OUT Connection occurring as soon as I hit the "Connect" Button on iPhone or I start the VPN Service on my Ubuntu Tablet. Nothing happens though.
After carefully setting up the Routes, at least I got to a Point where the Traffic coming in from one Connection / Upstream Router goes out through the same Connection / Upstream Router.
But it's still NOT handshaking. And still no obvious Stuff that is wrong. There does NOT seem to be anything getting denied in the Logs and I enabled to Log everything as Default).
Network Diagram & Screenshots:
https://imgur.com/a/yEjQs0R
EDIT 1: Excerpt from iPhone Logs
2025-06-24 14:32:35.221349: [APP] Tunnel 'RemoteAccess-FIBER01' connection status changed to 'connecting'
2025-06-24 14:32:35.309875: [NET] App version: 1.0.16 (27)
2025-06-24 14:32:35.310085: [NET] Starting tunnel from the OS directly, rather than the app
2025-06-24 14:32:35.333928: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:32:35.334624: [NET] Attaching to interface
2025-06-24 14:32:35.334960: [NET] Routine: encryption worker 1 - started
2025-06-24 14:32:35.334972: [NET] Routine: decryption worker 1 - started
2025-06-24 14:32:35.334994: [NET] UAPI: Updating private key
2025-06-24 14:32:35.334994: [NET] Routine: decryption worker 4 - started
2025-06-24 14:32:35.335012: [NET] Routine: handshake worker 5 - started
2025-06-24 14:32:35.335042: [NET] Routine: encryption worker 3 - started
2025-06-24 14:32:35.335039: [NET] Routine: handshake worker 4 - started
2025-06-24 14:32:35.335104: [NET] Routine: handshake worker 2 - started
2025-06-24 14:32:35.335104: [NET] Routine: decryption worker 3 - started
2025-06-24 14:32:35.335131: [NET] Routine: handshake worker 3 - started
2025-06-24 14:32:35.335135: [NET] Routine: handshake worker 1 - started
2025-06-24 14:32:35.335142: [NET] Routine: event worker - started
2025-06-24 14:32:35.335149: [NET] Routine: encryption worker 5 - started
2025-06-24 14:32:35.335186: [NET] Routine: encryption worker 4 - started
2025-06-24 14:32:35.335229: [NET] Routine: decryption worker 2 - started
2025-06-24 14:32:35.335252: [NET] Routine: decryption worker 5 - started
2025-06-24 14:32:35.335251: [NET] Routine: decryption worker 6 - started
2025-06-24 14:32:35.335269: [NET] Routine: handshake worker 6 - started
2025-06-24 14:32:35.335269: [NET] Routine: encryption worker 2 - started
2025-06-24 14:32:35.335291: [NET] Routine: TUN reader - started
2025-06-24 14:32:35.335305: [NET] Routine: encryption worker 6 - started
2025-06-24 14:32:35.335407: [NET] UAPI: Removing all peers
2025-06-24 14:32:35.335618: [NET] peer(/bTG...Z6TQ) - UAPI: Created
2025-06-24 14:32:35.335648: [NET] peer(/bTG...Z6TQ) - UAPI: Updating preshared key
2025-06-24 14:32:35.335685: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:32:35.335754: [NET] peer(/bTG...Z6TQ) - UAPI: Updating persistent keepalive interval
2025-06-24 14:32:35.335785: [NET] peer(/bTG...Z6TQ) - UAPI: Removing all allowedips
2025-06-24 14:32:35.335821: [NET] peer(/bTG...Z6TQ) - UAPI: Adding allowedip
2025-06-24 14:32:35.335901: [NET] peer(/bTG...Z6TQ) - UAPI: Adding allowedip
2025-06-24 14:32:35.336247: [NET] UDP bind has been updated
2025-06-24 14:32:35.336261: [NET] Routine: receive incoming v4 - started
2025-06-24 14:32:35.336283: [NET] peer(/bTG...Z6TQ) - Starting
2025-06-24 14:32:35.336304: [NET] Routine: receive incoming v6 - started
2025-06-24 14:32:35.336342: [NET] peer(/bTG...Z6TQ) - Sending keepalive packet
2025-06-24 14:32:35.336359: [NET] peer(/bTG...Z6TQ) - Routine: sequential receiver - started
2025-06-24 14:32:35.336401: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:32:35.336431: [NET] peer(/bTG...Z6TQ) - Routine: sequential sender - started
2025-06-24 14:32:35.337080: [NET] Interface state was Down, requested Up, now Up
2025-06-24 14:32:35.337117: [NET] Device started
2025-06-24 14:32:35.337210: [NET] Tunnel interface is utun5
2025-06-24 14:32:35.337543: [NET] Network change detected with satisfied route and interface order [pdp_ip0]
2025-06-24 14:32:35.337991: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:32:35.338098: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:32:35.338282: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:32:35.338343: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:32:35.338618: [APP] Tunnel 'RemoteAccess-FIBER01' connection status changed to 'connected'
2025-06-24 14:32:35.339078: [NET] Routine: receive incoming v4 - started
2025-06-24 14:32:35.339178: [NET] Routine: receive incoming v6 - started
2025-06-24 14:32:35.339229: [NET] UDP bind has been updated
2025-06-24 14:32:35.347368: [NET] Network change detected with satisfied route and interface order [pdp_ip0, utun5]
2025-06-24 14:32:35.348023: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:32:35.348171: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:32:35.348421: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:32:35.348474: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:32:35.348792: [NET] UDP bind has been updated
2025-06-24 14:32:35.348817: [NET] Routine: receive incoming v4 - started
2025-06-24 14:32:35.348859: [NET] Routine: receive incoming v6 - started
2025-06-24 14:32:40.356472: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 2)
2025-06-24 14:32:40.356825: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:32:45.523502: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 3)
2025-06-24 14:32:45.523828: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:32:50.756726: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 4)
2025-06-24 14:32:50.756938: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:32:55.971949: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 5)
2025-06-24 14:32:55.972261: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:01.095597: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 6)
2025-06-24 14:33:01.095946: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:06.338416: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 7)
2025-06-24 14:33:06.338771: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:11.374720: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 8)
2025-06-24 14:33:11.375030: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:16.459384: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 9)
2025-06-24 14:33:16.459869: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:21.577206: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 10)
2025-06-24 14:33:21.577559: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:26.771926: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 11)
2025-06-24 14:33:26.772266: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:32.083204: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 12)
2025-06-24 14:33:32.083388: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:37.104242: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 13)
2025-06-24 14:33:37.104530: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:42.333708: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 14)
2025-06-24 14:33:42.334024: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:47.587249: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 15)
2025-06-24 14:33:47.587374: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:52.667419: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 16)
2025-06-24 14:33:52.667739: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:57.941159: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 17)
2025-06-24 14:33:57.941498: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:03.070999: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 18)
2025-06-24 14:34:03.071291: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:06.349014: [NET] Network change detected with satisfied route and interface order [pdp_ip0, utun5]
2025-06-24 14:34:06.349536: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:34:06.349692: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:34:06.349846: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:34:06.349972: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:34:06.350135: [NET] UDP bind has been updated
2025-06-24 14:34:06.350179: [NET] Routine: receive incoming v6 - started
2025-06-24 14:34:06.350219: [NET] Routine: receive incoming v4 - started
2025-06-24 14:34:08.104404: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 2)
2025-06-24 14:34:08.104627: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:09.479983: [NET] Network change detected with satisfied route and interface order [pdp_ip0, utun5]
2025-06-24 14:34:09.480848: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:34:09.481026: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:34:09.481356: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:34:09.481408: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:34:09.481746: [NET] UDP bind has been updated
2025-06-24 14:34:09.481758: [NET] Routine: receive incoming v4 - started
2025-06-24 14:34:09.481815: [NET] Routine: receive incoming v6 - started
2025-06-24 14:34:13.304960: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 2)
2025-06-24 14:34:13.305286: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:18.485691: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 3)
2025-06-24 14:34:18.485964: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:23.746238: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 4)
2025-06-24 14:34:23.746510: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:28.898774: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 5)
2025-06-24 14:34:28.899075: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:32.691294: [NET] Network change detected with satisfied route and interface order [pdp_ip0, utun5]
2025-06-24 14:34:32.692257: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:34:32.692500: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:34:32.692770: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:34:32.692825: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:34:32.693150: [NET] UDP bind has been updated
2025-06-24 14:34:32.693159: [NET] Routine: receive incoming v4 - started
2025-06-24 14:34:32.693181: [NET] Routine: receive incoming v6 - started
2025-06-24 14:34:34.061434: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 2)
2025-06-24 14:34:34.061658: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:38.271187: [NET] Network change detected with satisfied route and interface order [pdp_ip0, utun5]
2025-06-24 14:34:38.271585: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:34:38.271675: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:34:38.271920: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:34:38.272043: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:34:38.272310: [NET] UDP bind has been updated
2025-06-24 14:34:38.272334: [NET] Routine: receive incoming v4 - started
2025-06-24 14:34:38.272401: [NET] Routine: receive incoming v6 - started
2025-06-24 14:34:39.095287: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 2)
2025-06-24 14:34:39.095625: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
EDIT 2: Added Screenshot showing Gateways (see linked Post since I cannot Post High Resolution Pictures)
EDIT 3: Not sure what is going on. To check that the basic Principle works, I did a basic Setup on my Hetzner Cloud VPS (behind NAT), and there it works as intended. Sure 1 x WAN Connection only instead of 2 (I tried to replicate here in my Homelab by unplugging the 2nd WAN Cable, so that all Traffic is forced through the remaining WAN Interface).
Any Idea ???
EDIT 4: While trying to search I found this Yesterday, not sure if it's really a Thing though.
https://www.reddit.com/r/opnsense/comments/171k5ap/comment/kj85q3f/
Is it possible that the Wireguard Peer Generator is seriously broken ?
EDIT 5: Actually it seems that I am getting the HomeAssistant Notifications only if I am NOT connected to VPN. Which makes the entire thing even more Confusing. Like need to be Disconnected from VPN to receive the Notifications, yet I need to be connected to VPN to be able to read them. What a Mess !
I should probably be able to dig into a previous Backup of OPNSense Configuration XML File, but I cannot see what changed in Terms of Firewall Rules. It should just work ...
Replying to my own Thread.
Solved mainly thanks to @zapotah over IRC Channel for the Routing Part and some Trial-Error on my End for the DNS Part.
List of Changes:
- Uncheck "Disable reply-to on WAN rules" in the Firewall -> Settings -> Advanced Section
- Set the Gateway to "default" (do **NOT** do Policy based Routing using explicit WAN_XX Gateway Selection) in every Rule under Firewall -> Rules -> [Floating], Firewall -> Rules -> [WAN_XX_...] and Firewall -> Rules -> [WG_REMOTE_XXX_...]
- ADD a Static Route for each of the Wireguard Instances under System -> Routes -> Contiguration with Network = 10.8.X.0/24 (Wireguard Network with Private Address Range) and Gateway = WAN_XX (192.168.200.1 for WAN01 / 192.168.205.1 for WAN02)
- Add the 192.168.0.0/20 Network in the AllowedIps Section of the Peer
- iPhone Wireguard App: make sure that the DNS Servers are specified and they are Comma-Separated, NOT **Space** Separated
- iPhone Wireguard App: make sure that the AllowedIPs includes also the 192.168.0.0/20 Target Network
- For Testing using something like (I have it always on) a docker.io/georgyo/ifconfig.io or docker.io/traefik/whoami Container can be useful to check your IP Address (I had to spin a separate Instance up and use Port 8080 to have Direct Access and NOT go through the Caddy Proxy, since the DNS was NOT working until I fixed the DNS Servers that were mistakenly Space-separated)