WireGuard gateway intermittently goes offline and stays offline

[tcm1010]

I have been working on this for a couple weeks now, and it is time to ask for help.

I have a (mostly) working setup configured with VPN via the excellent WireGuard documentation (instance, peer, interface, gateway, firewall rules, NAT), with a couple of additional tweaks to make it multi-WAN for redundancy: 

• gateway switching is enabled (for redundancy)
• gateway failover and failback states are enabled (so the clients don't loose access to the Internet)
• static routes for the WireGuard endpoints via the WAN interface (so one tunnel isn't routed via another tunnel)

When I first got it working, it was magic: a tunnel gateway would experience enough loss to trigger the gateway switch, the second gateway would become the default, and clients wouldn't notice anything had gone wrong.  Yea!!

But...I checked things every so often, and I would notice one of the tunnel gateways would be permanently down, i.e., showing 100% loss, for hours at a time, e.g., overnight.  Yet, the WireGuard instance and peer for that gateway would remain green/online.  This doesn't happen every time (of course, right?!).  I can see in the logs that the gateways will switch as loss happens on the higher priority gateway, and will switch back once the loss is low enough on the higher priority one, as expected.  But every so often, one of them gets stuck in the red/offline state.

To make a long thread short, I have discovered that when this happens, I can manually and easily fix the problem by executing in the OPNsense CLI the traceroute command through the offline gateway, and then instantly, I can ping the gateway, the gateway monitor IP no longer experiences loss, and the gateway will go green/online (and if it had the higher priority, it would switch and become the default).

I have tried this with several different WireGuard instances/peers at different locations (provider is ProtonVPN), and each one has experienced this issue.

Code Select Expand

root@OPNsense:~ # netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            10.2.0.1           UGS             wg0
10.2.0.1           link#12            UHS             wg0     # This is the currently active/default gateway
10.2.0.2           link#3             UH              lo0     # This is the currently active/default tunnel IP
[...]
10.2.3.1           link#15            UHS             wg3     # This is the problematic gateway
10.2.3.2           link#3             UH              lo0     # This is the problematic tunnel
[...]

root@OPNsense:~ # ping -S 10.2.3.2 -c 10 1.1.1.1              # ping something via the problematic tunnel - fail
PING 1.1.1.1 (1.1.1.1) from 10.2.3.2: 56 data bytes

--- 1.1.1.1 ping statistics ---
10 packets transmitted, 0 packets received, 100.0% packet loss

root@OPNsense:~ # time traceroute -s 10.2.3.2 1.1.1.1                                  # traceroute something via the problematic tunnel - works
traceroute to 1.1.1.1 (1.1.1.1) from 10.2.3.2, 64 hops max, 40 byte packets
 1  10.2.3.1 (10.2.3.1)  15.823 ms  14.536 ms  14.187 ms
 2  146.70.202.81 (146.70.202.81)  31.409 ms  29.895 ms  30.969 ms
 3  ae32-1932.agg4v.nyc1.us.m247.ro (146.70.1.249)  21.035 ms  20.754 ms  19.185 ms
[...]
^C
0.000u 0.006s 0:03.14 0.0% 0+0k 0+0io 0pf+0w                    # Only 3 seconds using traceroute

root@OPNsense:~ # ping -S 10.2.3.2  1.1.1.1                          # ping then works and the gateway then shows in the GUI as green/online
PING 1.1.1.1 (1.1.1.1) from 10.2.3.2: 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=53 time=18.939 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=53 time=26.877 ms
^C
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 18.939/22.908/26.877/3.969 ms

Any thoughts on what is happening to get into the stuck situation, and why "kicking" the offline gateway with traceroute seems to restore functionality?

OPNsense 26.1.2_5

[gogormonchi]

I can confirm this issue. Been troubleshooting this behaviour in my setup aswell, unfortunately with no luck. The traceroute workaround indeed fixes the issue for some time until "something" breaks it again. With a tcpdump on the wireguard interface I was able to find this:

Code Select Expand

root@vm-opnsense-1:~ # tcpdump -ni wg0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg0, link-type NULL (BSD loopback), snapshot length 262144 bytes
23:56:57.659028 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 256, seq 576, length 9
23:56:57.675027 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 256, seq 576, length 9          <= Wrong source and destination IP on the response packet

Dpinger sends the icmp packet out correctly but the response packet somehow gets its source and destination IPs transformed and is therefore not correctly matched to the initial process. This transformation is happening somehow on the opnsense itself. In my case this is my WAN facing IP Address (.247) and the regular WAN GW address (.1)

With the traceroute workaround you can see in real time how the behaviour switches to a functional state again.

Code Select Expand

root@vm-opnsense-1:~ # tcpdump -ni wg0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg0, link-type NULL (BSD loopback), snapshot length 262144 bytes
01:55:23.671267 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 24, length 9
01:55:23.687559 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 24, length 9
01:55:24.721268 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 25, length 9
01:55:24.737848 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 25, length 9
01:55:25.771371 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 26, length 9
01:55:25.787752 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 26, length 9
01:55:26.821309 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 27, length 9
01:55:26.837408 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 27, length 9
01:55:27.871295 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 28, length 9
01:55:27.887429 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 28, length 9
01:55:28.615346 IP 10.2.0.2.61637 > 10.2.0.1.33435: UDP, length 12                                          <= traceroute
01:55:28.921307 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 29, length 9
01:55:28.937276 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 29, length 9
01:55:29.971368 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 30, length 9
01:55:29.989760 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 30, length 9
01:55:31.021387 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 31, length 9
01:55:31.037842 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 31, length 9
01:55:32.071392 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 32, length 9
01:55:32.086992 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 32, length 9
01:55:33.121416 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 33, length 9
01:55:33.137765 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 33, length 9
01:55:33.671328 IP 10.2.0.2.61637 > 10.2.0.1.33436: UDP, length 12                                          <= traceroute
01:55:34.171394 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 34, length 9
01:55:34.187608 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 34, length 9
01:55:35.221393 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 35, length 9
01:55:35.237311 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 35, length 9
01:55:36.271310 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 36, length 9
01:55:36.287600 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 36, length 9
01:55:37.312655 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 37, length 9
01:55:37.328764 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 37, length 9
01:55:38.371303 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 38, length 9
01:55:38.387665 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 38, length 9
01:55:38.716191 IP 10.2.0.2.61637 > 10.2.0.1.33437: UDP, length 12                                          <= traceroute
01:55:39.384896 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 39, length 9
01:55:39.401338 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 39, length 9
01:55:40.418807 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 40, length 9
01:55:40.434959 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 40, length 9
01:55:41.471452 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 41, length 9
01:55:41.487620 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 41, length 9
01:55:42.521338 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 42, length 9
01:55:42.537770 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 42, length 9
01:55:43.571406 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 43, length 9
01:55:43.587306 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 43, length 9
01:55:43.771858 IP 10.2.0.2.61637 > 10.2.0.1.33438: UDP, length 12                                          <= traceroute
01:55:44.621431 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 44, length 9
01:55:44.637576 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 44, length 9
01:55:45.671337 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 45, length 9
01:55:45.687429 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 45, length 9
01:55:46.721357 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 46, length 9
01:55:46.737577 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 46, length 9
01:55:47.771463 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 47, length 9
01:55:47.787821 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 47, length 9
01:55:48.821353 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 48, length 9
01:55:48.821401 IP 10.2.0.2.61637 > 10.2.0.1.33439: UDP, length 12                                          <= traceroute
01:55:48.837608 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 48, length 9                           <= issue self-healed
01:55:49.871494 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 49, length 9
01:55:49.887955 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 49, length 9
01:55:50.921394 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 50, length 9
01:55:50.937420 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 50, length 9
01:55:51.971370 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 51, length 9
01:55:51.989668 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 51, length 9
01:55:53.021438 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 52, length 9
01:55:53.037889 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 52, length 9
01:55:53.871326 IP 10.2.0.2.61637 > 10.2.0.1.33440: UDP, length 12
01:55:54.071318 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 53, length 9
01:55:54.088002 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 53, length 9
01:55:55.121448 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 54, length 9
01:55:55.137603 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 54, length 9
01:55:56.171439 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 55, length 9
01:55:56.187354 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 55, length 9
01:55:57.221381 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 56, length 9
01:55:57.237847 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 56, length 9
01:55:58.271386 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 57, length 9
01:55:58.287646 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 57, length 9
01:55:59.321385 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 58, length 9
01:55:59.337847 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 58, length 9
01:56:00.332408 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 59, length 9
01:56:00.348297 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 59, length 9

I cannot find any reason in my setup why this rewrite would happen. Doesn't seem logical to me either.