Messages

I can confirm this issue. Been troubleshooting this behaviour in my setup aswell, unfortunately with no luck. The traceroute workaround indeed fixes the issue for some time until "something" breaks it again. With a tcpdump on the wireguard interface I was able to find this:

Code Select Expand

root@vm-opnsense-1:~ # tcpdump -ni wg0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg0, link-type NULL (BSD loopback), snapshot length 262144 bytes
23:56:57.659028 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 256, seq 576, length 9
23:56:57.675027 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 256, seq 576, length 9          <= Wrong source and destination IP on the response packet

Dpinger sends the icmp packet out correctly but the response packet somehow gets its source and destination IPs transformed and is therefore not correctly matched to the initial process. This transformation is happening somehow on the opnsense itself. In my case this is my WAN facing IP Address (.247) and the regular WAN GW address (.1)

With the traceroute workaround you can see in real time how the behaviour switches to a functional state again.

Code Select Expand

root@vm-opnsense-1:~ # tcpdump -ni wg0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg0, link-type NULL (BSD loopback), snapshot length 262144 bytes
01:55:23.671267 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 24, length 9
01:55:23.687559 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 24, length 9
01:55:24.721268 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 25, length 9
01:55:24.737848 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 25, length 9
01:55:25.771371 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 26, length 9
01:55:25.787752 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 26, length 9
01:55:26.821309 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 27, length 9
01:55:26.837408 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 27, length 9
01:55:27.871295 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 28, length 9
01:55:27.887429 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 28, length 9
01:55:28.615346 IP 10.2.0.2.61637 > 10.2.0.1.33435: UDP, length 12                                          <= traceroute
01:55:28.921307 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 29, length 9
01:55:28.937276 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 29, length 9
01:55:29.971368 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 30, length 9
01:55:29.989760 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 30, length 9
01:55:31.021387 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 31, length 9
01:55:31.037842 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 31, length 9
01:55:32.071392 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 32, length 9
01:55:32.086992 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 32, length 9
01:55:33.121416 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 33, length 9
01:55:33.137765 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 33, length 9
01:55:33.671328 IP 10.2.0.2.61637 > 10.2.0.1.33436: UDP, length 12                                          <= traceroute
01:55:34.171394 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 34, length 9
01:55:34.187608 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 34, length 9
01:55:35.221393 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 35, length 9
01:55:35.237311 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 35, length 9
01:55:36.271310 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 36, length 9
01:55:36.287600 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 36, length 9
01:55:37.312655 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 37, length 9
01:55:37.328764 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 37, length 9
01:55:38.371303 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 38, length 9
01:55:38.387665 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 38, length 9
01:55:38.716191 IP 10.2.0.2.61637 > 10.2.0.1.33437: UDP, length 12                                          <= traceroute
01:55:39.384896 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 39, length 9
01:55:39.401338 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 39, length 9
01:55:40.418807 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 40, length 9
01:55:40.434959 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 40, length 9
01:55:41.471452 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 41, length 9
01:55:41.487620 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 41, length 9
01:55:42.521338 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 42, length 9
01:55:42.537770 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 42, length 9
01:55:43.571406 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 43, length 9
01:55:43.587306 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 43, length 9
01:55:43.771858 IP 10.2.0.2.61637 > 10.2.0.1.33438: UDP, length 12                                          <= traceroute
01:55:44.621431 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 44, length 9
01:55:44.637576 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 44, length 9
01:55:45.671337 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 45, length 9
01:55:45.687429 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 45, length 9
01:55:46.721357 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 46, length 9
01:55:46.737577 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 46, length 9
01:55:47.771463 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 47, length 9
01:55:47.787821 IP 192.168.178.1 > 192.168.178.247: ICMP echo reply, id 677, seq 47, length 9
01:55:48.821353 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 48, length 9
01:55:48.821401 IP 10.2.0.2.61637 > 10.2.0.1.33439: UDP, length 12                                          <= traceroute
01:55:48.837608 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 48, length 9                           <= issue self-healed
01:55:49.871494 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 49, length 9
01:55:49.887955 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 49, length 9
01:55:50.921394 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 50, length 9
01:55:50.937420 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 50, length 9
01:55:51.971370 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 51, length 9
01:55:51.989668 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 51, length 9
01:55:53.021438 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 52, length 9
01:55:53.037889 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 52, length 9
01:55:53.871326 IP 10.2.0.2.61637 > 10.2.0.1.33440: UDP, length 12
01:55:54.071318 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 53, length 9
01:55:54.088002 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 53, length 9
01:55:55.121448 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 54, length 9
01:55:55.137603 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 54, length 9
01:55:56.171439 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 55, length 9
01:55:56.187354 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 55, length 9
01:55:57.221381 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 56, length 9
01:55:57.237847 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 56, length 9
01:55:58.271386 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 57, length 9
01:55:58.287646 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 57, length 9
01:55:59.321385 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 58, length 9
01:55:59.337847 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 58, length 9
01:56:00.332408 IP 10.2.0.2 > 10.2.0.1: ICMP echo request, id 677, seq 59, length 9
01:56:00.348297 IP 10.2.0.1 > 10.2.0.2: ICMP echo reply, id 677, seq 59, length 9

I cannot find any reason in my setup why this rewrite would happen. Doesn't seem logical to me either.