[SOLVED] Wireguard routing problem, I need idea!

[FrAllard]

Here is my problem at the moment.

I have a wireguard tunnel between two places, the tunnel works to an extend... Pings can flow through in both directions, tcp though can only flow in one direction and I really don't understand what is happening.

My OPNsense doesn't have a WAN interface, it's a wireguard gateway with a LAN interface and a wireguard interface. OPNsense has a gateway and that gateway is also the gateway of the hosts behind the problematic side of the tunnel.

So I have
10.65.87.1 (default gateway)
10.65.87.2 (OPNsense LAN)
10.65.87.21 (host that is trying to establish a tcp connection to the other side of the tunnel)
Routes of that host

Code Select Expand

$ ip r
default via 10.65.87.1 dev ens160 proto static
10.65.87.0/24 dev ens160 proto kernel scope link src 10.65.87.21
10.65.89.0/24 via 10.65.87.2 dev ens160 proto static
10.65.90.0/24 via 10.65.87.2 dev ens160 proto static
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br_pspdfkit_net proto kernel scope link src 172.18.0.1

Firewall rules on both side of the tunnel are wide open for now
Allow IPv4 * Source * Destination * everything *

I'm doing the following command on the 10.65.87.21 host : curl https:// 10.65.89.10:8006
(Inserted a space here to avoid the forum from creating a hyperlink)

Here is the capture done on the LAN interface on OPNsense, so packets do traverse the tunnel and get into the lan interface.

Code Select Expand

root@OPNsense:~ # tcpdump -n -i vmx0 host 10.65.89.10
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vmx0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:03:54.137159 IP 10.65.87.21.47858 > 10.65.89.10.8006: Flags [S], seq 3997485865, win 64240, options [mss 1460,sackOK,TS val 783913448 ecr 0,nop,wscale 7], length 0
15:03:54.144690 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340330385 ecr 783913448,nop,wscale 7], length 0
15:03:55.145738 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340331386 ecr 783913448,nop,wscale 7], length 0
15:03:55.146560 IP 10.65.87.21.47858 > 10.65.89.10.8006: Flags [S], seq 3997485865, win 64240, options [mss 1460,sackOK,TS val 783914458 ecr 0,nop,wscale 7], length 0
15:03:55.154193 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340331394 ecr 783913448,nop,wscale 7], length 0
15:03:57.161727 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340333402 ecr 783913448,nop,wscale 7], length 0
15:03:57.162497 IP 10.65.87.21.47858 > 10.65.89.10.8006: Flags [S], seq 3997485865, win 64240, options [mss 1460,sackOK,TS val 783916474 ecr 0,nop,wscale 7], length 0
15:03:57.169315 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340333410 ecr 783913448,nop,wscale 7], length 0
15:04:01.193744 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340337434 ecr 783913448,nop,wscale 7], length 0

Here is the tcpdump on  10.65.87.21 host trying to reach the other side, it never get the packets back even though opnsense does put it on the lan.

Code Select Expand

$ sudo tcpdump -n -i ens160 host 10.65.89.10
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
10:34:44.540703 IP 10.65.87.21.32778 > 10.65.89.10.8006: Flags [S], seq 4094067348, win 64240, options [mss 1460,sackOK,TS val 782163854 ecr 0,nop,wscale 7], length 0
10:34:45.544710 IP 10.65.87.21.32778 > 10.65.89.10.8006: Flags [S], seq 4094067348, win 64240, options [mss 1460,sackOK,TS val 782164858 ecr 0,nop,wscale 7], length 0
10:36:47.315603 IP 10.65.87.21.39506 > 10.65.89.10.8006: Flags [S], seq 2886622905, win 64240, options [mss 1460,sackOK,TS val 782286628 ecr 0,nop,wscale 7], length 0
10:36:48.328706 IP 10.65.87.21.39506 > 10.65.89.10.8006: Flags [S], seq 2886622905, win 64240, options [mss 1460,sackOK,TS val 782287642 ecr 0,nop,wscale 7], length 0
10:37:35.421371 IP 10.65.87.21.35480 > 10.65.89.10.8006: Flags [S], seq 1520415211, win 64240, options [mss 1460,sackOK,TS val 782334734 ecr 0,nop,wscale 7], length 0
10:37:36.424713 IP 10.65.87.21.35480 > 10.65.89.10.8006: Flags [S], seq 1520415211, win 64240, options [mss 1460,sackOK,TS val 782335738 ecr 0,nop,wscale 7], length 0
10:38:16.677116 IP 10.65.87.21.35994 > 10.65.89.10.8006: Flags [S], seq 4124073389, win 64240, options [mss 1460,sackOK,TS val 782375990 ecr 0,nop,wscale 7], length 0
10:38:17.704686 IP 10.65.87.21.35994 > 10.65.89.10.8006: Flags [S], seq 4124073389, win 64240, options [mss 1460,sackOK,TS val 782377018 ecr 0,nop,wscale 7], length 0

[FrAllard]

Doing traceroute from 10.65.87.21
traceroute 10.65.89.10 (udp) doesn't work... same behavior as above
traceroute -I 10.65.89.10 (icmp) works

[FrAllard]

Nevermind I solved my problem... It was the vmware NSX-V router screwwing thing up ...

I changed the default admin distance of the gateway to 19 then made a static route for the other side subnet in the nsx-v with a admin distance of 1 and deleted the static route in my vm and it just worked.

I knew OPNsense was doing its job!