[SOLVED] Wireguard routing problem, I need idea!

[FrAllard]

Here is my problem at the moment.

I have a wireguard tunnel between two places, the tunnel works to an extend... Pings can flow through in both directions, tcp though can only flow in one direction and I really don't understand what is happening.

My OPNsense doesn't have a WAN interface, it's a wireguard gateway with a LAN interface and a wireguard interface. OPNsense has a gateway and that gateway is also the gateway of the hosts behind the problematic side of the tunnel.

So I have
10.65.87.1 (default gateway)
10.65.87.2 (OPNsense LAN)
10.65.87.21 (host that is trying to establish a tcp connection to the other side of the tunnel)
Routes of that host

Code: [Select]

$ ip r
default via 10.65.87.1 dev ens160 proto static
10.65.87.0/24 dev ens160 proto kernel scope link src 10.65.87.21
10.65.89.0/24 via 10.65.87.2 dev ens160 proto static
10.65.90.0/24 via 10.65.87.2 dev ens160 proto static
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br_pspdfkit_net proto kernel scope link src 172.18.0.1
Firewall rules on both side of the tunnel are wide open for now
Allow IPv4 * Source * Destination * everything *

I'm doing the following command on the 10.65.87.21 host : curl https:// 10.65.89.10:8006
(Inserted a space here to avoid the forum from creating a hyperlink)

Here is the capture done on the LAN interface on OPNsense, so packets do traverse the tunnel and get into the lan interface.

Code: [Select]

root@OPNsense:~ # tcpdump -n -i vmx0 host 10.65.89.10
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vmx0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:03:54.137159 IP 10.65.87.21.47858 > 10.65.89.10.8006: Flags [S], seq 3997485865, win 64240, options [mss 1460,sackOK,TS val 783913448 ecr 0,nop,wscale 7], length 0
15:03:54.144690 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340330385 ecr 783913448,nop,wscale 7], length 0
15:03:55.145738 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340331386 ecr 783913448,nop,wscale 7], length 0
15:03:55.146560 IP 10.65.87.21.47858 > 10.65.89.10.8006: Flags [S], seq 3997485865, win 64240, options [mss 1460,sackOK,TS val 783914458 ecr 0,nop,wscale 7], length 0
15:03:55.154193 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340331394 ecr 783913448,nop,wscale 7], length 0
15:03:57.161727 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340333402 ecr 783913448,nop,wscale 7], length 0
15:03:57.162497 IP 10.65.87.21.47858 > 10.65.89.10.8006: Flags [S], seq 3997485865, win 64240, options [mss 1460,sackOK,TS val 783916474 ecr 0,nop,wscale 7], length 0
15:03:57.169315 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340333410 ecr 783913448,nop,wscale 7], length 0
15:04:01.193744 IP 10.65.89.10.8006 > 10.65.87.21.47858: Flags [S.], seq 3568795638, ack 3997485866, win 65160, options [mss 1460,sackOK,TS val 1340337434 ecr 783913448,nop,wscale 7], length 0
Here is the tcpdump on  10.65.87.21 host trying to reach the other side, it never get the packets back even though opnsense does put it on the lan.

Code: [Select]

$ sudo tcpdump -n -i ens160 host 10.65.89.10
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
10:34:44.540703 IP 10.65.87.21.32778 > 10.65.89.10.8006: Flags [S], seq 4094067348, win 64240, options [mss 1460,sackOK,TS val 782163854 ecr 0,nop,wscale 7], length 0
10:34:45.544710 IP 10.65.87.21.32778 > 10.65.89.10.8006: Flags [S], seq 4094067348, win 64240, options [mss 1460,sackOK,TS val 782164858 ecr 0,nop,wscale 7], length 0
10:36:47.315603 IP 10.65.87.21.39506 > 10.65.89.10.8006: Flags [S], seq 2886622905, win 64240, options [mss 1460,sackOK,TS val 782286628 ecr 0,nop,wscale 7], length 0
10:36:48.328706 IP 10.65.87.21.39506 > 10.65.89.10.8006: Flags [S], seq 2886622905, win 64240, options [mss 1460,sackOK,TS val 782287642 ecr 0,nop,wscale 7], length 0
10:37:35.421371 IP 10.65.87.21.35480 > 10.65.89.10.8006: Flags [S], seq 1520415211, win 64240, options [mss 1460,sackOK,TS val 782334734 ecr 0,nop,wscale 7], length 0
10:37:36.424713 IP 10.65.87.21.35480 > 10.65.89.10.8006: Flags [S], seq 1520415211, win 64240, options [mss 1460,sackOK,TS val 782335738 ecr 0,nop,wscale 7], length 0
10:38:16.677116 IP 10.65.87.21.35994 > 10.65.89.10.8006: Flags [S], seq 4124073389, win 64240, options [mss 1460,sackOK,TS val 782375990 ecr 0,nop,wscale 7], length 0
10:38:17.704686 IP 10.65.87.21.35994 > 10.65.89.10.8006: Flags [S], seq 4124073389, win 64240, options [mss 1460,sackOK,TS val 782377018 ecr 0,nop,wscale 7], length 0

[FrAllard]

Doing traceroute from 10.65.87.21
traceroute 10.65.89.10 (udp) doesn't work... same behavior as above
traceroute -I 10.65.89.10 (icmp) works

[FrAllard]

Nevermind I solved my problem... It was the vmware NSX-V router screwwing thing up ...

I changed the default admin distance of the gateway to 19 then made a static route for the other side subnet in the nsx-v with a admin distance of 1 and deleted the static route in my vm and it just worked.

I knew OPNsense was doing its job!