OPNsense Forum
Archive => 20.7 Legacy Series => Topic started by: TheLinuxGuy on January 20, 2021, 07:06:23 pm
-
I'm trying to setup Port Forwarding via VPN interface, the packets are being NAT'd correctly but when the response packets are being sent the VPN gateway (wg0) is being ignored and WAN (em0) is used.
Goal:
Network topology looks like:
Internet >> VPS (public IPv4) << wireguard tunnel >> opnsense FW (wg0) << client on DMZ interface
WAN-vps (assume its 1.2.3.4)
wg0 (vps-server IP 10.0.0.1)
wg0 (opnsense wg0 IP 10.0.0.10)
client (dmz IP 172.20.0.20)
My goal is to forward port 32400 for Plex media server.
IP 1.2.3.4:32500 is expected to connect to VPS via public IP, be forwarded via wg0 and arrive at opnsense.
looking to keep original network source TCP/IP addresses to track who is connecting and from where in my logs.
opnsense should NAT incoming wg0 traffic to port 32400 and translate to 172.20.0.20 (DMZ interface host).
opnsense is translating traffic and when client is responding they it is for public IP addresses - I expected by policy based routing rule on DMZ interface to apply here. I have explicitly set the gateway for non-local networks to be using wg0 interface but tcpdump of WAN on opnsense show otherwise. Packets are being sent out of the non-wireguard tunnel.
Any help here would be amazing. Been stuck here.
My test scenario, using: https://www.yougetsignal.com/tools/open-ports/ to check port 32400
On WAN-VPS (wireguard gateway with forwarding enabled - confirmed packet is coming in to Public IPv4):
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
17:54:11.413547 IP 76.175.136.3.50396 > 1.2.3.4.32400: Flags [S], seq 1644582163, win 65535, options [mss 1460,nop,wscale 7,sackOK,TS val 3902988 ecr 0], length 0
17:54:12.944644 IP 198.199.98.246.40123 > 1.2.3.4.32400: Flags [S], seq 3344110111, win 14600, options [mss 1460,sackOK,TS val 729869398 ecr 0,nop,wscale 8], length 0
17:54:13.942881 IP 198.199.98.246.40123 > 1.2.3.4.32400: Flags [S], seq 3344110111, win 14600, options [mss 1460,sackOK,TS val 729869648 ecr 0,nop,wscale 8], length 0
17:54:13.957298 IP 198.199.98.246.40125 > 1.2.3.4.32400: Flags [S], seq 1528187397, win 14600, options [mss 1460,sackOK,TS val 729869651 ecr 0,nop,wscale 8], length 0
17:54:14.954744 IP 198.199.98.246.40125 > 1.2.3.4.32400: Flags [S], seq 1528187397, win 14600, options [mss 1460,sackOK,TS val 729869901 ecr 0,nop,wscale 8], length 0
17:54:14.989967 IP 198.199.98.246.40128 > 1.2.3.4.32400: Flags [S], seq 1176045227, win 14600, options [mss 1460,sackOK,TS val 729869909 ecr 0,nop,wscale 8], length 0
17:54:15.986776 IP 198.199.98.246.40128 > 1.2.3.4.32400: Flags [S], seq 1176045227, win 14600, options [mss 1460,sackOK,TS val 729870159 ecr 0,nop,wscale 8], length 0
From VPS-wg0 interface it shows packet being forwarded with original IPv4 headers as wanted with a destination of 10.0.0.10 the opnsense wg0 interface:
17:54:11.413562 IP 76.175.136.3.50396 > 10.0.0.10.32400: Flags [S], seq 1644582163, win 65535, options [mss 1460,nop,wscale 7,sackOK,TS val 3902988 ecr 0], length 0
17:54:12.944663 IP 198.199.98.246.40123 > 10.0.0.10.32400: Flags [S], seq 3344110111, win 14600, options [mss 1460,sackOK,TS val 729869398 ecr 0,nop,wscale 8], length 0
17:54:13.942893 IP 198.199.98.246.40123 > 10.0.0.10.32400: Flags [S], seq 3344110111, win 14600, options [mss 1460,sackOK,TS val 729869648 ecr 0,nop,wscale 8], length 0
17:54:13.957321 IP 198.199.98.246.40125 > 10.0.0.10.32400: Flags [S], seq 1528187397, win 14600, options [mss 1460,sackOK,TS val 729869651 ecr 0,nop,wscale 8], length 0
17:54:14.954763 IP 198.199.98.246.40125 > 10.0.0.10.32400: Flags [S], seq 1528187397, win 14600, options [mss 1460,sackOK,TS val 729869901 ecr 0,nop,wscale 8], length 0
17:54:14.989988 IP 198.199.98.246.40128 > 10.0.0.10.32400: Flags [S], seq 1176045227, win 14600, options [mss 1460,sackOK,TS val 729869909 ecr 0,nop,wscale 8], length 0
17:54:15.986789 IP 198.199.98.246.40128 > 10.0.0.10.32400: Flags [S], seq 1176045227, win 14600, options [mss 1460,sackOK,TS val 729870159 ecr 0,nop,wscale 8], length 0
Now that we know the IP address of the host coming thru the VPS we see the final host (destination NAT 172.20.0.20 responding to the public IP address):
# tcpdump -i vtnet0_vlan700 host 198.199.98.246 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet0_vlan700, link-type EN10MB (Ethernet), capture size 262144 bytes
12:58:26.032396 IP 198.199.98.246.41128 > 172.20.0.20.32400: Flags [S], seq 1186752319, win 14600, options [mss 1460,sackOK,TS val 729932656 ecr 0,nop,wscale 8], length 0
12:58:26.032596 IP 172.20.0.20.32400 > 198.199.98.246.41128: Flags [S.], seq 3453310492, ack 1186752320, win 65160, options [mss 1460,sackOK,TS val 3269064663 ecr 729932656,nop,wscale 7], length 0
12:58:27.030216 IP 198.199.98.246.41128 > 172.20.0.20.32400: Flags [S], seq 1186752319, win 14600, options [mss 1460,sackOK,TS val 729932906 ecr 0,nop,wscale 8], length 0
12:58:27.030379 IP 172.20.0.20.32400 > 198.199.98.246.41128: Flags [S.], seq 3453310492, ack 1186752320, win 65160, options [mss 1460,sackOK,TS val 3269065661 ecr 729932656,nop,wscale 7], length 0
12:58:27.078113 IP 198.199.98.246.41131 > 172.20.0.20.32400: Flags [S], seq 1180124324, win 14600, options [mss 1460,sackOK,TS val 729932918 ecr 0,nop,wscale 8], length 0
12:58:27.078249 IP 172.20.0.20.32400 > 198.199.98.246.41131: Flags [S.], seq 1408018903, ack 1180124325, win 65160, options [mss 1460,sackOK,TS val 3269065708 ecr 729932918,nop,wscale 7], length 0
12:58:28.044387 IP 172.20.0.20.32400 > 198.199.98.246.41128: Flags [S.], seq 3453310492, ack 1186752320, win 65160, options [mss 1460,sackOK,TS val 3269066675 ecr 729932656,nop,wscale 7], length 0
12:58:28.074094 IP 198.199.98.246.41131 > 172.20.0.20.32400: Flags [S], seq 1180124324, win 14600, options [mss 1460,sackOK,TS val 729933168 ecr 0,nop,wscale 8], length 0
12:58:28.074298 IP 172.20.0.20.32400 > 198.199.98.246.41131: Flags [S.], seq 1408018903, ack 1180124325, win 65160, options [mss 1460,sackOK,TS val 3269066704 ecr 729932918,nop,wscale 7], length 0
12:58:28.124204 IP 198.199.98.246.41134 > 172.20.0.20.32400: Flags [S], seq 260678019, win 14600, options [mss 1460,sackOK,TS val 729933180 ecr 0,nop,wscale 8], length 0
12:58:28.124316 IP 172.20.0.20.32400 > 198.199.98.246.41134: Flags [S.], seq 2906162909, ack 260678020, win 65160, options [mss 1460,sackOK,TS val 3269066755 ecr 729933180,nop,wscale 7], length 0
12:58:29.100367 IP 172.20.0.20.32400 > 198.199.98.246.41131: Flags [S.], seq 1408018903, ack 1180124325, win 65160, options [mss 1460,sackOK,TS val 3269067731 ecr 729932918,nop,wscale 7], length 0
12:58:29.120326 IP 198.199.98.246.41134 > 172.20.0.20.32400: Flags [S], seq 260678019, win 14600, options [mss 1460,sackOK,TS val 729933430 ecr 0,nop,wscale 8], length 0
12:58:29.120921 IP 172.20.0.20.32400 > 198.199.98.246.41134: Flags [S.], seq 2906162909, ack 260678020, win 65160, options [mss 1460,sackOK,TS val 3269067751 ecr 729933180,nop,wscale 7], length 0
12:58:30.064379 IP 172.20.0.20.32400 > 198.199.98.246.41128: Flags [S.], seq 3453310492, ack 1186752320, win 65160, options [mss 1460,sackOK,TS val 3269068695 ecr 729932656,nop,wscale 7], length 0
12:58:30.124414 IP 172.20.0.20.32400 > 198.199.98.246.41134: Flags [S.], seq 2906162909, ack 260678020, win 65160, options [mss 1460,sackOK,TS val 3269068755 ecr 729933180,nop,wscale 7], length 0
12:58:31.116373 IP 172.20.0.20.32400 > 198.199.98.246.41131: Flags [S.], seq 1408018903, ack 1180124325, win 65160, options [mss 1460,sackOK,TS val 3269069747 ecr 729932918,nop,wscale 7], length 0
12:58:32.140411 IP 172.20.0.20.32400 > 198.199.98.246.41134: Flags [S.], seq 2906162909, ack 260678020, win 65160, options [mss 1460,sackOK,TS val 3269070771 ecr 729933180,nop,wscale 7], length 0
Problem is that wg0 is where opnsense should be sending those responses and it should outbound NAT 172.20.0.20 as 10.0.0.10. Routing this via wg0 seems to be not happening and instead going thru opnsense-WAN.
https://imgur.com/a/vZQyGvE << configs
Here is confirmation the responses are going thru default route / opnsense WAN and screenshots of the policy based rules that should have forced the traffic thru wg0 wireguard interface.
# tcpdump -i em0 host 198.199.98.246 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:02:11.952841 IP 10.0.0.10.32400 > 198.199.98.246.42000: Flags [S.], seq 1206282859, ack 3275283905, win 65160, options [mss 1460,sackOK,TS val 3269290582 ecr 729989138,nop,wscale 7], length 0
13:02:12.958465 IP 10.0.0.10.32400 > 198.199.98.246.42000: Flags [S.], seq 1206282859, ack 3275283905, win 65160, options [mss 1460,sackOK,TS val 3269291588 ecr 729989138,nop,wscale 7], length 0
13:02:12.999633 IP 10.0.0.10.32400 > 198.199.98.246.42004: Flags [S.], seq 965419102, ack 1952030132, win 65160, options [mss 1460,sackOK,TS val 3269291629 ecr 729989400,nop,wscale 7], length 0
13:02:13.964997 IP 10.0.0.10.32400 > 198.199.98.246.42000: Flags [S.], seq 1206282859, ack 3275283905, win 65160, options [mss 1460,sackOK,TS val 3269292595 ecr 729989138,nop,wscale 7], length 0
13:02:13.999727 IP 10.0.0.10.32400 > 198.199.98.246.42004: Flags [S.], seq 965419102, ack 1952030132, win 65160, options [mss 1460,sackOK,TS val 3269292629 ecr 729989400,nop,wscale 7], length 0
13:02:14.041662 IP 10.0.0.10.32400 > 198.199.98.246.42010: Flags [S.], seq 2726232845, ack 434760706, win 65160, options [mss 1460,sackOK,TS val 3269292671 ecr 729989661,nop,wscale 7], length 0
13:02:15.021120 IP 10.0.0.10.32400 > 198.199.98.246.42004: Flags [S.], seq 965419102, ack 1952030132, win 65160, options [mss 1460,sackOK,TS val 3269293651 ecr 729989400,nop,wscale 7], length 0
13:02:15.045706 IP 10.0.0.10.32400 > 198.199.98.246.42010: Flags [S.], seq 2726232845, ack 434760706, win 65160, options [mss 1460,sackOK,TS val 3269293675 ecr 729989661,nop,wscale 7], length 0
13:02:15.981011 IP 10.0.0.10.32400 > 198.199.98.246.42000: Flags [S.], seq 1206282859, ack 3275283905, win 65160, options [mss 1460,sackOK,TS val 3269294611 ecr 729989138,nop,wscale 7], length 0
13:02:16.076957 IP 10.0.0.10.32400 > 198.199.98.246.42010: Flags [S.], seq 2726232845, ack 434760706, win 65160, options [mss 1460,sackOK,TS val 3269294707 ecr 729989661,nop,wscale 7], length 0
13:02:17.036976 IP 10.0.0.10.32400 > 198.199.98.246.42004: Flags [S.], seq 965419102, ack 1952030132, win 65160, options [mss 1460,sackOK,TS val 3269295667 ecr 729989400,nop,wscale 7], length 0
13:02:18.093017 IP 10.0.0.10.32400 > 198.199.98.246.42010: Flags [S.], seq 2726232845, ack 434760706, win 65160, options [mss 1460,sackOK,TS val 3269296723 ecr 729989661,nop,wscale 7], length 0
-
If I add the following via terminal, the connection / port forward seems to work.
# route add -host 198.199.98.246 10.0.0.1
But this may be undesirable - 198.199.98.246 is a public address. is there a better way to tell pfsense that after processing NAT use a specific route table (say default gateway #2 that pertains only to this?)...
or a workaround?
These are the route-to rules
[root@fw /usr/local/etc/wireguard]# pfctl -sr | grep -e reply-to -e route-to
pass out log route-to (ovpnc2 10.8.8.1) inet from 10.8.8.3 to ! (ovpnc2:network) flags S/SA keep state allow-opts label "1b3cc5f53970f9ed462aeaae172e6a40"
pass out log route-to (wg0 10.0.0.1) inet from 10.0.0.10 to ! (wg0:network) flags S/SA keep state allow-opts label "8f94fc798bda6490718c255863a1b44b"
pass out log route-to (em0 192.168.12.1) inet from 192.168.12.241 to ! (em0:network) flags S/SA keep state allow-opts label "b41b7be98defffc9664e290483307513"
pass out log on em0 route-to (em0 fe80::e7c:28ff:fe8d:6ecc) inet6 from fe80::226:55ff:fed9:347e to ! (em0:network) flags S/SA keep state allow-opts label "ffbb2c655db9c62a861b03eb23cfa854"
pass out log route-to (em0 fe80::e7c:28ff:fe8d:6ecc) inet6 from 2607:fb90:92e6:bad6:226:55ff:fed9:347e to ! (em0:network) flags S/SA keep state allow-opts label "ffbb2c655db9c62a861b03eb23cfa854"
pass out log route-to (em0 fe80::e7c:28ff:fe8d:6ecc) inet6 from 2607:fb90:92e6:bad6:753b:6c47:0:c28 to ! (em0:network) flags S/SA keep state allow-opts label "ffbb2c655db9c62a861b03eb23cfa854"
pass in quick on em0 reply-to (em0 192.168.12.1) inet proto icmp from any to (em0) keep state label "36c461ebe1d3ab72bef3be5f6bb832a9"
pass in quick on em0 reply-to (em0 fe80::e7c:28ff:fe8d:6ecc) inet6 proto ipv6-icmp from any to (em0) keep state label "36c461ebe1d3ab72bef3be5f6bb832a9"
pass in log quick on em0 reply-to (em0 192.168.12.1) inet proto tcp from any to 172.16.0.20 port = 32400 flags S/SA keep state label "91761423d2f0b13ca21f64316568fd35"
pass in log quick on em0 route-to (wg0 10.0.0.1) inet from 10.0.0.10 to any flags S/SA keep state label "10f88dcc109e3b24af8aeb616bd038a7"
pass in quick on vtnet0 route-to (ovpnc2 10.8.8.1) inet from <win_pve> to any flags S/SA keep state label "67bd3a1545d2e6f663a32c14ad58a883"
pass in quick on vtnet0 route-to (ovpnc2 10.8.8.1) inet from 192.168.232.101 to any flags S/SA keep state label "f9a47d6928ee4e20a6b96ea0f5cbd983"
pass in quick on vtnet0_vlan700 route-to (ovpnc2 10.8.8.1) inet from <torrents> to any flags S/SA keep state label "d2f5665b37c7d4248400eb7643239e9c"
pass in log quick on vtnet0_vlan700 route-to (wg0 10.0.0.1) inet from 172.20.0.100 to any flags S/SA keep state label "91b197e08e5dfabf969724a5e5ce90a6"
pass in log quick on vtnet0_vlan700 route-to (wg0 10.0.0.1) inet from 172.20.0.101 to ! <internal_networks> flags S/SA keep state label "d00eb4c761d1022de0e38d5611b9d33e"
pass in log on vtnet0_vlan700 route-to (wg0 10.0.0.1) inet from <plex> to any flags S/SA keep state label "3b79ce325155e9998135e3ac7c6830a7"
Looks like another person using wireguard + port forwarding reported similar problem in: https://github.com/opnsense/core/issues/4389
-
I have exactly same problem and i switched to pfsense they fix this and in nat rules there is a difference.
On automatic WAN NAT rule i get not only LAN ip adresses but also ip adress from wireguard ip that how wiregurad interface can communicate tunnel over WAN interface.
On opnsense its not there thats why its disconnect wireguard gateway automatically when changing default gateway to wireguard interface :P
Wireguard interface ip subnet must be in WAN NAT source rule