OPNsense Forum
Archive => 22.1 Legacy Series => Topic started by: opn69a on March 09, 2022, 06:11:13 pm
-
I originally posted this in Tutorials and FAQ, but I'm thinking that was the wrong place to post this question... So re-posting here and editing original thread to point here (Feel free to delete the previous topic, mods)
Hi folks,
This is something I've been trying to tackle for a few days now and am just at a loss at what's going on. Let me explain my scenario first, what I'm trying to do, and then show what I'm seeing...
My Scenario:
I have my OPNsense installation setup with Wireguard and it hosts two VPN connections over two services. The Primary (WG0) is for almost all the network traffic to be masked around - no problem. The secondary (WG1) is for other traffic on the network - this works fine when going to specific external IP addresses or websites (thanks to certain NAT rules).
My primary VPN connection has a few open ports for certain things I need access to, and those work perfectly fine. I reach out to the open port over the public VPN IP and I can reach those services - great.
My secondary VPN connection now has a specific port I need open, and this is where I'm running into problems.
What I'm trying to do:
On the secondary VPN, I have opened the port to allow certain IP addresses in. The service on my server is listening to that port and accepting traffic, locally and externally (to those certain IP addresses). I can access this service locally, no problem. However, when I try to access this externally, it doesn't work. I've setup this port to be open and have validated in firewall logs that nothing is being blocked. I have also validated that the server gets the SYN request ands ends back a SYN ACK. My router in between the firewall and server also receives both the SYN and SYN ACK just fine and sends the SYN ACK back to the firewall. So as far as I can tell, the router and server running my service are both working exactly as intended.
The problem I'm seeing:
From what I can tell from monitoring packet capture, the firewall (OPNsense) appears to be receiving information on that port that's being forwarded just fine. However, when traffic comes back through from my router, it's trying to send it out through my primary VPN (WG0) instead of secondary (WG1). I've gone as far as trying to create certain rules for the secondary VPN to use the WG1 gateway when it receives requests on this open port but nothing I am doing seems to be working. I've tried everything I can possibly think of... and have finally got to the point of asking the forum to see if someone else has any insight on how to fix this.
For convenience, I have provided the packet capture logs below. I've masked IP addresses and ports for privacy purposes, but left them to be descriptive to hopefully give a visual on what's going on. I've also placed the interfaces (WG0/WG1/LAN) in order of when the calls were captured. As can be seen, WG1 receives the external source IP + port to forward. I believe LAN is coming in next (they're the other way in the packet capture log, but the "TS Val" appears to match up), which is where my router is telling the firewall where to send the SYN ACK to. Then, from here, is where things go wrong - the firewall is taking this and trying to send it back out through WG0 instead of WG1
EXTERNAL_SOURCE_IP -> source
SRC_P -> source IP port
INT_P -> port being forwarded to internal network
PRIVATE_VPN_IP -> Private VPN IP for secondary VPN (the one with the port being forwarded)
WG0 -> Belongs to primary VPN
WG1 -> Belongs to secondary VPN
LAN_IP -> IP of LAN network
WG1:
EXTERNAL_SOURCE_IP.SRC_P > PRIVATE_VPN_IP.INT_P: Flags [S], cksum 0xe2b2 (correct), seq 3055569791, win 65535, options [mss 1340,sackOK,TS val 3863205 ecr 0,nop,wscale 9], length 0
EXTERNAL_SOURCE_IP.SRC_P > PRIVATE_VPN_IP.INT_P: Flags [S], cksum 0xdec7 (correct), seq 3055569791, win 65535, options [mss 1340,sackOK,TS val 3864208 ecr 0,nop,wscale 9], length 0
EXTERNAL_SOURCE_IP.SRC_P > PRIVATE_VPN_IP.INT_P: Flags [S], cksum 0xd6f2 (correct), seq 3055569791, win 65535, options [mss 1340,sackOK,TS val 3866213 ecr 0,nop,wscale 9], length 0
LAN:
EXTERNAL_SOURCE_IP.SRC_P > LAN_IP.INT_P: Flags [S], cksum 0x761a (correct), seq 3055569791, win 65535, options [mss 1340,sackOK,TS val 3863205 ecr 0,nop,wscale 9], length 0
LAN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P: Flags [S.], cksum 0x8121 (correct), seq 1587531018, ack 3055569792, win 65160, options [mss 1460,sackOK,TS val 3550934679 ecr 3863205,nop,wscale 7], length 0
EXTERNAL_SOURCE_IP.SRC_P > LAN_IP.INT_P: Flags [S], cksum 0x722f (correct), seq 3055569791, win 65535, options [mss 1340,sackOK,TS val 3864208 ecr 0,nop,wscale 9], length 0
LAN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P: Flags [S.], cksum 0x7d36 (correct), seq 1587531018, ack 3055569792, win 65160, options [mss 1460,sackOK,TS val 3550935682 ecr 3863205,nop,wscale 7], length 0
LAN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P: Flags [S.], cksum 0x791a (correct), seq 1587531018, ack 3055569792, win 65160, options [mss 1460,sackOK,TS val 3550936734 ecr 3863205,nop,wscale 7], length 0
EXTERNAL_SOURCE_IP.SRC_P > LAN_IP.INT_P: Flags [S], cksum 0x6a5a (correct), seq 3055569791, win 65535, options [mss 1340,sackOK,TS val 3866213 ecr 0,nop,wscale 9], length 0
LAN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P: Flags [S.], cksum 0x7560 (correct), seq 1587531018, ack 3055569792, win 65160, options [mss 1460,sackOK,TS val 3550937688 ecr 3863205,nop,wscale 7], length 0
WG0:
PRIVATE_VPN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P: Flags [S.], cksum 0xedb9 (correct), seq 1587531018, ack 3055569792, win 65160, options [mss 1460,sackOK,TS val 3550934679 ecr 3863205,nop,wscale 7], length 0
PRIVATE_VPN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P: Flags [S.], cksum 0xe9ce (correct), seq 1587531018, ack 3055569792, win 65160, options [mss 1460,sackOK,TS val 3550935682 ecr 3863205,nop,wscale 7], length 0
PRIVATE_VPN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P: Flags [S.], cksum 0xe5b2 (correct), seq 1587531018, ack 3055569792, win 65160, options [mss 1460,sackOK,TS val 3550936734 ecr 3863205,nop,wscale 7], length 0
PRIVATE_VPN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P: Flags [S.], cksum 0xe1f8 (correct), seq 1587531018, ack 3055569792, win 65160, options [mss 1460,sackOK,TS val 3550937688 ecr 3863205,nop,wscale 7], length 0
My first guess initially was the gateway was being set incorrectly. However, I added rule and changed it around several ways to try and force the WG1 gateway... It didn't make a difference. I've tried the following (playing by memory here, didn't actually document it at the time):
Interface, Source, Source Port, Destination, Destination Port, Gateway
LAN, LAN Net, INT_P, *, *, WG1_Gateway
WG0, WG0 Net, INT_P, *, *, WG1_Gateway
LAN, WG0 Net, INT_P, *, *, WG1_Gateway
WG0, LAN Net, INT_P, *, *, WG1_Gateway
LAN, LAN Net, *, *, INT_P, WG1_Gateway
WG0, WG0 Net, *, *, INT_P, WG1_Gateway
LAN, WG0 Net, *, *, INT_P, WG1_Gateway
WG0, LAN Net, *, *, INT_P, WG1_Gateway
None of these seemed to work... the packet capture looked the same every time. At this point, I'm wondering if there's some setting or something I'm missing that needs to be set. Or perhaps I just don't have a full understanding on how the firewall listens to the interface (LAN) for the internal network traffic (since this is a SYN ACK situation and not outging NAT).
Any guidance/help would be much appreciated!
EDIT:
I'm still having trouble with this. I'm starting to really feel like this is a bug, lol. These lines in particular are why:
LAN:
EXTERNAL_SOURCE_IP.SRC_P > LAN_IP.INT_P: Flags [S], cksum 0x761a (correct), seq 3055569791, win 65535, options [mss 1340,sackOK,TS val 3863205 ecr 0,nop,wscale 9], length 0
LAN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P: Flags [S.], cksum 0x8121 (correct), seq 1587531018, ack 3055569792, win 65160, options [mss 1460,sackOK,TS val 3550934679 ecr 3863205,nop,wscale 7], length 0
LAN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P -> The LAN Interface is telling the firewall it wants to send back that SYNACK to the extneral source. Adding this Rule: LAN, LAN Net, INT_P, *, *, WG1_Gateway -- this should be changing it to go through the WG1 gateway. BUT I keep seeing this:
WG0:
PRIVATE_VPN_IP.INT_P > EXTERNAL_SOURCE_IP.SRC_P: Flags [S.], cksum 0xedb9 (correct), seq 1587531018, ack 3055569792, win 65160, options [mss 1460,sackOK,TS val 3550934679 ecr 3863205,nop,wscale 7], length 0
The PRIVATE_VPN_IP is correctly set to the WG1 address, but the interface it's being sent through is wrong... Is the Gateway configuration only updating the source IP and not the interface it's being sent through when responding? Just for grins, I also added the following NAT Outbound rule: "Interface: WG0, Source Port: INT_P, NAT Address: WG1 Address"... Didn't change anything from the packet capture.
-
FINALLY, got it figured out!!! I want to share my answer here for anyone else who may be looking for how to do this in the future.
Assuming your 2nd VPN interface is called WG1:
1. As you would probably expect, the port forward rule should follow:
Interface: WG1
Protocol: TCP (or whatever you're expecting it to be)
Source: any (or whatever you're allow-listing)
Source port: any
Destination: WG1 Address
Destination Port: The port you're forwarding
Redirect target IP: The specific LAN IP you're forwarding to
2. Now you need to create a LAN rule that has the following:
Interface: LAN
Direction: out
Protocol: Same as above
Source: Same as above
Source port: any
Destination: LAN net
Destination port: The port you're forwarding
Gateway: default
Advanced > reply-to: WG1 Gateway (IMPORTANT!!)
The key thing that I required was the Advanced > Reply-to setting. I hope this helps anyone else who may be trying to setup a secondary VPN w/ a port forward :)
-
Thank you so much for posting this.
Had the same issue and it was driving me crazy.
With you post I was able to make it work.