Dual gateway | Reply-to?

[CMogen]

Hi, I am in the process of migrating things from a Firebox to OPN. Both firewalls host SSLVPN, and share a subnet with the LAN.

Code Select Expand

      OLD                         New
    ┌────────────────────┐     ┌────────────────────┐
    │ext   7.7.7.25 /26  │     │     7.7.7.33 /26   │
    │                    │     │                    │
    │tun  10.9.9.1 /24   │     │    10.8.8.1 /24    │
    │                    │     │                    │
    │int  10.10.10.1 /24 │     │    10.10.10.2 /24  │
    └────────────────────┘     └────────────────────┘ 
Things work fine if the target LAN machine's gateway matches the VPN entry point, otherwise the return traffic gets sent to the other firewall. The eventual plan is to change all the LAN to the new gateway, but I was hoping to figure out how to get this to work in parallel, if only to better understand the *sense inner workings and terminology.

I have tried disabling the reply-to WAN option, created an interface for OpenVPN, tried some different NAT options there, even created a static route on a LAN machine for the New tunnel network > New gateway, but I don't see that traffic coming back, so now I'm lost and not entirely sure what OPNsense is doing behind the scenes.

Is reply-to the right setting for this, or one of the NAT reflection WAN general rules?
What IP should the LAN nodes see coming from OpenVPN- the tunnel network IP, default WAN gateway, some other NAT? Does this change with NAT option x?
Do I need a outbound NAT rewrite to make the VPN traffic source look like New gateway perhaps?

If someone could break down the NAT automatic outbound/1:1/reflection/reply-to terminology and use cases I would really appreciate it. I've re-read the documentation on each several times and still can't get my head around what they actually do or when/where that would be applied, especially with ovpn in the mix. TIA

[dseven]

Your description is a bit vague. You say "things work fine", with no description of "things". I'm going to assume you have VPN clients connecting to these firewalls and they're trying to access some services running on LAN hosts?

So a client on "New" would get an IP address like 10.8.8.100, and might be trying to access some service on, say, 10.10.10.100. Because the default gateway on that LAN host is 10.10.10.1, the return traffic (destined for IP address 10.8.8.100) gets sent to "OLD" instead of "New". A static route for 10.8.8.0/24 pointing to 10.10.10.2 should solve that. Maybe look at that again. If you're sure you've configured it correctly, try a packet capture on the LAN interface of OPNsense to see what's happening (or not happening).

The options you're wondering about pertain primarily to the WAN side of things, but (if my assumptions/interpretation are correct), your problem is on the LAN side.

[CMogen]

I'm going to assume you have VPN clients connecting to these firewalls and they're trying to access some services running on LAN hosts?

Correct.

I believe most of the problem is coming from UDP used for VPN encapsulation. No matter what options I set on the New/OPN firewall, the return traffic always seems to follow the OS's default gateway, even skipping the

static route for 10.8.8.0/24 pointing to 10.10.10.2 should solve that

OS route entry.

I ended up restoring a backup from a couple days ago because I was so deep in the weeds with adding gateways, reply-tos, NAT, I wasn't sure what all was changed. After restoring the backup and creating a new TCP-based tunnel, it's still not working out of the box, but I assume I need to (re)set a gateway and reply-to for the VPN traffic?

Would this be a situation where I should have a LAN gateway (marked upstream)? Most of the docs advise against that unless absolutely necessary, but I'm not sure how else to bind the VPN to an interface and/or specify the reply-to as the LAN interface IP.

[dseven]

What do you mean by "OS route entry"? The static route route need to be added *on the LAN host*.

What makes you think that the problem has anything to do with VPN encapsulation? If that was the case, why would it work when the default gateway of the LAN host points to the firewall handling the VPN?

[CMogen]

What do you mean by "OS route entry"? The static route route need to be added *on the LAN host*.

It was, sorry if it wasn't clear. I said OS meaning Win/Linux route table vs something set in OPNsense.

What makes you think that the problem has anything to do with VPN encapsulation? If that was the case, why would it work when the default gateway of the LAN host points to the firewall handling the VPN?

Documentation I guess. I assume the same would apply in OPNsense. From https://docs.netgate.com/pfsense/en/latest/vpn/openvpn/multi-wan.html

The protocol choice for UDP on IPv4 and IPv6 on all interfaces (multihome) will work properly on all WANs and respond back using the address clients expect.

These other UDP modes in OpenVPN are limited by the connectionless nature of UDP. In these cases, the OpenVPN instance replies back to the client, but the Operating System selects the route and source address based on what the routing table believes is the best path to reach the peer. For non-default WANs, that will not be the correct path or the address the peer used when contacting this VPN.

[dseven]

You seem to be fixated on the idea that this is an OPNsense multi-WAN issue, but you don't have multiple WAN interfaces on your OPNsense firewall (or if you do, you have not described that). The documentation you quote pertains to the VPN protocol traffic routing back to the VPN client though the appropriate WAN interface (on OPNsense). That's not your issue (or not what you've described).

The static route on the LAN host should work. If it doesn't, I'd be doing packet captures to see what's going on.

Another possibility might be to add a static route for the VPN client subnet on the old firewall, pointing to the new firewall's LAN address. The old firewall might send an ICMP redirect, and the LAN host might follow it. This would have the advantage of only one place to add the route, rather than ever LAN host, but there are a few "mights" that have to happen for this to work.

[CMogen]

Well, it is a multi-WAN setup (from the perspective of a LAN host), they're just not on the same device, and I think that's most of the problem. If both LAN gateways were on the OPNsense, I'm sure conntrack (or whatever BSD/OPN uses) would auto-magically fix it. The UDP scenario could apply to my situation, and seems like that's exactly what's happening. It's either getting the external IP as a reply to, or it's a weird stateless multi-home UDP thing like this

You're right though, I guess I need to break out Wireshark and try to figure out what's going on. I was hoping someone might point me to some additional docs/workshop/utils, to better explain the flow and when/how those various NAT/reflection/reply options are set. I was trying to avoid manually figuring that out via trial and error and a packet sniffer.

Especially when I've had configs get "stuck" on a few occasions. Like deleting a gateway or interface, and seeing that if's rules still being applied to traffic somehow when I'm testing option F 20 minutes later. Without rebooting between every change, I'm now questioning if what I see in the GUI matches the running config. It's tedious already, and I haven't alt-tabbed to a packet sniffer between every change yet.

[viragomann]

Well, it is a multi-WAN setup (from the perspective of a LAN host), they're just not on the same device, and I think that's most of the problem.

You have an issue with VPN traffic. This has nothing to do with the WANs.

I don't expect, that a simple static route on the old router would work. I assume, it would end up in asymmetric routing issues and OPNsense might block the forwarded packets.
It would be necessary to separate the non-default gateway from the LAN completely and configure a transit network between the firewalls.

If only need to allow access from the VPN client to your LAN and don't care about the clients source IP, you can simply masquerade the traffic with the LAN IP of OPNsense.
This can be done with an outbound NAT rule like this:
interface: LAN
source: VPN tunnel subnet
destination: LAN network
translation: interface IP (LAN)

The outbound NAT must be set to hybrid mode (or manual) to enable this rule.