Return traffic for VPN going out over default gw instead of the VPN.

[ferryvanaesch]

           |           _____________________        _______
           |__________/  GRE over WireGuard \______/ Linux \
        ___|____      \_____________________/      \_______/
       /  WAN   \
       \________/
           |
           |
        ________
       /  OPN   \
       \________/
           |
           |
        ________
       /  LAN   \
       \________/



Hi,

I'm not sure if this can be done or not. I'm currently running a pfSense firewall with an OpenVPN tunnel to a remote Linux server. When a connection comes in on the remote server's IP on say port 443, I do a NAT to direct the connection to a server on 'LAN'. The SYN packet comes in over the OpenVPN tunnel to the pfSense, the pfSense allows it through to the webserver, the webserver responds, and the SYNACK goes back over the OpenVPN tunnel to the remote server and things just work.
For performance considerations I want to start using WireGuard, and for that reason I'm setting up OPNsense and say goodbye to pfSense. I've set up the above. I can ping the WireGuard endpoint on the far side, and because I'll be routing from anywhere on the planet over this, I've set up a GRE tunnel on top of the VPN. And it almost works... Kinda.
Pinging from the local interfaces across the WireGuard and GRE all works, no issues there. However, when a connection comes in from a far-flung machine, the NAT'ed packet reaches the OPNsense on the GRE tunnel, who forwards it to the webserver, but then the problems begin: the return packet, when it reaches the OPNsense, doesn't go out over the GRE tunnel but instead goes to the default gw as per the routing table.

Is there any way to make these connections 'stick' to the incoming interface, in this case the GRE tunnel?

Regards,
Ferry van Aesch

[optic]

i have been having the same problem that i am unable to solve. The "best" solution so far was doing a 1:1 NAT that will work both ways. I had the same issue over Wireguard and GRE

[ferryvanaesch]

Interesting. My problem with a NAT is that I lose the original source IP, which I need for logging. (Or don’t I; how did you set this up?)
Does it work with a straight GRE tunnel without a VPN tunnel, or haven’t you tried that?

[optic]

i tried with a straight VPN tunnel. I also don't like the 1:1 NAT because of the same reason. I am kind of stuck on this problems for weeks now, please let us know if you find a solution.

[ferryvanaesch]

Will do. I'll try playing around with a few combinations to see if this is in anyway possible. The strange thing is that it works with OpenVPN on pfSense (so I assume it works with OpenVPN on OPNsense as well, but perhaps I should try that too...)

[ferryvanaesch]

Well, I found a way of getting it to work and keeping the source IP. It involves two NATs so I feel very dirty....

It's a workaround, but it will have to do as the performance gains of WireGuard over OpenVPN are too large to ignore.

So, instead of doing a NAT on the Linux server straight to the LAN server, I do a NAT to the GRE tunnel's IP on the OPNsense box. On the OPNsense box I add a second Inbound NAT rule to take that packet and forward it to the server on the LAN. This way the return traffic finds its way back on the GRE tunnel back to the Linux server and from there to the Internet. I preserve the source IP, and things now work.

Not an ideal solution by any stretch of the imagination, but again, it will have to do.

I'd love for OPNsense to start using the multiple routing table feature in FreeBSD. I use it extensively on Linux and it creates for a lot of flexibility.

Hope this helps your case a little.

[optic]

ferryvanaesch: i remember something i found out when i tested. Say you are accessing from a VPS with IP a.b.c.d - set a static route for a.b.c.d/32 via your remote GRE tunnel IP (gateway). It will work, so what is missing, is the default route for the GRE. Since it is not working even when setting up multiwan, i wonder if it is a bug.

[ferryvanaesch]

I’m not sure if it’s a bug or by design. PfSense acts the same way, which makes me think it’s a kernel ‘thing’.  I’m just not sure why it does work with OpenVPN, at least on pfSense.

I’m going to continue with the double NAT thing for now, hoping that sometime soon I can get rid of that second step. It feels like a dirty hack this.

[ferryvanaesch]

Ok, definitely some strange behaviour here. Possibly my fault, but I wouldn't know what/how. I added a few more tunnels, didn't change any of the NAT rules, and now OPNsense decided to send return traffic out on the WAN instead of the GRE tunnel again.
I simply must be doing something wrong, but I have no idea what...

[ferryvanaesch]

Ok, I think I just might have to give up on this and go create a Linux router just for the GRE/WG tunnel.
I did find this in the pf.conf man page, but not sure how this is handled by OPNsense:
(The reply-to is interesting)

Code: [Select]

ROUTING
     If a packet matches a rule with a route option set, the packet filter
     will route the packet according to the type of route option.  When such a
     rule creates state, the route option is also applied to all packets
     matching the same connection.

     route-to
   The route-to option routes the packet to the specified interface
   with an optional address for the next hop.  When a route-to rule
   creates state, only packets that pass in the same direction as the
   filter rule specifies will be routed in this way.  Packets passing
   in the opposite direction (replies) are not affected and are routed
   normally.

     reply-to
   The reply-to option is similar to route-to, but routes packets that
   pass in the opposite direction (replies) to the specified inter-
   face.  Opposite direction is only defined in the context of a state
   entry, and reply-to is useful only in rules that create state.  It
   can be used on systems with multiple external connections to route
   all outgoing packets of a connection through the interface the in-
   coming connection arrived through (symmetric routing enforcement).

     dup-to
   The dup-to option creates a duplicate of the packet and routes it
   like route-to.  The original packet gets routed as it normally
   would.

If I run a pfctl -s rules on the CLI, I don't see any reply-to. Could this be the problem?

[ferryvanaesch]

Well, getting there. I manually inserted "reply-to ( gre0 10.1.11.1 )" in the forwarding rule, and reloaded the rules with "pfctl -f /tmp/rules.debug". And that does the trick.

Code: [Select]

pass in log quick on gre0 reply-to ( gre0 10.1.11.1 ) inet proto tcp from {any} to $IP_Lab port {25} keep state label "d96d282773c24f3269134267e65aad05"
So I think my question is going to become a lot simpler. Am I correct in assuming that these reply-to tags are only inserted on WAN-type interfaces? If that's correct, how do I get OPNsense to think that my gre0 interface is a WAN-type interface?

[optic]

i wonder if this is simply a missing Gateway setting that would also set the reply-to for that interface. I tried "playing around" with that without any success.

thanks for keeping us posted!

[ferryvanaesch]

No problem, I keep going at this.

So, I read that in order for an interface to be considered a WAN interface, there needs to be a gateway assigned to the interface. To do this, I created a dummy gateway with IP 8.8.8.8 (Google must be sick at this stage of people using this IP for random stuff), and in the Interface settings for my gre0 tunnel I assigned this as the gateway.
Now OPNsense considers this a WAN interface, and the rules include the reply-to tags.

Things work....

Going to attempt a reboot now and see if the world explodes...

Update: Different problem, but I'll fix that.
So the reply-to issue works after a reboot, that's great. The GRE tunnel doesn't come up after a reboot though, that's something I'll need to look at. I have to go into the GRE tunnel settings, change nothing, and hit Save and Apply, and then it works. Must have something to do with the timing of the WireGuard VPN and the tunnel coming up at boot time. Could try with an IPSec tunnel as that's natively supported.
But the good news is that once the GRE tunnel is up, we're in business. The gateway on the Interface was the trick apparently.

[optic]

so i am getting grey hair over this setting up just a GRE tunnel :-)

What IPs do you give the GRE Tunnel - and what IPs the GRE Interface itself? Do you use only public IPv4, do you use private IPv4s on the GRE Tunnel and the public v4 on the GRE Interface? Some example screenshots/setup would be much appreciated! :-) Thank you!

[ferryvanaesch]

Ok, I'll try. Attached is a quick drawing of the different layers:

Internet obviously public IPs, on top of that I have the WireGuard tunnel between the two public IPs. It tunnels 10.1.10.0/30, where the OPNsense box has 10.1.10.2 and the Linux server 10.1.10.1. These are also the IPs of the interfaces (WG on OPNsense and wg0 on Linux).
On top of that I created a GRE tunnel between 10.1.10.1 and 10.1.10.2, and it routes 10.1.11.0/30. Linux gets 10.1.11.1 and OPNSense gets 10.1.11.2. These are also the IPs of the respective interfaces.
On the Linux server, I have a static route that points the LAN (192.168.1.0/24 for instance) to 10.1.11.2 (GRE endpoint on OPNsense) (ip route add 192.168.1.0/24 via 10.1.11.2 dev gre0)

The the last trick in the toolbox was to create a Gateway on OPNsense. I called it BogusGW, Interface WGGRE, IP is 8.8.8.8, checked 'Far Gateway' and 'Disable Gateway Monitoring'.
Then in the Interface section, on the WGGRE Interface, I use BogusGW as the IPv4 Upstream Gateway. This will enable the reply-to statements in the rules and then things magically start to work.

Hope this helps!

Update: Don't forget to drop the MTU to something like 1396 on the GRE interface to prevent fragmentation.