Best Practice for Routing

[m.desaive]

Hi all,

I could use some advice, as I am stomping my head over and over on a problem.

tldr; I have difficulties routing back replies into the initial entrypoint - an ipsec tunnel. I tried various ideas and finally got it to work with kind of a workaround.
 
Could be I am just thinking the wrong direction and try to use screwdrivers as hammers - I am new to OPNSense and maybe did not yet adopt the right picture. But I now have the time to learn it properly and would like to do it the "right" way. Maybe you can share some advice.

My question is not primarily on troubleshooting but more on: "What is the right tool for my problem and how to use it?"

Here the situation:

I want to connect my office to some resources in the datacenter. So I have two OPNSense boxes: gw.dc and gw.office which are connected by an IPSec tunnel. For various reasons I would like to keep the default route of the office traffic on my DSL uplink. (Maybe this is the reason for all the problems and stupid. I could revise this requirement, but actually think this _should_ be possible.)

With this setup I would like to let some services in the office, be reachable through gw.dc. That works perfectly with the web-resources. I address them using the haproxy plugin on the gw.dc. In that case, after the packets had been processed by the haproxy on gw.dc, they leave nicely into the ipsec tunnel with a source IP of the gw.dc. On the way back the routing rules work nicely and sends them back into the tunnel again.

The problem:

Packets for TCP and UDP services - I thought it would be straight forward to do a simple portforwarding/dNAT for those on the gw.dc. But this way, they start with the original internet IP as source IP and the return packets get routed to the default gateway on dc.office and never enter back into the tunnel.

What I tried:

1) With the nice behaviour of the haproxy packets, I thought I could simply sNAT the packets to some address of the gw.dc. So that replies should automatically find their way back. I read about the specialties and tunables for NAT on ipsec but I did not get that to work. The sNAT rules simply did not kick in. If that would be the "right" way I could put more effort on trying to solve this. And on the gw.dc I already have version 26 installed, here I tried both options: sNAT and Outgoing NAT.

2) Define rules on the gw.office for packets originating from the services ip and port and route them to the tunnel. Also the rules simply just did not kick in. I double,triple, thousand times checked but there should have been not rules "cathing" early. Maybe some problem with states? As I sayed I am relatively new to OPNSense/FreeBSD and still learning. Would it be worth to investigate further here?

3) I did a short journey with trying to tag incoming packets that arrive on the gw.offices ipsec interface and use the tag on the way back to do the routing decision? But I abandoned it, because it looks like these tags, will not be persistent when the packets leave the box and the replies will not carry the tag. Right?

What I took in to account but not yet tried:

4) The KI suggested it could be possible to do some magic with with general rules that only catch certain states and lead only the packets necessary to track the state of the connetion into the tunnel and let the main, big data packets travel throught the default gateway? Would remind me a little bit on "direct return" on a loadbalancer and could be really nice because I want to keep traffic over the DC as low as possible? But I did not dive into this further. Would it be worth? Or was this hallucination of a KI?

What now is working:

5) A tcp service for the haproxy or caddy on the gw.dc. Caddy should also make udp services possible, haproxy is not offering udp in the ui. right? But I expect this solution raise the latency for requests to my upd/tcp services. This actual setup is more a private hobby and learning thing, as I do Minecraft projects for for my son, his friends and some projects at a NGO. And the kids are critical customers and complain heavily if latency goes too high. ;) So if I could save some latency with the rdr solution, I would appreciate this.

So, seeing this picture - if I want to do it the right way, into what solution should I investigate troubleshooting? Or did I even miss something obvious?

Cheers,

Melanie

[viragomann]

So I asume, you've set up a traditional policy-based IPSec.
With this the VPN just connects the networks according to the policy.

So yes, if you want to forward traffic from public sources to the other side, you would need to use the network 0.0.0.0/0 for the datacenter side, which would lead into routing all upsream traffic from the office to it, which is not desired.

I don't know, how HAproxy or Caddy behaves in TCP or UDP mode respectively to replacing the source address. If it replaces it, it possibly could work with the "static route workaround".
This means, you have to add a gateway with the LAN IP (or any other interface IP which is treated by the IPSec policy) on the datacenter node.
Then add a static route for the remote network and point it to the LAN gateway.

If this doesn't work either your only option will be to use another VPN. Routing should work with any other: IPSec VTI, OpenVPN, Wireguard