Best practice for my specific setup?

[gardiol]

I have a home network behind two ISPs (one for failover) both behind CG-NAT. In the middle i have opnSense (updated to latest) configured with one LAN and two WANs for failover, no load balancing.

Inside the home network (flat 10.10.10.0/24 IPv4) i have a web-server providing many web services on port 443

To ensure access from when i am not home, i have a public domain (mydomain.com) and an external server with static IPv4 address. I have direct SSH tunnel from the internal web-server to the external server so that any connection to external-server:443 gets tunneled back to internal-webserver:443.

Everything works perfectly, up to this point, opnSense is not involved (except for default-routing internal-webserver outside)

When i am outside home, i access my services perfectly.

When i am inside home, i want that my traffic gets routed inside, not outside home: first of all because i don't want to waste my bandwidth, my data caps (yes... sad), and in general for privacy and security, since my web server is internal, i want it to respond to connections from within the home network directly.

The solution i found so far is to add to opnSense Unbound some overrides so that when i am inside home, services that would normally resolve to the external IP would then resolve to the internal IP of the web-server. This is... ok, but somehow is causing issues with mobile devices that, when they switch from cell network to internal wifi, quite often don't play well. I guess that the dns resolution gets messed up and such. Moreover, this approach would not work if i enabled DNSSEC on the domain itself, of course (not an issue atm, but...)

What i would like is the opnSens to automatically route all traffic from hosts inside the home network directed to the external IP directly to the internal web server, some kind of DNAT or port-forwarding in my mind.

I tried to setup a port-forwarding rule in opnSense, but it doesnt work. When i do so, things breaks and tcpdump gives me errors like:

Code Select Expand

09:26:25.964003 IP 10.10.10.99.35664 > mydomain.com.https: Flags [S], seq 449436517, win 32120, options [mss 1460,sackOK,TS val 3557438781 ecr 0,nop,wscale 7], length 0
09:26:25.967983 IP 10.10.10.99.35668 > mydomain.com.https: Flags [S], seq 2980147547, win 32120, options [mss 1460,sackOK,TS val 3557438785 ecr 0,nop,wscale 7], length 0
09:26:25.996378 IP mydomain.com > 10.10.10.99: ICMP host mydomain.com unreachable - admin prohibited filter, length 60
09:26:26.281152 IP mydomain.com > 10.10.10.99: ICMP host mydomain.com unreachable - admin prohibited filter, length 60

this errors when i try to open mydomain.com from a chrome browser inside the home network.

where 10.10.10.99 is my client (a linux box), while the internal server has 10.10.10.1, openSense has 10.10.10.254 and the external server has "mydomain.com".

The port-forward rule is pretty simple: from LAN, take any source traffic directed to ip of mydomain.com, port HTTPS, and forward it to 10.10.10.1 port HTTPS.

I followed the docs, i think correctly. Also, the second outbound NAT rule "should" not be needed in this case, as both client and internal server are on the same subnet, i think?

Is there a better, or correct, way to do this? Why is it not working according to the docs, it should?

Any help, inputs or clarification would be great!

[bartjsmit]

Have you looked at an overlay network? Tailscale or Zerotier will only route over the WAN if you are external.

[gardiol]

Can you elaborate a bit? What is an overlay network? Aren't tailscale and zerotier paid services depending on an external company? Something I want to avoid

[bartjsmit]

Nodes on an overlay make outbound connections only. You can route to all your internal hosts by setting OPNsense up as a subnet router. That means that your nodes connect to the internal IP's when they are on the LAN and use the overlay when they are not.

I use Tailscale which is freemium with a 100 node free plan. https://tailscale.com/kb/1097/install-opnsense

There is a FOSS version if you want to self host or VPS: https://headscale.net/

Bart...

[gardiol]

I see, but this is not what im looking forward to. I need to DNAT a single IP address, nothing else. The external access issues are already solved in my setup, without using anything additional.

[gardiol]

I have finally found the solution.

Porting here for future reference, as well for other users.

I could NOT make it work on the same LAN, which might make sense since that would mess up standard ethernet routing. So i added a DMZ interface on a different subnet, to the server.

First of all, you need DNAT (port forwarding) on BOTH DMZ and LAN itnerfaces, from "NET net" as source, destination set to the public IP i want to redirect from, and the DMZ server address as redirection to.

Then you ALSO need a SNAT (outbound) rule to nat traffic from LAN to DMZ, because in my case my server is BOTH on LAN and DMZ networks. What was happening was that the request would be DNATted to the DMZ, but with the source address of my NAT network client, so the reply from the server would be routed trough the server LAN interface, and could not be matched by the client from the reply, because it was coming from a different IP address.

Hope it clear...

TLDR:
- rule 1: DNAT (use BOTH interfaces under "interface", and this baffles me, plus the /32 on the destination host)
- rule 2: SNAT from LAN to DMZ to ensure responses can route back.

Note: rule 2 might not be needed if your server is NOT on LAN as well as on DMZ.