NAT reflection with reverse proxy running on OPNsense

[silentdragon95]

Hi,

I am running a NGINX reverse proxy on OPNsense with Let's Encrypt certificates (via the ACME Client) to be able to access various services from the internet, which works well.

However, I have the issue that I simply can't seem to get NAT reflection to work properly. Now, I can still my services using their external domain from the WAN network, but that is only because I am using two physical internet connections (with load balancing & failover), so every time I try to do this, the traffic exits via WAN2 and comes back in on WAN1 after routing through the internet. This is obviously not ideal, especially for large file transfers. Note that my externally offered services all go through WAN1, since WAN2 is a cable connection without a dedicated IPv4 address.

My issue is that all guides seemingly assume that the reverse proxy is running on another IP address on the WAN network, which would allow to route the traffic there - however that is of course not the case for me. I also tried to solve this via DNS rewrites, but pointing it to the IP of the reverse proxy (i.e. the internal IP of OPNsense) didn't work.

What am I missing?

Thanks in advance for your help.

[Monviech (Cedrik)]

Why do these connections route through the internet?

Even if the OPNsense has two WAN connections with two IP addresses, these IP addresses exist on the OPNsense.

The route precedence prefers local routes (interface routes), thats why the traffic should stick inside the OPNsense and get routed directly to either WAN1 or WAN2, hit your reverse proxy, and get then passed to your backends.

There is no reason for NAT reflection here, since no NAT is happening.

[silentdragon95]

Why do these connections route through the internet?

That's the million dollar question I guess. I can see the traffic exiting on WAN2 and coming in on WAN1 in the dashboard when uploading files to my Sharry filesharing service (also, the transfer speed matches with the upload speed of WAN2).

Could it be a DNS issue? I am currently running AdGuard via the OPNsense plug-in, however I am aware that this is technically not best practice.

[Monviech (Cedrik)]

It's probably forcing the traffic to the gateway because of the "reply-to" rule that is standard on the WAN interfaces.

Maybe if you put a firewall rule in either LAN or WAN with "This Firewall" as destination (before your other rules and with quick enabled), and select in the advanced features "reply-to" "disable" this behavior stops.

[silentdragon95]

I played around with your suggestion a bit and unfortunately it doesn't seem to fix the issue.

I was however able to observe that enabling the rule on WAN1 as well as WAN2 somehow results in the upload and the download being sent through WAN1, which I thought was impossible and the entire reason for NAT reflection existing. Oh well.

[Monviech (Cedrik)]

Another idea would be to specify the Gateway in the firewall rule so it prefers WAN2 for reaching "This Firewall".

If nothing helps, create a split horizon DNS where your domain names resolve to the LAN Ipv4 address instead, creating a firewall rule in the LAN that allows the traffic to This Firewall on 80 and 443.

[silentdragon95]

Okay so as it turns out, when both the upload as well as the download showed up on WAN1 yesterday, this actually meant that nothing was leaving the firewall and the traffic was instead being routed internally (turns out it probably would have been impossible otherwise after all, I feel less stupid now ).

I was able to verify this by checking the logs of the upstream DSL modem/Router. It also means that I need to tweak the settings of my Sharry webservice, because it currently doesn't seem to allow very fast upload speeds, which is why I didn't immediately notice. Anyway, that's a problem for another day.

There is still one issue with OPNsense however, it only seems to do this sometimes, other times it will still use WAN2 for the upload and in that case I can that the traffic is in fact being routed through the internet. Any ideas on how I can tell OPNsense not to use WAN2 in this case? Thanks for your help.

[silentdragon95]

Okay that last one turned out to be an easy fix (I should do less network stuff when tired). All that was needed was a rule on the LAN interface that specified WAN1 as the gateway for any traffic to the IP of my domain (and for this, one can use an Alias).

It works as intended now.