NAT reflection issues with double NAT

[Sahbi]

I couldn't find any thread about my specific scenario, hence my creation of a new one.

My OPNsense firewall is behind my ISP's modem router (double NAT) for a couple of reasons. We have one ISP connection shared amongst me and a handful of other tenants here and I don't trust them enough to just put the modem router in bridge mode and put OPN right behind it (since I'd have to physically move it into a public space). Also they have no business directly connecting to my devices over LAN instead of WAN anyways, as I run many devices and having to firewall each of them individually would be too much of a hassle (just for fun: I have 36 static IPs as well as a DHCP scope of 101 addresses). Some don't even have firewalling capabilities without writing something for it yourself.

The double NAT in itself doesn't seem to be a problem though and it's not inherently bad regardless, as long as you understand certain key points. OPN has been assigned as the DMZ of the modem router so I can access any ports I've forwarded to devices in my inner network without issue, and even VPNs work just fine. The problem arises when I access my public IP from within. Since OPN doesn't know about that IP, it sends it to the modem router which in turn doesn't do NAT reflection (I believe). I've been having some issues with packets being dropped upstream over the last few days, and if I access e.g. HTTP on my public IP it sometimes errors out. If it did do NAT reflection then upstream issues shouldn't be a problem.

My goal is to reroute traffic to my public IP back into the port forwards so that when I access e.g. port 80, it ends up at my web server. Ideally the web server also sees the client's LAN IP and not the OPN gateway, but I doubt that's possible because the client expects a reply from the gateway. Setting up split-horizon DNS is not really an option; I have many zones, records and servers/services so it would be a major pain to maintain all that. NAT reflection is really the only way.

I figured it would be as simple as attaching a virtual IP to the external interface and making sure NAT reflection is enabled on the port forward rule as well as Firewall > Settings > Advanced > Reflection for port forwards and Automatic outbound NAT for Reflection. However, when I do that then all traffic from inside to the public IP never gets to its destination. I can see it entering OPNsense through tcpdump (marked as "pass" and not "block"), but when I run tcpdump on the web server I don't see a single request coming in. Once I change the virtual IP to something else (last octet +1) then the exact same tcpdump shows a request from my WAN IP.

I even started messing around with outbound NAT, loopback interfaces, policy-based gateways and static routes but it simply wouldn't work reliably. At one point I was getting requests from OPN's IP on entirely different VLANs. :DD

Any ideas?

[Fright]

and how many services you need to reflect back to lan?

[Sahbi]

Pretty much all of them.

[Fright]

"Pretty much all of them."
great answer 
anyway
split dns imho is the right solution. but if you need more complexity in network:

Reflection is not some special technology. just automation of rule creation.
since opnsense knows nothing about real external IP you just need to create Port Forward and Outbound rules manualy.

one outbound rule for all traffic from lan to lan:
interface: LAN, source: LAN Net, source port: tcp/*, destination: Lan Net, dest port: tcp/*, NAT address: interface address.

and port forward rules for your services:
eg if want to do this with tcp 80:
Port Forward:source: interface LAN, proto tcp, address LAN net. Destination: *YourRealPublicIP*, ports: HTTP/S, redirect target: *lanaddressofyourhttpserver*, port HTTP/S

[Sahbi]

since opnsense knows nothing about real external IP

Well I simply figured that by attaching a virtual IP to the external interface, it would automatically pick up on that address and use it in its auto-reflection rules.

one outbound rule for all traffic from lan to lan:
interface: LAN, source: LAN Net, source port: tcp/*, destination: Lan Net, dest port: tcp/*, NAT address: interface address.

and port forward rules for your services:
eg if want to do this with tcp 80:
Port Forward:source: interface LAN, proto tcp, address LAN net. Destination: *YourRealPublicIP*, ports: HTTP/S, redirect target: *lanaddressofyourhttpserver*, port HTTP/S

Yeah exactly that is one of the earlier things I've tried, but in that case this happens:

getting requests from OPN's IP on entirely different VLANs. :DD

Those VLANs are even assigned to a different interface (igb2) than where my HTTP server is connected to (igb1). Oddly enough only the source IP is changed, it doesn't actually get a VLAN tag. For some reason the outbound NAT rule rotates between OPN's interface addresses.

[Fright]

i just quick tested solution i mentioned.
it works. no messing with virtual IP or VLANs needed.
it is not necessary that the ip must be on the firewall for rdr rule (port forward) to work.
and nat rule is needed just because target server and client is in the same subnet and server will reply directly to the client if requset will be forwarded without nat

[muchacha_grande]

I've been using Fright's method for a long while and it works ok.

[Sahbi]

it works. no messing with virtual IP or VLANs needed.

I'm not messing with VLANs as an attempt to fix the issue in this topic though, I simply need VLANs for my network.

it is not necessary that the ip must be on the firewall for rdr rule (port forward) to work.

It's not, I removed it because it only gave me more issues:

However, when I do that then all traffic from inside to the public IP never gets to its destination. I can see it entering OPNsense through tcpdump (marked as "pass" and not "block"), but when I run tcpdump on the web server I don't see a single request coming in. Once I change the virtual IP to something else (last octet +1) then the exact same tcpdump shows a request from my WAN IP.

The method you described generally works fine, except for this:

getting requests from OPN's IP on entirely different VLANs. :DD

Those VLANs are even assigned to a different interface (igb2) than where my HTTP server is connected to (igb1). Oddly enough only the source IP is changed, it doesn't actually get a VLAN tag. For some reason the outbound NAT rule rotates between OPN's interface addresses.

I figured out the cause of this in the meantime and it's not really a problem with the NAT rule in itself. I have an interface group AnyLAN, which contains my primary LAN interface (igb1) as well as the VLAN trunk interface (igb2). Now if you use that group in the outbound NAT rule for the Interface field and its "net" for Source, OPN seems to regard the IPs of all interfaces in that group as "interface address", instead of only the interface actually participating/receiving the client's packet (primary LAN). This results in OPN simply rotating through those IPs in a linear fashion (first primary LAN, then VLAN A, followed by VLAN B and then back to primary LAN). This seems like a bug to me, could you try reproducing that on your end?

[Fright]

sorry, from the first post I could not understand that you are using vlans not only for reflection testing.
can you please descripe your setup in more details?
and now i cant understand how your nat rule relates to forward rule.
what is in your port foward rule?
as i understand your client is on igb1 interface. on which interface is your server?
and most important: i realy think that use of interface groups in nat rule is not a perfect idea. its realy confusing. have you tried to make the NAT rules one by one (one rdr rule for specific client-server traffic -> one related nat rule) and check results?

[Sahbi]

can you please descripe your setup in more details?

Let's see if my ASCII art turns out ok:

Code: [Select]

OPN igb0 ------------------------------- ISP modem/router
OPN igb1 ----- switch ----- switch ----- server enp0s31f6
                  |
                  |-------- client
OPN igb2 ------------------------------- server enp6s0


The switch nearest the server is a dumb switch and doesn't support jumbo frames, so VLAN tags would be stripped which is why I have my VLAN trunk on a different interface (connected directly to a dedicated interface bridge on the server). My primary LAN is VLAN-less as I only need access to them from my server anyways.

and now i cant understand how your nat rule relates to forward rule.
what is in your port foward rule?

It's pretty simple:

NAT > Port Forward

• Interface: AnyExternal -- an interface group containing my external interfaces
• Protocol: TCP
• Source Address: *
• Source Ports: *
• Destination Address: Home_WAN -- an alias containing the external IP of igb0 (local to the ISP router, for its DMZ setting) as well as my actual external IP, if I just keep it at AnyExternal address (or net) then OPN still sends the request to the ISP router (which is to be expected)
  
• Destination Ports: 80
• NAT IP: web server
• NAT Ports: 80

NAT > Outbound

• Interface: AnyLAN
• Source: AnyLAN net
• Source Port: *
• Destination: AnyLAN net
• Destination Port: *
• NAT Address: Interface address
• NAT Port: *
• Static Port: no

When doing it like that, my web server receives requests from OPN's interface address on my primary LAN, then VLAN A, followed by VLAN B and then back to primary LAN. If I change the Interface in the Outbound NAT rule to Trusted_LAN instead (which is the name of the actual igb1 interface, not a group), then it only shows requests from OPN's IP on that network.

and most important: i realy think that use of interface groups in nat rule is not a perfect idea. its realy confusing. have you tried to make the NAT rules one by one (one rdr rule for specific client-server traffic -> one related nat rule) and check results?

That works, but it pretty much defeats the point of an interface group. The fact that such groups even appear in the settings of the NAT rules shows that they are/should be supported for that. If I decide to add a couple extra VLANs I have to remember to add more NAT rules instead of it being automagically handled by making them part of the proper group (AnyLAN). I don't really see how using a group would be confusing as long as the name is clear, though.

[Fright]

ASCII is wonderful 
became a little clearer.
sorry, one unrelated question: what will vlan do in this configuration?
on topic. Am I right? do you have only one port forward rule and it contains only external interfaces ? in this case, try to add intrenal interfaces to this rule. why do we need to drive a packet from the internal network to the external interface and back to the internal ones. let it go from one internal to another
if this is not enough and nat rule with a interfaces group still does not work, then you will need to look at the actual rules that the opnsense generates for pf. while the GUI does not allow it (I just requested this feature https://github.com/opnsense/core/issues/4331). Please, in the shell, give the command pfctl -snat and share the result. it will be interesting how opnsense registers nat rule with interfaces group

[Sahbi]

ASCII is wonderful 

Cool. :>

sorry, one unrelated question: what will vlan do in this configuration?

The VLANs are actually not really relevant, besides the fact they're part of the AnyLAN interface group. For example, I have one VLAN set to bypass a couple filtering options, e.g. the HTTP proxy.

on topic. Am I right? do you have only one port forward rule and it contains only external interfaces ? in this case, try to add intrenal interfaces to this rule. why do we need to drive a packet from the internal network to the external interface and back to the internal ones. let it go from one internal to another

I've added AnyLAN to the port forward rule and changed the outbound rule to work with AnyLAN instead of a specific interface. OPN still rotates between the 3 mentioned interface addresses when rewriting requests.

if this is not enough and nat rule with a interfaces group still does not work, then you will need to look at the actual rules that the opnsense generates for pf. while the GUI does not allow it (I just requested this feature https://github.com/opnsense/core/issues/4331). Please, in the shell, give the command pfctl -snat and share the result. it will be interesting how opnsense registers nat rule with interfaces group

Well, that clearly shows the problem:

Code: [Select]

rdr on AnyExternal inet proto tcp from any to <Home_WAN> port = http -> <DBWSRV> port 80 round-robin
nat on AnyLAN inet from (AnyLAN:network) to (AnyLAN:network) -> (AnyLAN:0) port 1024:65535 round-robin
For some reason it's added with a round-robin flag, but even changing the Pool Options to Source Hash doesn't do the trick. It does have a little note saying "Only Round Robin types work with Host Aliases. Any type can be used with a Subnet.". And sure enough, the pfctl command still says round-robin after changing it for both NAT rules. Which is weird imo, since the source address is still a subnet (AnyLAN:network). I generally find it pretty odd that round-robin is the default, I can't think of a good reason why you would want this behaviour (at least not for the source, you could use it for destination for load-balancing purposes).

[Fright]

Yes. this is exactly what I meant when I talked about the confusion when using groups in nat rules. it is not clear how to use the group in translation/target. shortly - no way)
and I do not think that Source Hash will help here. it will assign the translated address to the source, but it is not clear which address it will assign.
if we consider specifically your ASCII scheme and all servers in the igb1 network, then you can try to change your rule like this:
nat on igb1 inet from (AnyLAN: network) to any -> (igb1: 0) port 1024: 65535

or try to abandon nat and try asymmetric routing

and one more moment:
in rdr rule
rdr on AnyExternal inet proto tcp from any to <Home_WAN> port = http -> <DBWSRV> port 80 round-robin
i dont see "AnyLAN". why dont you want to  reflect client packet at AnyLAN interfaces?

[Sahbi]

Yes. this is exactly what I meant when I talked about the confusion when using groups in nat rules. it is not clear how to use the group in translation/target. shortly - no way)

But as I said, just the fact that you can choose those groups means it should be supported in an expected manner. Also the whole reason groups exists is so you don't have to duplicate the same rule, which is exactly the scenario I have here. The groups themselves aren't even the problem, it's that OPN's source IP rotates between the interfaces it has in that group instead of using only the interface actually involved. It doesn't even add VLAN tags if it decides to use one of the VLAN's gateway IPs. This is weird, unexpected and undesired behaviour.

and I do not think that Source Hash will help here. it will assign the translated address to the source, but it is not clear which address it will assign.

Yeah I know, it was just a test to see if I could get it off the round-robin algorithm. But no matter what option I pick, it will insist on using it.

if we consider specifically your ASCII scheme and all servers in the igb1 network, then you can try to change your rule like this:
nat on igb1 inet from (AnyLAN: network) to any -> (igb1: 0) port 1024: 65535

I tried that before though:

That works, but it pretty much defeats the point of an interface group. .... If I decide to add a couple extra VLANs I have to remember to add more NAT rules instead of it being automagically handled by making them part of the proper group (AnyLAN).

or try to abandon nat and try asymmetric routing

I don't think asymmetric routing will work in this case though? There's only 1 path between the client and server.

and one more moment:
in rdr rule
rdr on AnyExternal inet proto tcp from any to <Home_WAN> port = http -> <DBWSRV> port 80 round-robin
i dont see "AnyLAN". why dont you want to  reflect client packet at AnyLAN interfaces?

I'm just mostly testing some stuff, see if it works if I use standard rules for external connections and add NAT reflection/WAN loopback on top of that (without modifying the port forward rules themselves). But even if I change the interface to one of the LANs the same issue with the address rotation remains. I may have to change it to AnyLAN at some point anyways, e.g. if the external is down it may not trigger the rule because it's not passing out of the external interface. But I'll mess with that later.

Just to be clear: the redirection works fine when using only AnyExternal, the only remaining problem is OPN's rotation of source IPs when rewriting requests (which falls under Outbound NAT where I'm using AnyLAN, and not Port Forward). I think I'll bug the developers about that, since there's not a lot you can do about it anyways. But at least you helped me figure out the cause of this (I didn't even know about the pfctl command), so thanks for that. ;]

[Fright]

yes, sorry, asymmetric routing will not work in any way. client expects reply from public ip(
I don't think this is an OPN bug - OPN is exactly generating the rule you are asking for.
Interface groups usage is not an opnsense feature. its PFs feature (and OPN creates interface groups through ifconfig. not just stores in its configuration )
it's just that pf doesn't interpret the rule the way you want/expect it to. I have read pf man several times and do think that pf cannot do what you need (use the ip of the actual outgoing interface). this logic is not mentioned anywhere.  pf wait for (( redirhost | "{" redirhost-list "}) at this place of the rule. just addresses - no interfaces info at all. (so you cant tell pf "use current egress interface ip". he wants ip or list of ips)
long story short "Translation/target" is not interface-type field. its address-type field. and InterfaceGroup feature shouldnt work here by design.
so the opnsense team needs to write some feature that will "expand" your interface group and generate a set of rules (for each interface in group)? or ... just hide the interface groups in the Translation / Target dropdown (or add some more info text to the help div).
imho all this is just a slight difference in understanding of  the purpose of interface groups and where to use them