[SOLVED] Assistance needed in setup of Reflection and Hairpin NAT

[cryotize]

Hi all, OPNSense newbie here...

I am having a lot of trouble setting up reflection and hairpin NAT. I have followed the offical best practice on how to configure it. See: https://docs.opnsense.org/manual/how-tos/nat_reflection.html#method-1-creating-manual-port-forward-nat-dnat-manual-outbound-nat-snat-and-automatic-firewall-rules

As stated, Method 1 is the preferred way, but none of the 3 described methods fully work in my case, and i don't know why.

Here's how my network is setup:
I have multiple VLANs / Interfaces (only the relevant ones are listed here):
ID 10 - Internal (10.10.10.0/24)
ID 100 - DMZ (10.10.100.0/24)
Interface WAN - one IP provided via PPPoE
Server IP: 10.10.100.6

Problem: I cannot access my Website from VLAN_10_Internal when using the external DNS Name. Funny thing is, Hairpin NAT works with solution 1, which is good. External access to my website (via Smartphone on LTE) also works perfectly.

I have tried many workarounds / solutions from this forum, all without any luck.Here is how i have setup the rules, following method 1 from the best practice:

Firewall -> Settings -> Advanced
Reflection for port forwards -> Unchecked
Reflection for 1:1 -> Unchecked
Automatic outbound NAT for Reflection -> Unchecked

Firewall -> NAT -> Port Forward
Interface: VLAN_10_Internal, VLAN_100_DMZ, WAN
Protocol: TCP
Source: Any
Source Port Range: Any
Destination: WAN address
Destination Port Range: from HTTPS to HTTPS
Redirect Target IP: Hosts Alias for IP "10.10.100.6"
Redirect Target Port: HTTPS
NAT Reflection: Use System Default
Filter rule association: Ass associated filter rule

Firewall -> NAT -> Outbound
Mode: Hybrid outbound NAT rule generation

Manual rule:
Interface: VLAN_100_DMZ
Protocol: TCP
Source Address: VLAN_100_DMZ net
Source Port: Any
Destination Address: Hosts Alias for IP "10.10.100.6"
Destination Port: HTTPS
Translation/Target: VLAN_100_DMZ address


According to the best practice, this should allow me to access the webserver from "VLAN_10_Internal" via external URL, but this does not work in my case. As already mentioned before, via "VLAN_100_DMZ" the webserver can access itself via external URL. (Hairpin NAT works, DNAT does NOT)

Thanks for any help, it would be greatly appreciated!
cryo

Edit: I have added a screenshot. I can see that the firewall lets traffic pass from my Client in VLAN_10_Internal to the webserver in the DMZ when i try to access it via external URL. But i still get a timeout on the Client side.

[Monviech (Cedrik)]

Hello, good job following the tutorial, everything looks right at first glance.

Please provide a complete tcpdump if you can.

Client - tcpdump or wireshark as it initiates the http request till failure.
Opnsense - at the same time, LAN and DMZ interfaces tcpdumps
Webserver - at the same time, Webserver tcpdump.

In total thats 4 tcpdumps running at exactly the same time, logging the packets of the same http request.

Here is a tcpdump example:

Code: [Select]

OPNsense A:~ # tcpdump -i hn0 proto ICMP -n
08:47:05.654822 IP 192.168.1.100 > 192.168.2.3: ICMP echo request, id 45, seq 1, length 64
08:47:05.656662 IP 192.168.2.3 > 192.168.1.100: ICMP echo reply, id 45, seq 1, length 64

Maybe its something simple like HTTP and HTTPS being needed both. Or the firewall of the webserver blocks requests from 10.10.10.0/24. (Because when there is DNAT only the destination is changed, not the source. Both your WAN and DMZ have additional SNAT rules that change the source to the IP of the DMZ)

[cryotize]

Thank you for your reply!

Of course, i have captured those 4 files at the exact same time on the required devices / vlans. They are all attached below.
Please note that i have captured only Port 443 in these 4 files, i might be able to capture port 80 aswell, but that would result in 8 files i guess.

Don't be confused by the client called "desk" in my client capture, desk is the hostname of my client. The real internal client IP is 10.10.10.100.

I will also setup the same rules for Port 80, like you've said. Maybe this is needed too. Additionally, i will try it with the SSH port too, just for the sake of trying.

[Monviech (Cedrik)]

Client
10.10.10.100 -> 185.72.67.157 -> SYN

VLAN10
10.10.10.100 -> 185.72.67.157 -> SYN

DNAT happening

VLAN100
10.10.10.100 -> 10.10.100.6 -> SYN

Server
10.10.10.100 -> 10.10.100.6 -> SYN

There's no ACK or SYN ACK.

The Server is the problem, it doesn't answer the SYN request from the Client. Maybe there's an ACL or Firewall there. The OPNsense does everything right judging the pcaps

[cryotize]

Interesting...

I followed the Guide for the HTTP and SSH Port aswell, without any luck.

Server seems fine to me, i use a dockerized nginx Webserver. Docker opens Ports automatically on Ubuntu server.
Now i have added 3 UFW rules manually, which allow access from any -> 80, 443 and SSH. Sadly, this didn't change anything either...

Do you have any other idea?

[Monviech (Cedrik)]

EDIT:

You can do a SNAT rule to circumvent this:

Firewall -> NAT -> Outbound
Mode: Hybrid outbound NAT rule generation

Manual rule:
Interface: VLAN_100_DMZ
Protocol: TCP
Source Address: VLAN_10_LAN net
Source Port: Any
Destination Address: Hosts Alias for IP "10.10.100.6"
Destination Port: HTTPS
Translation/Target: VLAN_100_DMZ address

[cryotize]

You're a GOD!

Thank you so much! It works now Many hours wasted... and still, i don't really understand where the problem lies.

Would you mind sharing some links or information, which explain this behavior and your proposed solution? I'd really like to understand this, so that i could help other people if they have the same problem.

Again, big thank you <3

[Monviech (Cedrik)]

Im glad it works for you now 

Well the problem is only circumvented this way.

The problem is your server, it only accepts requests from its own subnet. Aka 10.10.100.0/24. When a request from 10.10.10.0/24 is received, it doesnt answer.

What the SNAT rule does is this:

Client
10.10.10.100 -> 185.72.67.157 -> SYN

VLAN10
10.10.10.100 -> 185.72.67.157 -> SYN

DNAT happening

VLAN100
10.10.10.100 -> 10.10.100.6 -> SYN

SNAT happening

Server
10.10.100.1 -> 10.10.100.6 -> SYN

As you can see, the server now receives the request from the Firewall IP instead of the real client IP. The server needs to be configured right to allow requests from any IP address.

It can also be that your docker network does an additional NAT or Routing internally and doesnt know the route back to your client Network.

[cryotize]

Thanks - This seems to make sense.

If i check on which ports and interfaces the server is listening on, this doesn't seem to make much sense anymore :/
See the output of command "sudo ss -ltn" (obviously, i removed all other, unnecessary ports which are open too):

State             Recv-Q            Send-Q                        Local Address:Port                          Peer Address:Port            Process
LISTEN            0                 4096                                0.0.0.0:443                                0.0.0.0:*                                                   
LISTEN            0                 4096                                0.0.0.0:80                                  0.0.0.0:* 
LISTEN            0                 4096                                   [::]:443                                    [::]:*
LISTEN            0                 4096                                   [::]:80                                      [::]:* 

In your opinion - Should i still try to find out where the real problem lies, or can the circumvention be used as a permanent solution, without any security implications?

Edit: I will check the Docker + NAT thing and will give an updated report.

[Monviech (Cedrik)]

You can use this solution, it doesnt matter most of the time. Bit if youre curious, find out why you need it.

[cookiemonster]

just one thought. ss is showing there is a listener listening on those 80 and 443 ports. That's not the issue, but the listening process ie application might be configured to only accept requests from a particular network.
So is not a networking but a server/app configuration one -potentially-.