State Tracking causing issues for my LAN reverse proxy

[MiguelBazil]

Hello everyone, I'm very new to OPNsense (and proper firewalling), although I have passable knowledge in networking. I've hit a snag where I'm stuck in how to debug or solve this properly.

First things first, what I have and what I'm trying to achieve. My previous network was under a standard ISP router, so I was trying to first achieve some form of "parity" with what I had.

I have a PiHole for the network, which is configured as the default DNS, and it works quite well for what I want. I also have an Unraid server where I keep most of my network services. This includes an NPM (nginx) service, that works to provide network names for my services through reverse proxying. I don't do it directly on the Pi because I want to not remember the ports for all of them, so that means NPM does that job. I am using a custom domain name, out of the fun of it (.rabbit). What I always had was something like:

browser > plex.rabbit/ > get directed to plex > watch stuff
browser > nginx.rabbit/ > get directed to NPMs web page to configure my redirections
etc.

After I placed the OPNsense in between my ISP router and the rest of the network, part of those services stopped being properly redirected, or even allowing access through direct IP:Port to the webpages, and that's what I'm trying to solve.

The normal DNS resolution works perfectly (I know OPNs has something similar to PiHole, but I'm not going to use it for now), internet access seems to be working as expected, and a good portion of my services still work. The services that mostly failed critically for me are admin access to PiHole and nginx configuration pages.

After some sniffing around in the live view, I figured it was due to the firewall dropping TCP packets, which had the SA flags, which I'm assuming that the FW detected them as out of order, or the connection being incorrectly closed. Once I set a rule to disable state tracking inside my LAN, things started working again.

My network is as follows:

• My ISP router is now connected to my OPNsense box and works as the WAN, with network 10.0.0.0/24;
• My Lan is currently set up as 192.168.68.0/22, because I want a wider number of IPs, due to assigning a good portion of them to services (I know it can be done differently, but I have fun doing it this way);
• DHCPv4 is assigned from OPN, only in the .68 range. The LAN IP from OPN is 192.168.69.1;
• Unraid is on 69.254
• PiHole is on 69.250
• NPM is on 69.244

Name resolution for the LAN goes Device -> OPNs -> PiHole -> Nginx -> Service.

Now, I don't quite understand what's causing these packets to drop. I can see that the name is resolved as it should, but then it gets dropped once it is IP redirected. If I try to access the PiHole admin page "192.168.69.250/admin" from the browser, it still gets dropped, so it's not just from the reverse proxying acting up.

Can anyone give me some clues as to how to debug this? Or is this a known type of issue already that I completely missed when researching, because I don't know some important name of the issue I'm facing?

[dseven]

My guess is that you have an incorrect netmask on either the NPM server or the clients. If one of those is using /24, it will route traffic for the other half of your /23 via the firewall instead of delivering it directly on your LAN (because it thinks it's not part of the local subnet). If the other host is using /23, it will route directly. This results in the firewall seeing only one side of the conversation, which will be invalid for (strict) state tracking. If you want traffic between the clients and the servers to pass through the firewall, you should use separate VLANs/subnets.

[MiguelBazil]

So if I understand correctly, some unexpected routing is happening in this process. I will double check all netmasks just to be sure. Another thing is that I use an L2 switch after OPN, I do wonder if an L3 wouldn't do the trick too. Another point is that a lot of these services are running also inside the docker network, so I suppose some routing decisions might happen in the process that could cause this.

If you want traffic between the clients and the servers to pass through the firewall, you should use separate VLANs/subnets.

In essence, since OPN can't see the whole exchange, it's refusing the packets because it can't determine the state of the communication. Solutions are to either separate things physically or logically to force all communication to happen through the FW, or figure what part is being routed when it shouldn't, right?

[dseven]

Yes, I think that's right. A third option might be to use "sloppy" state tracking, but that's a bit of a hack, IMO.

You could do some packet captures to verify whether or not this is actually what's happening (i.e. that the firewall only sees traffic in one direction).

[Monviech (Cedrik)]

You can solve this 3 ways:

1. Put Reverse Proxy in a seperate vlan than all your servers or clients.

2. Use a Reverse Proxy on the OPNsense directly (I suggest Caddy)
https://docs.opnsense.org/manual/how-tos/caddy.html

3. If Reverse Proxy and servers/clients share the same network, use Hairpin NAT.
https://docs.opnsense.org/manual/how-tos/nat_reflection.html

[MiguelBazil]

A third option might be to use "sloppy" state tracking, but that's a bit of a hack, IMO.

Not bigger than the no state tracking I'm using right now, for the LAN interface ;)

In any case, thanks for all the suggestions. I'm going to dig down on them and see which makes more sense in my case.