LAN traffic being randomly blocked by "Default deny rule", can't figure out why!

[edjados]

Hi folks,

I have just deployed OPNsense as the gateway router for my home network, in place of an ISP provided router. All seems to be going pretty well, except for a developing problem with LAN to LAN traffic being blocked seemingly at random. This problem did not exist before the introduction of OPNsense in my LAN.

In my home network, I host a number of services behind a pfSense router, in a different subnet (10.0.0.0/22). OPNsense is configured with a static route to route this traffic to the WAN IP of the pfSense (192.168.1.11).

I can connect to these services just fine - the problem is keeping the connection! The firewall component of OPNsense is blocking packets destined for this subnet seemingly at random. This results in symptoms including RDP sessions regularly re-connecting, SSH sessions terminating after minute at best, and webapp use glitching or re-connecting.

Here is a screenshot showing these blocked packets in the log - https://imgea.uk/i/dvridjci.png
Here is a screenshot of the info view for one of these blocked packets - https://imgea.uk/i/temi64cs.png

That information points to Rule number 6, "Default deny rule" as the cause. Clicking the RID URL just leads to a 404, but I think I found the relevant rule in the Firewall Statistics view, screenshot here - https://imgea.uk/i/18ede2ma.png

I've created a diagram of how the LAN is structured, if it's of any use! - https://imgea.uk/i/iqjbcvx0.png

I just can't get my head around what might be causing this! I believe I've ruled out the pfSense box as the cause - the firewall log there is clear, and this issue only started happening with the introduction of OPNsense as my gateway router. I've heard "Out of state" traffic could cause this - I don't fully understand this, but I can say this appears to be replicable 100% of the time after initiating a new connection to a machine in the 10.0.0.0/22 subnet. Neither of the switches are doing anything funky such as port mirroring.

Could anybody here chuck any pointers my way for what could be causing this? Thank you

[EdwinKM]

Bit similar to https://forum.opnsense.org/index.php?topic=27894.0

Thanks for the clear drawing.

Why this double NAT? Would use 1 router/firewall to avoid all this problems.
pfSense will also normally block RFC1918 IP's on the WAN port. You must enable this feature if you really want/need this (usually only for lab experiments)

This guy also explained some things: https://homenetworkguy.com/how-to/use-opnsense-router-behind-another-router/

[sja1440]

The blocked packet that you provide seems to be a tcp ACK being sent from your Source Device (SD), 192.168.1.20, to your Destination Device (DD), 10.0.1.11. Unfortunately MAC addresses are not given in the logs.

My understanding of your configuration is that for busy tcp connections initiated by SD, then all packets pass through the OPNSense Router (RO) whilst for busy tcp connections initiated by DD packets do not pass through RO (this because 192.168.1.11 is on the same network as 192,168.1.20). On its own this asymmetry is probably not a good thing and will almost certainly cause problems.

I believe the scope for problems is further compounded by the presence of the left hand green switch.  Promiscuous mode on the OPNSsense together with REJECT (instead of BLOCK) rules would not help either.

Given that you say the problem hits intermittently after a few minutes of a working connection, my first thought, given your configuration, is that the problem might involve one (or all of):

• left green switch mac-port cache table entry being timed out. Switches maintain a cache of mac-port pairs which time out typically after a minute or so.  The important point being that when an ethernet packet arrives at a switch for an  destination not in its cache, the switch sends the packet out to to ALL ports.
• pfSense router ip neigh cache entry is timing out. This cache hold ip-mac entries. When the ip stack receives a request to transmit to an ip that is not in its cache, it will do an ARP request either for the MAC of the destination, if on the same network, or for its configured gateway otherwise.
• if OPNSense has the lan interface (em1) in promiscuos mode, then packets would be sent up the ip stack even if the destination mac was not that of the OPNSense router. Possibly the blocked ACK was infact in an ethernet packet sent to the MAC address of SD but because of the switch cache timeout, the same packet also got sent to OPNSsense which then rejected the message tearing down your connection

The precise details of how your problem is provoked would depend on the dynamics of who sends what and when.

If you really wanted to understand then you could use wireshark and monitor macs and ip addresses of the various messages on the various links.

In any case I would suggest that you change your network configuration. 

If you cannot change the network config then, depending on the details of your config, it might be worth trying the following possible dirty workaround:
* changing the LAN default firewall rule in question to be BLOCK instead of REJECT (in the OPNsense logs on my system they are both marked as "block') as this would prevent the OPNSence from sending an icmp reject to DD. With a bit of luck everything else might continue to work as it did before you swapped in the OPNsense.

Hope that helps.

[edjados]

Thanks for your reply an apologies for my delayed response! You put me on the right track there talking about 'asymmetry'.

After failing to replicate the issue when connecting to devices on the other side of those two switches (i.e. on the 'LAN2' side physically speaking), but with their gateway set to LAN1's OPNsense router, I was able to rule out the switches and point the blame at the routing/Layer 3 stuff. Wireshark showed clear issues with a ton of retransmits, "Spurious retransmits", "Duplicate ACKs", which pointed toward routing issues.

There was a static route on OPNsense directing all traffic bound for 10.0.0.0/22 to the WAN interface of the pfSense box - but there was not an opposing static route on the pfSense one. You described it better than me, but it seems traffic going from LAN1>LAN2 was going through both(?) routers, yet the return traffic was going straight from device to device, skipping out the routers (or it might have just been skipping out the OPNsense but still routing via the pfSense, I'm unsure on this).

I am still vague on the cause and haven't fully understood it yet myself, but when I implemented a static route on the pfSense box directing traffic bound for 192.168.1.20 to 192.168.1.1 (the OPNsense router), the issue immediately disappeared when using 192.168.1.20 as my test source device. It will not let me create a static route for the whole of 192.168.0.0/23, as it gives the error that it conflicts with the pfSense's WAN interface IP.

I thought this topology, which removed double-NATting by routing between the subnets using static routes, was the ideal setup, but it seems this is actually a bizarre setup which has caused issues. I plan on rebuilding the LAN2 (home lab) network using OPNsense, and creating a double-NAT setup to fully segregate that network and do away with the static routes.