Firewall LAN rules not working as expected (Lan -> Lan blocked)

[Koloa]

Hello all,

I've done as much debugging on this one as I can work out, so, would appreciate suggestions.

I have a LAN network defined as 192.168.50.0/23 (with the LAN IP being 192.168.50.1).

I've set up the DHCPv4 server so that 51.10 through 51.200 are for DHCP leases.

I have a static IP on a local Raspberry Pi on 50.2, and the https port on the Pi is reachable from anything in the 50.0/24 part of the /23, but, nothing in the DHCP lease pool can reach the Pi.

The Pi's eth0 is correct:

Code: [Select]

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.50.2  netmask 255.255.254.0  broadcast 192.168.51.255

Most of the devices in the 51.0/24 part of the network are WiFi, and, I've made sure the issue is not the WiFi access point; I've also set a WiFI device to have a static IP of 192.168.50.12 and using WiFi that can reach the 50.2 host no problem.  It's only when the DHCPv4 lease IP of 51.x is used that the Default Deny rule of the firewall in OPNsense kicks in.

I can see the packets logged as such:

LAN      2022-09-13T09:08:35   192.168.50.2:443   192.168.51.10:53027   tcp   Default deny / state violation rule   
LAN      2022-09-13T09:08:34   192.168.50.2:443   192.168.51.10:53027   tcp   Default deny / state violation rule   
LAN      2022-09-13T09:08:33   192.168.50.2:443   192.168.51.10:53027   tcp   Default deny / state violation rule   
LAN      2022-09-13T09:08:33   192.168.50.2:443   192.168.51.10:53027   tcp   Default deny / state violation rule   
LAN      2022-09-13T09:08:32   192.168.50.2:443   192.168.51.10:53027   tcp   Default deny / state violation rule   
LAN      2022-09-13T09:08:31   192.168.50.2:443   192.168.51.10:53027   tcp   Default deny / state violation rule

And if you look at the details of one of the rules, it says:

__timestamp__   2022-09-13T09:08:35
ack   2551861249
action    [block]
anchorname   
datalen   0
dir    [in]
dst   192.168.51.10
dstport   53027
ecn   
id   0
interface   igb0
interface_name   LAN
ipflags   DF
ipversion   4
label   Default deny / state violation rule
length   60
offset   0
protoname   tcp
protonum   6
reason   match
rid   02f4bab031b57d1e30553ce08e0ec131
rulenr   14
seq   3121781003
src   192.168.50.2
srcport   443
subrulenr   
tcpflags   SAE
tcpopts   
tos   0x0
ttl   64
urp   65160

And the default rules which are in Firewall -> Rules -> LAN include:

Code: [Select]

 
IPv4 * LAN net * * * * * Default allow LAN to any rule    
IPv6 * LAN net * * * * * Default allow LAN IPv6 to any rule

Those rules are Pass, and In.

I have also tried adding rules which were IPv4+IPv6 in pass for any/any source destination, not specifically the LAN net, and, yes, I've also tried rules for OUT which were any/any.

The SYN packets are making it to the Pi, and the Pi responds:

Code: [Select]

09:14:50.815803 IP 192.168.51.10.53036 > 192.168.50.2.443: Flags [SEW], seq 2284235656, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2341413872 ecr 0,sackOK,eol], length 0
09:14:50.815934 IP 192.168.50.2.443 > 192.168.51.10.53036: Flags [S.E], seq 2610416432, ack 2284235657, win 65160, options [mss 1460,sackOK,TS val 1298482186 ecr 2341413872,nop,wscale 7], length 0
09:14:51.818903 IP 192.168.50.2.443 > 192.168.51.10.53036: Flags [S.E], seq 2610416432, ack 2284235657, win 65160, options [mss 1460,sackOK,TS val 1298483189 ecr 2341413872,nop,wscale 7], length 0
But the firewall is blocking the returning packets from the Pi to the 51.10 IP (in this case, an iPhone).

Curiously, I CAN reach other 50.0/24 hosts from the iPhone, which is puzzling.

And anything at all with 50.0/24 can reach 50.2 - it's purely running into that default deny rule for packets returning to 51.0/24 within the overall /23.

And the LAN network IP is correct as well:

Code: [Select]

igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
inet 192.168.50.1 netmask 0xfffffe00 broadcast 192.168.51.255

Any obvious thoughts as to what I'm overlooking for this incredibly simple and stupid question? 

[Koloa]

There's nothing like staring at a problem and changing something that you think "this could NOT possibly have ANY impact to the problem" - but it does.

Fixed.

WHY it broke is still unclear to me, but the fix was for me to move the 50.2 server to the same switch that is on my LAN port of my OPNsense device.

It was attached to a different switch, which is actually my WiFi AP device. 

So the traffic WAS:

iPhone -> AP -> Server (physically attached to AP)

And is now:

iPhone -> AP -> Switch -> Server

I'm guessing that the Asus WiFi device in AP mode is doing something funky that is confusing the OPNsense firewall as to where the packet originated from.