SOLVED: Multiple NICs. Multiple LANs. Can't get traffic back from IOT printer

[really_lost]

EDIT: see second post for details.  I had static routes on the system I was trying to reach the printer using an older (still existing) router/firewall.

I'm clearly missing something with inter-LAN traffic rules.

I've got multiple VLANs on my network.  However, I have zero VLANs defined on my OPNsense box.  OPNsense has multiple NICs and each is plugged into a switch interface where the untagged traffic is the LAN defined for that interface.  So, VLANs, but just different LANs as far as OPNsense is concerned.

For the most part, I want all the various LANs isolated, but I do want to permit some LANs to contact a printer in the IOT LAN.

It appears that whatever I do, the return traffic for that (e.g. https) is getting blocked on the IN of the IOT firewall.

Here's a simplified explanation.  I'm actually trying to permit a few different ports, but this example just uses https/443 since that is easy to test.


WAN -- (WAN/igb0) OPNsense (LAN/igb5) -- LAN (192.168.252.0/24)
                 (opt3/igb3)
                      |
                      |
                     IOT (192.168.255.0/24)
                      |
                      |
                   printer (192.168.255.196)   


Traffic seems to make it from LAN to printer but is always blocked from printer back to LAN at the IOT interface.

Outbound NATs for all LANs.  All using the same WAN address.

I've created a Group called ALL_LANS.  That group includes LAN and IOT.

There's a rule before any DENIES in ALL_LANS that permits "ALL_LANS net" to connect to the printer IP/32 using TCP/UDP port 443.

I even added an explicit ALL_LANS rule that permits the src IP/32 of the printer using a src port of 443, though I would think that was not needed.  I added this after having issues reaching the printer from the default LAN.

The IOT chain of rules looks like:

1. permit any traffic to the IOT address for DNS

2. permit any ICMP traffic to the IOT address.

3. permit printer ip/32 src port 443 to any.

4. block any traffic to dst port DNS and log.

5. block any traffic to dst net LAN and log.

6. permit src IOT net to any.

7. Deny any to any and log.


When I try to connect a LAN computer to the printer via https, it fails, and firewall logs show it's the return traffic from IOT printer to LAN computer that is blocked.  I suspect this is due to the return traffic not being recognized as related traffic.  I'm stumped as to what to do.

With rules set as above, that return IOT printer traffic is blocked and logged using rule 5.  If I disable rule 5, it is blocked and logged using rule 7.  If I disable rule 7, it is blocked and logged using the automatically generated rule of "Default deny / state violation rule."

I'm terribly confused.  I'd think if this return traffic is causing a state violation, the block would always happend in the autogenerated rules.

I've even tried creating NATs for LAN/IOT and placing those above the default WAN nat for LAN/IOT.  That did not seem to make any difference.

I'm certain I'm missing something simple and obvious. Any suggestions or questions welcome.  Pointing out where I'm clearly an idiot is really welcome if it helps me fix this and solve my issue.

[really_lost]

Fixed this.  I am an idiot.  What's worse, I didn't even describe enough in the original post to let anyone guess at the idiocy.

The multi-NIC OPNsense box is replacing a vlan aware linux box that has been acting as firewall/router and also hosting services on public IPs.  The machine I was testing printer connectivity to had had static routes set up to the other vlans using that still existing linux box.

I realized something weird was going on when I realized my ALL_LANS rule to permit this traffic wasn't showing any counters.

So packets were going my system -> linux router -> printer.  Return packets were going printer -> OPNsense, which dropped them because no state existed for the return packets.