Issues replacing Fortinet F60 + Eero mesh with OPNSense

[dragonator72]

Hello all,

I'm attempting to replace my existing Fortinet F60 with a MiniPC (N100) running OPNSense. The OPNSense unit has 6 i226 nics configured as WAN (ig0), LAN (ig1) with ig2 and ig3 bridged to the LAN. The remaining 2 interfaces are currently unused.

I have an Eero 6 pro mesh network configured as a bridge for wifi access. At this point there are no plugins, etc., just base code. I'm having issues getting access for anything connecting via the Eero side of things via IP4. I have removed any devices connected to anything except the LAN interface and moved them all to a switch. At this point switching between the Fortinet and the OPNSense consists of simply plugging the LAN interface into the switch from whichever FW I want active.

DHCP is provided by my internal AD integrated DHCP servers. This all works flawlessly connected to the Fortigate but when I switch to the OPNSense the only things that have full access are wired connections. Anything connected via the Eero can't even ping the LAN interface on the FW. I did discover during troubleshooting that IPV6 does appear to work, just IPV4 is failing. I can ping all internal devices via IPv4 regardless, just no traffic getting to/through the FW. Kind of out of ideas on where to look next.

This is all setup behind my AT&T fiber connection. The FW WAN interface is on the 192.168.1.x subnet, the LAN is on 192.168.16.x.
At this point I'm wondering if there's some weirdness going on with the Eero but given that it's passing all the IPv4 traffic internally I don't know what that might be or where to look.

Any suggestions would be appreciated.

[pfry]

Have you examined "Firewall: Diagnostics: Sessions" or "[...]States" or used the live log ("Firewall: Log Files: Live View") on OPNsense? I use the live log quite a bit when troubleshooting - I select more lines on the right and hover over the "Auto refresh", as 25 lines will blow right by on my firewall. (Note that I have all rule logging enabled, including those under "Firewall: Settings: Advanced" -> "Logging". "Firewall: Log Files: Plain View" is useful, but you'll need this.)

I switched from a 60E; the only non-obvious difference I noted between the Fortigate and OPNsense was that the Fortigate silently handled (some permitted, some denied) various protocols, particularly broadcast/multicast in transparent mode and traffic aimed at the firewall itself. If you want to duplicate the Fortgate, you may have to characterize its actual behavior (as opposed to the visible policies) and manually configure OPNsense. Or not. One way to find out.

Somewhat aside: On my 60E I eliminated the switch (so the LAN ports appeared as individual ports) and used two vdoms, one transparent, one routed (to handle my bridged Internet service with static IPs). So my setup was pretty easy to duplicate on OPNsense. In particular, going to individual interfaces with interface-based filtering, even using bridges on OPNsense. If I was going from a virtual switch config with zone-based filters and no intrazone filtering to OPNsense with filtered bridges and interface-based filters, the design effort would have been a bit greater.

[dragonator72]

I haven't tried the diagnostics yet but I did look at the live log, that's why I say traffic isn't making it to the LAN interface. Those pings don't show up in the log if I ping via IPv4, but do via IPv6.

I'm not familiar with VDOMs. Virtual Domains perhaps? References or clarification would be greatly appreciated.

Thanks

[pfry]

I haven't tried the diagnostics yet but I did look at the live log, that's why I say traffic isn't making it to the LAN interface. Those pings don't show up in the log if I ping via IPv4, but do via IPv6.

Most IPv6 ICMP is permitted on OPNsense via automatic rules. You can view those rules under "Firewall: Rules: [interface]" by clicking on the button to the right of "Automatically generated rules" at the top of the ruleset. You can enable logging for those rules under "Firewall: Settings: Advanced" -> "Logging".

Edit: I got that backward. In order to see IPv4 ICMP, you'd have to have a rule that processes it (could be as simple as "pass all" or "block/reject all". Individual rule logging is under "Firewall: Rules: [interface]" -> [rule] -> "Log".

I'm not familiar with VDOMs. Virtual Domains perhaps? References or clarification would be greatly appreciated.

Ah, that's a construct on the Fortigate. It does not offer bridge support in routed mode and does not offer NAT in transparent (bridged) mode. So to get both, I configured the device for both modes, logically splitting the device into two (vdoms). It's not relevant to OPNsense - it was just an anecdote on porting from the Fortigate.

Speaking of bridges on OPNsense - do you have a bridge configured on yours? "LAN (ig1) with ig2 and ig3 bridged to the LAN" suggests you may, and you might reconsider that, as it might not behave the way you expect. Hard to say without more detail on your topology. If you're still having difficulty, post your interface configs and rules, and possibly shots of the live log and/or sessions/states. It is possible to post too much detail, of course, but it's easier to post too little.

I'm not familiar with eero, so I have no expectation of any unusual behaviors. (I cripple my wi-fi APs, so my experience is limited to that.)

[dragonator72]

OK, progress at least. I reconfigured and am just using igc1 directly as the LAN and everything seems to be working, so your comment about the bridge seems on target. Guess I need to do some more reading and figure out what I did wrong setting it up. Duplicating the Fortinet isn't a requirement, but it would be convenient to use the extra ports as I specifically bought that hardware to avoid needing a switch at that location.

I guess we can flag this a solved and I'll address bridge issues separately if required.

Thanks for the assist.