Weird Gateway/Routing Failure

[rkubes]

I have OPNsense installed on a device with 4 Intel 2.5GB NIC cards, and one Wi-Fi card.

One of the four NICs are used for the WAN. The Wi-Fi card is used for a Wi-Fi failover, in case WAN goes down (to connect to my hotspot). It's very rarely used, but it works correctly when needed.

The other three NICs are for three different LAN networks. All of my firewall rules are working correctly and as expected under normal circumstances.

With that said, on one of the three networks I don't always have the downstream switch powered on (usually off over the weekend). It does have a firewall rule that allows the devices on that network to reach out using the failover gateway (same as the other two networks). So, it will primarily use my main WAN, but can failover to use the Wi-Fi backup when needed.

What is odd is sometimes when I power up that network and its devices, nothing on that network will be able to reach out over the WAN. Everything internal between the LANs will work. All of the devices that are supposed to be able to route to each other over the three different LANs will all be fine. But that third LAN will not have WAN access.

I can "correct" it, by going into the firewall rules, and pick any rule, and just toggle a setting. Such as turn logging on, then turn logging back off. Once I do this the "Apply" button appears and I can hit apply and that network regains it's WAN access. I know it's some kind of issue/defect (as opposed to an incorrect or missing allow firewall rule) since I'm not actually changing a configuration to get it going, all I'm doing is making it think I changed a config so I can get it to reload the firewall.

This "workaround" will hold until the next time I power down that network for a long time. However, usually after being off for a weekend, when I get it back on, the issue occasionally (usually) presents itself again. I believe this started happening with 23.7.10, but it may have started with a release prior to that. This is definitely a "newer" issue over the last couple of months, after being stable for almost the full year of 2023.

Are there any other tips/tricks that I can use to try to diagnose exactly what is happening, and why that network does not seem to respond right away? How is it that getting the "Apply" button with no changes gets it to work? Should I just try restarting the "pf" service or some other service next time? Are there any specific logs I should review to try to identify what's going on?

Any assistance will be greatly appreciated. This isn't an "urgent" issue as I'm able to work around it. The workaround does hold as long as I keep the downstream switch powered on. However, I usually turn that whole network off over the weekend, thus it's an annoyance to have to log in and "fix" this most weeks.

[Patrick M. Hausen]

When you power down the switch the corresponding port on OPNsense will go down, too. That means the IP address of that interface will go away, all routes using that interface will be deleted from the routing table and all services with explicit listen interface set will stop listening in that interface. E.g. Unbound.

Depending on more details of your configuration things will not come back up consistently when you power on the switch.

Solution 1: don't do that (power down the switch)

Possible solution 2: make sure all services have their listen address/interface set to "all (recommended)".

[rkubes]

Thanks. I checked Unbound and it is already configured to listen on all interfaces. One of the tests I did when it wasn't working was pinging IPs directly to rule out a DNS issue, and I could ping anything local, even the router itself - but could not ping anything on the other side of WAN.

Is there a service that specifically handles the routing through the gateway that I can check? Or any other configurations you suggest reviewing?

I appreciate the thought for "Solution 1", but unfortunately it is also just a work around. I could not accept an environment where unplugging and replugging the LAN cable would break access to the Internet until services on the router are restarted.

Edit:
Also, if it was an effect of services no longer listening, because the port goes down, wouldn't it be consistently reproducible? It's not every time that the port cycles that the WAN access/routing doesn't come back up, it's just more often than not. I'm not sure yet if the length of time its down has an impact.

[Patrick M. Hausen]

These were just the ideas that came to my mind. I would youse packet tracing (tcpdump) to analyse the system once it's in the failure state. Check IP address of LAN, check local ping, check DNS resolution, trace and observe packet flow ...