Dual-WAN - one interface misbehaving causes problems for entire router

[ZPrime]

I am running dual WAN with two different cable ISPs. Both modems are in bridge mode, both WAN interfaces are DHCP. The two ISPs are Spectrum and WOW Cable.

I do not (currently) attempt any load balancing; I just have two different gateway groups that are setup for failover, with one connection preferred over the other. However, I also have a few rules in place to policy-route traffic from specific hosts out each of the connections - these hosts are "monitor-io" monitoring devices, so I can get "user friendly" visual status indication of problems with one provider or the other. (The monitor-io is a really cool device to give easy at-a-glance status of your uplink, I use them at home and one at my parents' house... but it's offtopic here.) Important to this story: I currently have WOW set as the primary connection, and Spectrum is the secondary.

Yesterday, Spectrum was having significant problems; packet loss was through the roof, and latency was very high as well. dpinger was constantly toggling the connection between "alarm" and "clear" states every minute or two. (I have all of the interval / timer settings for gateway monitoring at the defaults.)

It seemed like every time the Spectrum gateway went from bad to good status, something was being reloaded on opnsense, and it was causing traffic interruptions for management sessions to the firewall, as well as causing a fair amount of delay on traffic going out to the (still functional) WOW connection.  I also noticed (via SSH log output, before I'd get kicked off) that the "nut" service (UPS monitoring) seems to get restarted each time this "something" gets reloaded, too.

In order to get everything to a usable state again, I had to disable the gateway for Spectrum (clicked the enable/disable toggle to the left of the entry, by the checkbox, then applied).

After disabling the gateway for Spectrum, policy routing was not behaving like it should. One of my policy routes is supposed to send one of the two monitor-io devices out the Spectrum connection, and only that connection... but it was not working. The device that should've been monitoring Spectrum was showing "all good," even though I had manually disabled that gateway.

So, my two main questions:

• What exactly is supposed to happen when a gateway goes from online to warning / bad due to packetloss or latency? Should it be interrupting nut? Should it be affecting SSH sessions to the firewall/router itself? Any idea why it was impacting other traffic flow through the firewall via the remaining good WAN link?
• When a gateway is manually disabled (not "marked as always down," but fully disabled), should policy routing still be working against that gateway? My intent is for certain traffic to just be dropped on the floor in this state, but it wasn't happening.

The hardware in play here is not exactly new - quad-core Atom C2558 on a Supermicro A1SAi, quad Intel igb interfaces, and plenty of RAM (32GB ECC). Beyond nut and UPnP, I don't have anything special on the system; it's a pretty basic firewall (other than dual WAN and a few policy routing rules). I did not notice significant load spikes (even when dpinger was changing the gateway state)... CPU would blip a little, but it wasn't sitting at full load or anything. Load average was no higher than 1 or 1.5 on the shorter end, the longer timeframes were obviously lower.

I'm happy to share my rules or config if someone can give me a suggestion on the best way to do that while anonymizing passwords / etc.

[Heathy65]

Hi,
I just posted a question which, whilst not the same as yours, seems to perhaps have some similarities to your issue: https://forum.opnsense.org/index.php?topic=23961.0

Did you get to any conclusion/resolution with your issue?