Dual WAN Failover config - Firewall logs show traffic still using down interface

[jamesb2147]

WAN1 is primary, WAN2 is secondary. Primary is used for all traffic unless monitoring stats aren't met (loss or latency), in which case it fails to secondary. Primary has 8x the bandwidth of secondary, hence the preference.

When I pull the upstream on primary (so its local L2 link stays online, but it cannot reach its gateway or internet), it fails to the secondary. I can successfully load web pages. However, I almost immediately start having trouble loading web pages. Looking at the firewall logs in OPNsense, I can see lots of traffic still being "allowed" out using the primary/WAN1 public IP address. However, tonight I did a packet capture on the secondary/WAN2, and found that all the packets had the appropriate secondary/WAN2 public IP address.

I'm not sure what to make of this, but it seems like there's a significant amount of traffic that's likely still trying to route via the primary/WAN1. I'm at a loss as to why that might be. Is there a common failover misconfiguration that might lead to something like this?

If you've read this far, thank you for your time and have a great day!

Version: OPNsense 20.7.7_1-amd64

[lar.hed]

Since you run trigger level as "loss or latency" instead of "member down" I think this is to be expected.

Or to put it like this: I have WAN_FTTH (fiber to the home, 250 mbit for the moment) and WAN_LTE (4G LTE16 that can push a rather large capacity if WAN_FTTH goes down). If I run "loss or latency", and both WANs are online, I consume about 3gb / month on the WAN_LTE just to keep it alive so to speak, so there is for sure a lot of traffic on my backup WAN_LTE. Changing to "member down" and well it still consumes some data on WAN_LTE but it is not even close to old usage (and since I pay for volume on WAN_LTE I like to keep that one low).