[SOLVED] WAN Failover - Gateway Monitoring - Not working as intended?

[ProximusAl]

I need some help figuring something out with WAN Failover and monitoring, because myself and ChatGPT cannot figure out why this is happening.

My ISP perhaps once a day, has a little blip where I encounter some loss and latency of a very small margin.
We are talking perhaps 10-15 seconds of loss, and an increase in latency.

Normally, OPNSense would failover in this scenario, so I tweaked the settings to try and accommodate this loss.
Last night, it did it at 04:58, but my settings are configured below:

Latency Low: 200
Latency High: 2000   - This high to accommodate at least 2 seconds of latency before considering failover
Packet Loss Low: 10
Packet Loss High: 75 - This high to accommodate 75% of lost packets in 60 seconds, which equates to 45 seconds based on probe.
Probe Interval: 1
Time Period: 60      
Loss Interval: 20   - Immediately fail over if 20 consecutive pings are lost (20 seconds)

Last night, OPNSense failed over when I think it should NOT have done. Logs below

Gateway Log File:
2024-06-25T04:59:15   Notice   dpinger   ALERT: VIRGIN_DHCP (Addr: 1.1.1.1 Alarm: delay -> none RTT: 20.6 ms RTTd: 21.8 ms Loss: 1.0 %)   
2024-06-25T04:59:04   Notice   dpinger   ALERT: VIRGIN_DHCP (Addr: 1.1.1.1 Alarm: delay+loss -> delay RTT: 410.9 ms RTTd: 1240.2 ms Loss: 10.0 %)
2024-06-25T04:58:27   Notice   dpinger   ALERT: VIRGIN_DHCP (Addr: 1.1.1.1 Alarm: delay -> delay+loss RTT: 433.3 ms RTTd: 1272.6 ms Loss: 12.0 %)
2024-06-25T04:58:15   Notice   dpinger   ALERT: VIRGIN_DHCP (Addr: 1.1.1.1 Alarm: none -> delay RTT: 419.1 ms RTTd: 1263.4 ms Loss: 0.0 %)

You can see from the above, VIRGIN_DHCP never reaches the *down* status, so why failover?

Backend
2024-06-25T04:58:27   Notice   configd.py   [ec6f463a-0689-49ac-9396-c5444a7e5d7d] reconfiguring routing due to gateway alarm

Again, why.....it's only an alert, not a down.

Health
1030   Tue Jun 25 2024 04:58:00 GMT+0100 (British Summer Time)   3.402372   0.10887092677   0.28939024999
1031   Tue Jun 25 2024 04:59:00 GMT+0100 (British Summer Time)   12.35422725   0.35960223911   1.0485299431

The above is the total loss 3.4 + 12.35 = 15.75 well below the configured 20.

Can anyone explain why it failed over?  It looks like just the alert on it's own was enough to do it, in which case I need to increase my *LOW* thresholds to stop the alarm.

Regards

[ProximusAl]

Just looking at this again, is it perhaps I have my GateWay Group trigger as "Packet Loss".

Perhaps I should change this to "Member Down"

I can only surmise, that it detected packet loss...due to the alert only, and therefore failed over?

[ProximusAl]

Yep, just to confirm, it was because I had the gateway trigger set to Packet Loss.

I wrongly assumed that when set to Packet Loss, it would have to breach the Packet Loss *High* threshold on the monitor, but it's actually the *Low* threshold that triggers the fail over.

I've now changed it to "Member Down" which will only now failover if the *High* thresholds (Latency/Loss) are breached...or the ping count exceeds my limit.

Had a blip last night, and it didn't fail over.......joy..

2024-06-26T03:18:03   Notice   dpinger   ALERT: VIRGIN_DHCP (Addr: 1.1.1.1 Alarm: delay -> none RTT: 182.6 ms RTTd: 660.6 ms Loss: 0.0 %)   
2024-06-26T03:17:06   Notice   dpinger   ALERT: VIRGIN_DHCP (Addr: 1.1.1.1 Alarm: none -> delay RTT: 1307.5 ms RTTd: 3039.4 ms Loss: 0.0 %)