Dual WAN Failover : loose primary Gateway when workload is higher

[NEOSA]

Dual WAN Failover : loose primary Gateway when workload is higher

Hi All !

We are facing an issue with a DualWAN OpnSense Configuration.

Details :

- OpnSense version :    OPNsense 23.7.10_1-amd64
FreeBSD 13.2-RELEASE-p7
OpenSSL 1.1.1w

We have two identicals nodes, running as HA Cluster

- Fisrt Gateway :
Static IP - Priority : 254
Behind a 5G LTE Zyxel NR7101 Routeur
WAN IP is static

- Second gateway :
Static IP - Priority 255
Behind an aDSL routeur

- Gateway Group
Tier1 : Fisrt Gateway
Tier 2 : Second gateway
Trigger Level : Member down

- Issue we meet : when a quite big workload is initiate (ie trying to download a several Gb file), the lantency / packet loss grows up for the Fisrt Gateway, the firewall toggles from Tier1 to Tier2, but the Fisrt Gateway is ssen as back about 10 to 20 seconds later :-(

I tried to setup the gateways monitoring from 8.8.8.8 / 8.8.4.4 to ISP DNS servers : same issue.

I tried to change the Default Packet Loss thresholds for the First Gateway, from Default is 10/20 to 20 /40 : same issue.

Here is one of the last dpinger log :

Code: [Select]

2024-03-08T15:47:17 Notice dpinger ALERT: WAN_5G (Addr: 185.61.146.1 Alarm: loss -> none RTT: 30.7 ms RTTd: 6.3 ms Loss: 20.0 %)
2024-03-08T15:46:23 Notice dpinger ALERT: WAN_5G (Addr: 185.61.146.1 Alarm: none -> loss RTT: 31.2 ms RTTd: 6.7 ms Loss: 21.0 %)
2024-03-08T15:38:44 Notice dpinger Reloaded gateway watcher configuration on SIGHUP
2024-03-08T15:38:44 Warning dpinger send_interval 1000ms loss_interval 4000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 0ms loss_alarm 0% alarm_hold 10000ms dest_addr 185.61.147.1 bind_addr 192.168.1.251 identifier "LIVEBOX_ORANGE_GWv4 "
2024-03-08T15:38:44 Warning dpinger send_interval 1000ms loss_interval 4000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 0ms loss_alarm 0% alarm_hold 10000ms dest_addr 185.61.146.1 bind_addr 192.168.100.251 identifier "WAN_5G "
2024-03-08T15:38:44 Warning dpinger exiting on signal 15
2024-03-08T15:38:44 Warning dpinger exiting on signal 15
2024-03-08T15:37:16 Notice dpinger ALERT: WAN_5G (Addr: 185.61.146.1 Alarm: delay -> none RTT: 85.3 ms RTTd: 116.9 ms Loss: 0.0 %)
2024-03-08T15:37:04 Notice dpinger ALERT: WAN_5G (Addr: 185.61.146.1 Alarm: none -> delay RTT: 203.7 ms RTTd: 184.8 ms Loss: 0.0 %)
Do I need to increase the Packet Loss thresholds ?

The latency for this 5G LTE routeur is slitgly higher than the aDSL one, but no so much.

Thank you so for your opinon and/or any clues / ideas, if you also met such behaviours.