Multi-WAN Gateway Not Failing Back to Primary Correctly

[mmitchell]

TL;DR Firewall will not send all traffic out WAN1 after failing back from WAN2

Reddit Link: https://www.reddit.com/r/opnsense/comments/113yj1w/multiwan_gateway_not_failing_back_to_primary/

Details: I have a pair of HA firewalls with dual WAN. Each ISP has a /29 CIDR and the firewalls share an IP on each ISP using CARP. All of this is working as expected with one issue. After an internet outage on WAN1, when WAN1 comes back online most of our traffic does not move back to WAN1.

We will see about half the states, including new states, continue to stick to WAN2. Rebooting or unplugging WAN2 modem is the only way to force the states back over to WAN1. We waited as much as four hours before to see if we needed to just wait for states to time out, and this did not appear to change anything. The states remained evenly balanced between WAN1 and WAN2.

We have had a computer that was off during the failover get turned on and start new states through WAN2 even though WAN1 was already back online.

Important settings we have set:
System > Gateways > Single
* WAN1: Priority 100, Monitor IP 8.8.4.4
* WAN2: Priority 200, Monitor IP 1.0.0.1
System > Gateways > Group
* WAN1_failsto_WAN2: WAN1=Tier1, WAN2=Tier2
System > Settings > General
* Gateway switching: Checked
Firewall > Settings > Advanced
* Skip rules: Checked (was unchecked at one point with no change)
* Sticky Connections: Unchecked
* Shared forwarding: Checked (was unchecked at one point with no change)
* Disable force gateway: Unchecked (was checked at one point with no change)

Any thoughts on what we could be missing?

Log from one of our outages:
WAN monitoring:
2023-02-04T14:11:24-06:00 Warning dpinger WAN2_GWv4 1.0.0.1: Clear latency 29730us stddev 20756us loss 5%
2023-02-04T13:59:51-06:00 Warning dpinger WAN2_GWv4 1.0.0.1: Alarm latency 41388us stddev 4916us loss 22%
2023-02-04T07:01:33-06:00 Warning dpinger WAN1_GWv4 8.8.4.4: Clear latency 25941us stddev 4819us loss 13%
2023-02-04T07:00:44-06:00 Warning dpinger WAN1_GWv4 8.8.4.4: Alarm latency 21424us stddev 7845us loss 21%

Connections through UPN and spectrum throughout the day by the hour.
00 UPN: 18548
00 Spectrum: 3164
01 UPN: 18378
01 Spectrum: 3244
02 UPN: 17876
02 Spectrum: 3175
03 UPN: 18168
03 Spectrum: 3219
04 UPN: 17219
04 Spectrum: 3147
05 UPN: 16847
05 Spectrum: 3119
06 UPN: 18052
06 Spectrum: 3087
07 UPN: 10332
07 Spectrum: 18524
08 UPN: 9610
08 Spectrum: 12821
09 UPN: 11441
09 Spectrum: 15655
10 UPN: 14269
10 Spectrum: 15711
11 UPN: 11715
11 Spectrum: 18733
12 UPN: 12083
12 Spectrum: 14692
13 UPN: 10790
13 Spectrum: 13894
14 UPN: 19213
14 Spectrum: 4692
15 UPN: 17701
15 Spectrum: 3028
16 UPN: 19409
16 Spectrum: 3084
17 UPN: 16856
17 Spectrum: 3025
18 UPN: 17460
18 Spectrum: 3039
19 UPN: 17726
19 Spectrum: 2953
20 UPN: 33214
20 Spectrum: 3033
21 UPN: 22011
21 Spectrum: 2972
22 UPN: 17515
22 Spectrum: 3016
23 UPN: 19728
23 Spectrum: 2855

[BrummyGit]

Did you resolve this - I am facing the same challenge and burning all of my 5G allowances on my backup circuit