Multi WAN Gateway not failing back correctly.

[FullyBorked]

I'm not exactly sure what's going on here. But I'm having some struggles getting multi wan to fail back to primary interface once network is restored. 

Example.  Primary WAN fails, (in my instance it flapped for a few hours last night), WAN routing fails over to secondary and is mostly seamless.  However once everything stabilized OPNsense continues to send traffic down the secondary WAN even new connections.  Booted my PC for work this morning and all connections were routed down my secondary WAN.  The only fix I've found is to reboot the Secondary Modem, which forces another failover event.  This one is less seamless for some reason seems more "harsh" for lack of a better term causing things to break temporarily. 

Primary WAN has a priority of 253 and is set to tier 1 and secondary has a primary of 254 and is set to tier 2.  I understand sticky connections and why a connection that has failed over might want to remain there.  But new connections should take the new route to the primary WAN once it has failed back.  Not exactly sure how to troubleshoot this or improve upon it.       

[FullyBorked]

I believe the behavior I'm seeing is because "sticky connections" is enabled by default.  This is good if you are load balancing your connections as you don't want some security focused applications being disrupted due to packets exiting a different WAN interface.  However in a failover instance where my secondary ISP is metered and only for scenarios where my primary is unavailable, sticky connections were leaving existing connections "attached" to my secondary WAN connections until they expire.  So some connections were staying there for extended times. Disabling sticky connections seems to resolve this issue and allow for a more "forceful" failover and failback. 

[rhubarb]

I'm not exactly sure what's going on here. But I'm having some struggles getting multi wan to fail back to primary interface once network is restored. 
...

Did you get this resolved?  I am having the same issue, and Sticky Connections is not set.

[FullyBorked]

I'm not exactly sure what's going on here. But I'm having some struggles getting multi wan to fail back to primary interface once network is restored. 
...

Did you get this resolved?  I am having the same issue, and Sticky Connections is not set.

Not 100% disabling sticky connections helped.  With full on gateway outage, cable unplugged or isp fully down.  However when the isp flaps or doesn't go totally down, think heavy packet loss, it doesn't fail over either direction very well.  Not sure how to resolve it.

Sent from my IN2025 using Tapatalk

[klamath]

Having the same issue, I found that if I disable and enable any firewall rule and apply it the problem is fixed.  There must be something that isnt reloading the HA rules on a gateway failure.

[FullyBorked]

Having the same issue, I found that if I disable and enable any firewall rule and apply it the problem is fixed.  There must be something that isnt reloading the HA rules on a gateway failure.

I'm having this issue right this moment.  Primary ISP was flapping for a bit due to a storm.  Now I'm stuck on my secondary ISP.  Tried your method, but unfortunately that didn't seem to have any affect for me. Still can't figure out if this is a bug or just normal behavior.  I've used multi-wan on Sophos and Watchguard and they have settings to hard flip the connection back if you want.  Not what you want in a load balancing situation, but failing over from a backup metered connection that is a very nice feature. 

[DoomSalamander]

The seems very similiar to my issue. One of my WAN loses connection and routes the traffic like it should to the different WAN but at times doesn't route the traffic through the other WAN again when it is back up. I also don't use sticky connections.

[russella]

I'm not exactly sure what's going on here. But I'm having some struggles getting multi wan to fail back to primary interface once network is restored. 
...

Did you get this resolved?  I am having the same issue, and Sticky Connections is not set.

Not 100% disabling sticky connections helped.  With full on gateway outage, cable unplugged or isp fully down.  However when the isp flaps or doesn't go totally down, think heavy packet loss, it doesn't fail over either direction very well.  Not sure how to resolve it.

Sent from my IN2025 using Tapatalk

Mine works OK. You can change the thresholds for latency and packet loss on the System->Gateways->Single page. and choose which value triggers a switch System->Gateways->Group->Trigger Level.

My understanding of how it works (or can be configured) is as follows:

Member Down means triggers when either Packet Loss or High Latency exceeds higher threshold (so called To or Down status)
Packet Loss means triggers when Packet Loss exceeds lower threshold (so called From or Alert status)
High Latency means triggers when High Latency exceeds lower threshold (so called From or Alert status)
Packet Loss or High Latency means triggers when either Packet Loss or High Latency exceeds lower threshold (so called From or Alert status).

I use Member Down as the trigger with all the Thresholds set to their default values.

Also, I have System->Gateways->Single->Priority set to 255 for both the Tier1 and Tier2 gateway. You only need different priority if you have multiple gateways on the same Tier (e.g. when load balancing)