multi-wan failover problem

[hescominsoon]

in the previous release i could unplug one wan port and the system would fail over to number 2 without issue.  When the primary was restored it would fail back to the primary.  This is now NOT happening in the newest release.  This has apparently been a bug before..but now short of rebooting hte firewall it will not restore states back to the primary after a fail over to secondary.  any ideas?

[axsdenied]

I actually plan setting this configuration up over the weekend.  Will post here if I run into the same issue.

[hescominsoon]

so after further testing:

this is on official hardware the dec3850.
If we either pull the primary wan connection physically or disable it in the web gui failover to the secondary takes 5 seconds.  It used to be instant.  Another wrinkle is when the primary is restored it refuses to switch back,  Hitting save 0n an interface has no effect.  Disabling the secondary causes a 5 second loss of connectivity.  Otherwise a reboot is required.

[axsdenied]

I assume the monitor IP's are setup correctly so that it knows to switch back?  And you set the thresholds?

[hescominsoon]

everything worked correctly in the previous version.  only upon upgrading to 22.7 did it break.

[tcpip]

I also ran into this issue. Try setting static routes for the monitored IPs via the corresponding gateway. This solved it for me.

[hescominsoon]

i'm not familiar with that..what parameters in the static route would i use?

[hescominsoon]

i'm not familiar with that..what parameters in the static route would i use?

so set a static route on each gateway to the monitoring ip addresses if i am reading this correctly....

[hescominsoon]

trying to figure out why this is not working..

[hescominsoon]

I think I found the issue.
Wan1 has a route set to the monitoring ip of 1.1.1.1 AND the ip of 8.8.8.8 3even though 8.8.8.8. is cearly seutp for monitoring on wan2.  Looks like a bug either in freebsd or the opnsense code.

[hescominsoon]

duh forgot the cidr notation..got it.

[tcpip]

I think the issue is that the route for the monitoring IP of the WAN link gets removed as soon as the link is down. Therefore the monitoring checks don't work anymore. At least this is the case when I disconnect my primary WAN link. Setting the routes manually seems to be a decent workaround. However, I agree that it looks like a bug. I did't find time yet to dig deeper into the issue and file an issue on Github.

How is your multi WAN setup configured? Do you just use gateway switching or employ the gateway groups? Keep in mind that switchting back from WAN2 to WAN1 does not force all existing connections to switch back. The pf states are kept.

[hescominsoon]

gateway groups as failover.  what's weird is in 22.1 it would fail bac to the primary ip after ab out a minute.  i didn't have to do anything.  in 22.7 i now have to either forcibly disable the secondary wan or reboot the firewall for it to fal back.  if this non-going back to the primary is expected behavior..this is not the solution for me and my clients and will have to go back to another product.

[hescominsoon]

gateway groups as failover.  what's weird is in 22.1 it would fail bac to the primary ip after ab out a minute.  i didn't have to do anything.  in 22.7 i now have to either forcibly disable the secondary wan or reboot the firewall for it to fal back.  if this non-going back to the primary is expected behavior..this is not the solution for me and my clients and will have to go back to another product.

we installed 22.1 back on the appliance and restored the config.  It now reverts back to primary within seconds as verified by ipchicken on a desktop behind the opnsense.  No need for a static route eithe.  I think 22.7 needs a ton of work at this point.

[ProximusAl]

Interesting. I’m new to OPNSense and started with 22.7 beta.

You can see in my post here how I deal with the failover back to primary.

https://forum.opnsense.org/index.php?topic=29749.0

I just assumed OPNSense never did it, but did think it strange.

Maybe I should have started with 22.1, but I guess if I did that, I’d have the same issue as you (Technically still do)

EDIT: I should clarify “NEW” connections do use the primary ip when it’s back, but OPNSense itself is reluctant to fail ack, hence why I down the interface