WAN failover problem - Seems to route through wrong WAN interface

[fishingboat]

Hello!

Setup:
WAN1 is connected through a mobile gateway, static IP, no DHCP.
WAN2 is connected through a VSAT (satellite) gateway, static IP, no DHCP.
Two LANs, each with their own physical interface and subnet, static IPs, no DHCP.
WAN failover configured as per the OPNsense documentation.
Zenarmor Sensei is running on the system.

Description of the problem:
When WAN1 fails, the System>Gateways>Single menu shows WAN2 as active, to be expected.
Furthermore, the system routing table shows WAN2 as the default gateway, also to be expected.
The live view log on the firewall however,  shows traffic trying to leave through WAN1, leaving the LANs completely without internet access.
This does not happen every failover, but when it happens, the system does not fall back and I have to resort to a system restart or disabling the WAN1 gateway.

Other Information:
I've had this happen on both 21.x and 22.x, on different machines.
I had to fix it immediately this time so I simply disabled one of the gateways, the following screenshots reflect that.

WAN1 Gateway:


WAN2 Gateway:


The single gateway overview:


Gateway Group:
This screenshot is taken after I disabled the WAN1 gateway, when enabled, it's set as Tier 1


System>Settings>General - Networking
The top DNS is set to the WAN1 gateway when the gateway is enabled


Firewall rules for the main LAN:
It has to be able to connect to devices on all networks, no matter the current default gateway.


I'm pretty new at this, but I've searched the forums and tried applying settings that other people dealing with WAN failover problems have suggested, such as fidgeting with Sticky Connections and Reply-To.

If I'm missing some crucial information please let me know.

Any insight greatly appreciated :)

[fishingboat]

I just noticed now that I cannot access the router of WAN1 after having disabled the gateway, which I used to be able to when I first set the system up, no matter the default gateway being used. 
Doesn't that sound like some sort of routing problem?  It'd be in line with the problem of the post.

[Tre144]

Man, the support for opnsense sucks.


So many posts have no replies.

I too am struggling with a dual gateway setup.

[franco]

True story: so many posts also have replies.


Cheers,
Franco

[SomebodySysop]

I don't have a solution to this as I am struggling to design a failover setup myself.

Question:  What is your hardware config for the basic failover setup from WAN1 to WAN2?

WAN1-----|
               | OPNSense Router |----->LAN
WAN2-----|

I guess what I really want to know is: are the ethernet ports for both WAN1 and WAN2 both physically on the OPNSense router?  My router only has two physical ethernet ports: WAN and LAN.  I'm trying to figure out if I need to upgrade the hardware to accomodate 2 WANs (att broadband, att cellular) + LAN, or is there some other workaround?

[BoratsBodyguard]

Yes.  I have 4 NICs physically on my OPNsense firewall.  WAN1, WAN2, LAN, spare

Same issues when a WAN fails
   If WAN1 (preferred / heavier weighted) fails and the firewall fails-over to WAN2 it sticks on WAN2 and never fails-back to WAN1 when it comes back online.  I have normally had to physically unplug WAN2 for a short duration to create the fail-back to WAN1 - which works but not how it's supposed to work.

Both WAN1 and WAN2 are DHCP with manually configured DNS.
WAN failover configured as per the OPNsense documentation.

Has there been any work on this issue?

I don't have a solution to this as I am struggling to design a failover setup myself.

Question:  What is your hardware config for the basic failover setup from WAN1 to WAN2?

WAN1-----|
               | OPNSense Router |----->LAN
WAN2-----|

I guess what I really want to know is: are the ethernet ports for both WAN1 and WAN2 both physically on the OPNSense router?  My router only has two physical ethernet ports: WAN and LAN.  I'm trying to figure out if I need to upgrade the hardware to accomodate 2 WANs (att broadband, att cellular) + LAN, or is there some other workaround?