Multi-Wan Failover with Cellular - Gateways don't recover

[ddickenson]

I've been trying to setup a multi-wan system with the following documentation: https://docs.opnsense.org/manual/how-tos/multiwan.html

I've had both gateways "working" as in passing traffic but when one is disabled or simulated disabled (by pulling the actual RJ out of the interface) first the failover doesn't seem to work and then when you plug the interface back in and it comes up the gateway stays disabled and doesn't come back online without a reboot or without manually enabling from the gateways-single interface.

I have the "main" gateway with a priority of 1 and the backup gateway with a priority of 254.  Once the "main" gateway is taken offline the secondary will indeed show active but it won't access the internet and even when the "main" interface is back up and online with an IP address the gateway shows grayed out triangle on the left and the priority shows "defunct."  If I go in and edit that gateway I can see "disabled" is checked and if I un-click that and apply or click the gray triangle and apply it generally comes back online.   Any idea why this should be needed if I'm using a gatway group?  Also if one of the gateways is "down" the gateway group acts like it doesn't even exist.  It'll only show the one that's active.  That means building the gateway group or editing it can not be done unless both gateways are online. 

I've got to figure  I'm missing something very important here but can't seem to find it in the docs

[lucky]

Hi

I have created a similar setup using

https://www.thomas-krenn.com/de/wiki/OPNsense_Multi_WAN

Failover is working fine, with different
   

Code Select Expand

       
                WAN                      WAN
                 :                        :
                 : DSL-Provider           : LTE-Provider
                 :                        :
             .---+---.                 .--+--.
         WAN |  DSL  |     Modems      | LTE | WAN2
             '---+---'                 '--+--'
                 |                        |
           PPPoE |                        | Ethernet
                 |                        |
            .----+----.              .----+----.
            | Router1 |    Router    | Router2 |
            '----+----'              '----+----'
192.168.120.1/24 |                        | 192.168.130.1/24
                 |      .----------.      |
                 +------| OPNsense |------+
     192.168.120.50/24 '----+-----' 192.168.130.50/24
                             |
                         LAN | 192.168.12.1/24
                             |
                       .-----+------.
                       | LAN-Switch |
                       '-----+------'
                             |
                     ...-----+-----...
                     (Clients/Servers)

I need to ensure that NO Data packet is going to the WAN2 LTE Modem/Router as long as WAN1/DSL-Modem is up. The redundant WAN has a "pay only if you use it" tariff
Any ideas how this can be achieved ? The ping via the second gateway is not preferable should I use "consider as always up"? or disable gateway monitoring)

I apologize for "hijacking" your thread but it looks so similar to my problem

Rgds  Lucky