Prevent traffic being redirected to another gateway

[j_s]

I have 2 ISPs, once is via cablemodem, the backup is a "wireless fiber".  Basically its to help me avoid downtime if my cable internet goes down.  I have learned that my wireless fiber tends to have issues randomly, and the "fix" is to power cycle the PoE tower on my roof.  I'd like this to "autocorrect itself", so I bought one of these:

https://dlidirect.com/products/new-pro-switch

Seems to work well, and I can manually trip it appropriately.

So my plan was to have the Pro Switch ping 1.1.1.1 via my cable internet, and 8.8.8.8 via my wireless fiber.  If pings fail for 10 minutes, reboot the associated.  So I set it up, and via gateway settings in my LAN interface, I have it direct to the appropriate gateway.  Works great, and the traffic goes where it's supposed to.

The problem, if I make the wireless fiber go offline, then those ICMP packets start going out my cable internet and since those succeed, the Pro Switch never knows that anything is wrong, so it won't power cycle the interfaces.

I believe that my settings are to blame.  Under the gateway configuration I have both ISPs with my cable internet with a priority of 10, and my wireless fiber with a priority of 50.  I already had 2 gateway groups, one with my cable internet as a tier 1 and wireless fiber as tier2, and another with the opposite.  I did even create gateway groups where each of my isps is tier 1 independently and tried setting that in opnsense but I still cannot get the traffic to fail.

After some back and forth with Googling and discussion with ChatGPT, apparently opnsense seems to choose the "good gateway" regardless.  Can someone explain this to me?  I don't think this is the answer despite that being the observed behavior.  I'd like the pings to actually fail if the associated gateway is not working.  This is my first time trying to force basically "all traffic out one ISP", but also allow "some traffic out the other ISP, and for this traffic to fail if the ISP fails".

[j_s]

Okay, hopefully this will help someone someday.  After more than 10 hours of testing things over 2 days, I think I got it figured out.

First, get your Gateways setup how you want them to be with priority, etc.

Now here's how I got it to work on 25.7_4:

1.  Verified that "Allow default gateway switching" was checked by going to System -> Settings -> General.  Don't forget to Save if you changed the setting. (I did this because I'd rather let everything go out whatever is the "default" *except* my Web Switch Pro.
2.  Go to Firewall -> Settings -> Advanced and check "Skip rules when gateway is down".  Don't forget to Save.
3.  Create 4 rules on your LAN interface.

# Action Interface Protocol Source Destination Gateway Description
1 Pass LAN ICMP (Web Pro Switch IP) 1.1.1.1 Main_ISP_Gateway Web Pro Switch to 1.1.1.1 (Enforce Main ISP)
2 Block LAN ICMP (Web Pro Switch IP) 1.1.1.1 * (default) Block 1.1.1.1 if Main ISP Fails
3 Pass LAN ICMP (Web Pro Switch IP) 8.8.8.8 Secondary_ISP_Gateway Web Pro Switch to 8.8.8.8 (Enforce Secondary ISP)
4 Block LAN ICMP (Web Pro Switch IP) 8.8.8.8 * (default) Block 8.8.8.8 if Secondary ISP Fails


Make sure your 4 rules above are before your "Default LAN to any" rule.  Apply and Save.

Rules 1 and 3 work when the associated Gateway is up.  Rules 2 and 4 prevent the traffic from being redirected to the other gateway on failure of that ISP.  The "default" gateway is basically a "gateway of last resort".  By blocking that traffic, you're blocking the pings from being redirected to the "gateway of last resort".