Failover wont work between Comcast to HugesNet

[JPERRY]

I have comcast as a primary (Opt1) and HughesNet into Opt2 on a ProtectLI.  I've followed the Mulit Lap directions on OPNsense https://docs.opnsense.org/manual/how-tos/multiwan.html. 
I setup the DNS monitoring on comcast of 75.75.75.75 and I put the HughesNet modem on 1.1.1.1.
When I unplug Comcast, I see the OPNsense interface switch over to HughesNet but the users don't get internet.  I'm new the OPNsense so I'm sure I'm missing something but I can't figure out what. I've gone over the OPN manual a dozen times.  I'm wondering if I didn't setup my WAN2 incorrectly but can't seem to find a clear manual on that.  I can ping over to HughesNet . When I go to the HughesNet modem IP (42.1) on an end-user PC I get the OPNsense login screen.  I can also ping it from OPN.
What on earth am I missing?

[the-mk]

what did you use for priority on both gateways? both the same or did you try using a lower one on your primary gateway?
what does system > route > status say who is the default gateway?

[JPERRY]

Comcast is 254 and HughesNet is 255.

[the-mk]

how many interfaces/networks do you have on LAN side?
dns and gateway/WAN-group firewall rules are applied?
rebooting the firewall does not change the situation?

[JPERRY]

It's pretty basic.  Just one network on a 5.1 network.  It's only 5 users and a file server.  The comcast circuit has been dropping service since moving to the office so we installed a HughesNet to catch those outages.
I just have two gateways, one for comcast and one for HughesNet.  HughesNet has one DNS of 1.1.1.1 (I monitor for failover) but the Comcast modem insists I use two DNS servers. I'm monitoring 75.75.75.75 but the modem also has 75.75.76.76.
I'll attach pics of the rules I'm using which are minimal. 

THANK YOU so much for helping me out!

[JPERRY]

Sorry... some of the rules were collapsed so I'll reattach and updated image.  Some aren't included but they're IP6 which we're not using.

[JPERRY]

Upon further examination I'm pretty sure I'm having a problem with DNS on WAN2 HoughsNet port.  When I connect my laptop to the LAN port of the firewall I can't ping 8.8.8.8 or anything else.  When in failover mode, under the Dashboard the WAN is red and says "Ethernet autoselect".

[the-mk]

i assume 192.168.5.1 is the IP of OPNsense on LAN side and WANComcastGroup is the name of your Gatewaygroup where Comcast is Tier1 and the other is Tier2?

when you say "Ethernet Autoselect" - you mean the Interfaces Widget on the dashboard? (all of my interfaces in my config say Ethernet Autoselect - getting the IPs on WAN-side with DHCP...)

I am not the expert on WAN-Failover configuration - just finished my setup a few days ago to cover my DOCSIS/cable outages of my provider with LTE as a failover... also based on the documentation... but since configuring the WAN-failover no outage happened on the DOCSIS/cable WAN side.

[JPERRY]

Ha, right... I'm confident once I get this working we'll stop having outages.  Yes, the 5.1 is the firewall and dhcp server for the LAN service 5.20 - 5.254 to the users.  Tier1 is comcast and Tier2 is Hughes within the WANComcastGroup.
I'm realizing my HughesNet isn't working because it can't get to a DNS server.  I'm trying to configure one on OPNsense router but seem to be tripping up on that.  The WAN1 autoconfigured with the Wizard so I think my problem is with the manual addition of WAN2.  Do you have any advice on that or directions?  I can't seem to use the wizard for WAN2.
((Last night I unplugged WAN1 to put it into failover... then connected my laptop directly to the LAN port and I couldn't ping and DNS servers like 8.8.8.8.  If I can fix that I think I'm good))
Thank you!

[the-mk]

can you attach directly to the Hughes-router and test if you can ping the monitored IP which you defined in your Hughes-Gateway-definition (I guess you tried that already)?
can you try to "monitor" an alternate IP instead of the DNS you can't reach?

I had both of my WANs configured automatically via DHCP (my DOCSIS-router and my LTE-router both gave me an IP via DHCP), so I only had to untag the "don't monitor that gateway" option and entered a monitor IP, adjust the priority of the gateways (primary a lower number, the LTE a higher number so the DOCSIS connection is prefered), combined both WANs in the gatewaygroup where the DOCSIS one is Tier1 and the LTE is Tier2, added those two firewall rules as mentioned in docu (one for local DNS, the other for the fail-over-gateway-group) and I think that was it... just followed the docs and adjusted the gateway priority which was not clearly mentioned there (but in some other chapter)

so when you look at the Gateways widget on the dashboard - what does it say about your Hughes connection?

[JPERRY]

Yes... I have a priority of 254 on the primary and 255 on the failover.  The gateways widget shows both as online in green when it's in normal function.  I'm going to zoom over tonight and put it back over in failover mode and see what it does. 
I noticed in my Services - DHCPv4, under each of the gateways I didn't have the DNS listed there. Maybe that's why the HughesNet couldn't ping anything in failover mode.
(Everything pings great when it's up because it's using the Comcast).  Should I check the box for "Allow DNS servers to be overridden by DHCP/PPP on WAN"?

How did you setup WAN2 using the System\Wizard? When I go in it looks like it just wants to configure the WAN1 only.  I don't want to follow through with it and it replace my WAN1, ha.

[the-mk]

I did not use the wizard, I started small and added the second WAN later on.

"allow DNS servers to be overridden by DHCP/PPP on WAN" - I checked this one (not sure if it is on by default?), since I want to use the DNS servers that are available depending on which WAN is used. but I also entered both google DNS IPs, the 8.8.8.8 for my primary WAN, the 8.8.4.4 for the secondary WAN. I am not sure, which one is used now (those from my ISP coming via DHCP or the google ones I entered there), when the "allow DNS servers to be overridden" switch is set?!?

I guess I need to install another OPNsense test vm on my ESX to perform some testing if failover is really working and which DNS is used with my settings - a task for during the holidays, but not this week...

[JPERRY]

Right!  Enjoy your holidays!
I do a lot from remote so I figured out I could remote into the primary ISP modem and do a reboot after hours... which takes about 4 minutes.  Just enough time to test the failover, ha!