Setting up WAN failover

[beneix]

I lost our fibre connection this morning so I am trying to set up WAN failover. I have got far enough that I can ping ip addresses but not web addresses from my OPNSense and from LAN clients, but I have no other connectivity. That leads me to believe I have a problem with my firewall. I am still running legacy rules, and this is what I have added to enable the failover:

LAN: (in) Any source and port to any destination and port via gateway WAN_FAILOVER (my new group): pass
WAN_tethered_USB: only auto-generated rules
NAT outbound: only auto-generated rules, which includes newly created rules for my failover WAN identical to the ones for the fibre WAN.

The new WAN_FAILOVER gateway group has my fibre gateway as Tier 1 and my new USB modem as Tier 2. The modem interface is showing green and, as I said, I can ping the world from an SSH shell on the box. No other traffic seems to work (including the "Announcements" section on the dashboard).

The fact that I can ping out tells me that there is no problem with the interface, gateway or gateway group setup but that there must be a firewall issue. Any suggestions?

[viragomann]

I am still running legacy rules, and this is what I have added to enable the failover:

LAN: Any source and port to any destination and port via gateway WAN_FAILOVER (my new group): pass

This rule forces all LAN traffic to the active gateway. If this rule is on upper position, it also catches traffic, which is destined to OPNsense itself, e.g. DNS requests.

If the default route goes to this gateway group, there is no need to state it in the rule at all. So this rule would only be applied to outbound traffic.

Otherwise you need separate rules to allow access to local destinations, which you have put above of the policy routing rule.

[beneix]

I am still running legacy rules, and this is what I have added to enable the failover:

LAN: Any source and port to any destination and port via gateway WAN_FAILOVER (my new group): pass

This rule forces all LAN traffic to the active gateway. If this rule is on upper position, it also catches traffic, which is destined to OPNsense itself, e.g. DNS requests.

If the default route goes to this gateway group, there is no need to state it in the rule at all. So this rule would only be applied to outbound traffic.

Otherwise you need separate rules to allow access to local destinations, which you have put above of the policy routing rule.

That rule is at the botttom. I have just added a rule above it for DNS from the LAN to reach the router itself. Thats has actually fixed general traffic from my LAN to the failover WAN so now I have achieved my prinicpal objective. I also had to add rules similar to the DNS for two services running on the router at different ports. Thanks for the pointer.

I just have one last problem - the router itself can't reach the internet for system updates, plugins, announcements, etc. I find that strange since the new WAN has the same rules as the main one, and on the LAN there are no rules for standard traffic with the source (self). What do I need to add to give OPNSense access to the new WAN via the new gateway group?

[viragomann]

However, I suddenly cannot reach two services running on the OPNSense. These (obviously) have the same IP address, but different port numbers.

Which services? Which port, IP, protocol?

Need some details on what doesn't work actually and what your LAN rule are.

Why would traffic from the LAN to the router suddenly be blocked because the WAN gateway is different?

It is blocked, because a policy routing rule forces any matching packet to the stated gateway. Hence it cannot reach a local service.
It's a good advise to limit a policy routing rule, meant for upstream traffic, to public destinations (exclude local destinations).

[beneix]

I rebooted the router. Now everything is working (beats me).

Next step is to take the plunge on migrating my legacy firewall rules...I think I will let that wait until I have my fibre broadband back, the hassle around this failover WAN setup has caused enough pain 😊.

[viragomann]

I rebooted the router. Now everything is working (beats me).

I guess, there were existing states for these connections you were trying to access. So following packets were forced to the gateway as well.

You can simply flush all states in Firewall: Diagnostics: States: Action or search for the particular connection in the state table and drop it.