I recently got a second ISP connection for a backup link. It is a metered connection so I only want to use it in the event of an outage for my main ISP and only for a few of my VLANs. I was following the OPNsense Multi WAN Documentation (https://docs.opnsense.org/manual/how-tos/multiwan.html#step-2-add-gateway-group)
- I created a new interface for new new ISP. Primary is WAN_FIBER, secondary is WAN_CABLE.
- I added the new interface as a gateway.
- I enabled gateway switching. (unbound is my DNS)
- I enabled monitoring for both my WAN interfaces. (8.8.8.8 primary and 8.8.4.4 secondary)
- I added a DNS server (same as monitoring IPs) for each interface in the general settings.
- I created a new gateway group (WAN_FAILOVER). WAN_FIBER=Tier 1 and WAN_CABLE=Tier 2.
- I added a new firewall rule for DNS for each interface I want to fail over.
- I updated the inbound firewall rule for the interfaces I want to fail over.
The above seemed to work. When I disconnected the WAN_FIBER connection everything seemed to fail over to WAN_CABLE. The issue is everything failed over, not just the subnets I added the WAN_FAILOVER gateway too. The end goal is to only allow specific subnets to fail over. I have 8 VLANs and I only want to allow 2 of them to fail over (due to the metered connection).
What is the ideal way to achieve this?
EDIT:
I have also noticed that some things don't 'fail back' very well. My site-to-site WireGuard VPN didn't transition back to the Tier1 selection after it was restored.
EDIT2:
The more I mess around with this the more it feels like it is VERY complicated to allow 2 networks to fail over and 6 to not all while allowing LAN access for all 8 networks. I have to add several firewall rules to the 2 networks just to allow access back to the LAN because the way OPNsense handles the gateways.
I was hoping the failing over, would be happening at a higher level and just changing the systems default route, but it looks like it happens on a the interface level.