Messages

"Right...the different priorities are because I want one service to be preferred over the other. The MAIN circuit 200/10 while the BKUP circuit is 10/2 (i.e. only for emergencies). Thus, what I want to have happen is to have the MAIN circuit be used for internet access whenever it's available, and only fall back to BKUP if there's no other choice (i.e. some access is better than none)."

Yeah, I thought that was why you were doing it. But its not the right way to achieve it. For both load balancing and failover you should use Gateway Groups.

After setting up your gateways (System->Gateways->Single) you should then create a gateway group (System->Gateways->Group).

For a failover group set your primary (MAIN) to Tier 1 and the backup (BKUP) to Tier 2. In your case I would set the Trigger Level to 'Member Down' (Supposedly triggers with 100% packet loss).

You may wish to consider the other Trigger Level options (e.g. High Latency, Packet Loss or Both). Although I can't find any documentation to confirm it. I believe the Trigger Levels for High Latency or Packet Loss are the higher 'To' values you set on the Gateways->Single page (i.e. if you accepted the defaults then a Latency above 500 milliseconds OR a Packet Loss above 20%).

Once a gateway is marked as down, if there are no other gateways in the same tier it will failover to the next tier.

If you have multiple primaries or backups and want to load balance these in a failover scenario you would put the primaries on the same Tier (e.g. MAIN1 and MAIN2 on Tier 1 and BKUP1 and BKUP2 on Tier2). You have up to 5 tiers to play with so you could have a backup for your backup if you wanted (e.g. put BKUP2 on Tier 3).

Also, if you happen to want asymmetric load balancing on a tier you achieve that by setting the Weight value on the Single Gateway settings (The higher the weight value, the more traffic goes via that gateway)

There's a bit more to do after that as you need to set the Gateway in the Firewall->Rules->LAN "Default allow LAN to any rule" to the failover group gateway you created under System->Gateways->Group. Take a look at this link (I found it invaluable) for more information: https://www.thomas-krenn.com/de/wiki/OPNsense_Multi_WAN#Failover

I don't know if this will help. (Please Note: I don't use VPNs). But my failover setup has worked perfectly for at least a couple of years with all versions of OPNsense from 17 on up and I noticed a difference between your setup and mine. I notice you have set the Single gateway priorities to different values. In my setup I have them both set to exactly the same value (255). I set the priority in the Gateway Group settings differently with the primary set to Tier 1 and the backup set to Tier 2.

If it helps as a workaround I don't use DHCP to assign an IP address to my WAN interfaces I use static IP addresses.

I've had multi-wan (failover) working with OPNsense for over a couple of years now and have found it perfectly reliable with all the versions I've used from 17.7 to 20.1. I've used the information from this link to configure a working setup, suggest you try it or at least compare it with your setup to look for any differences:
https://www.thomas-krenn.com/de/wiki/OPNsense_Multi_WAN

Try the following: Goto Interfaces->Settings and set Hardware CRC, Hardware TSO and Hardware LRO all to Disabled (i.e. Checked) and set VLAN Hardware Filtering to Disable VLAN Hardware Filtering

The issue for me was apparently caused by the Intrusion Detection system.
Disabling Intrusion Detection meant that DHCP on the VLAN interface would work. Enabling it and it wouldn't work.

Not wanting to the leave the Intrusion Detection system disabled, I then discovered that disabling VLAN hardware filtering on the Interface settings also meant that DHCP on the VLAN interface would work even with the Intrusion Detection Enabled.

So the solution is: Interfaces->Settings-> disable Hardware checksum offload, disable hardware TCP S O,disable Hardware L R O and lastly but most importantly Disable VLAN Hardware Filtering.