Multi WAN - works in fallback but oddness ensues when load balancing

[sesquipedality]

I have set up my Opnsense 20.1.7-AMD64 router (running on a Dell R220 in a vm under qemu) with multiple WAN connections.  I have an IP modem on vtnet2 connected to Virgin Media UK using DHCP and an VDSL2 modem on vtnet1 connected to Vodafone UK (because Vodafone unhelpfully don't let you run their supplied router in modem mode).   I used the guide at https://docs.opnsense.org/manual/how-tos/multiwan.html to set these two links up in fallover mode with the Voda connection being used when the Virgin one falls over (since the Virgin one is twice the speed).   

Both gateways are assigned to WAN_GROUP, and the default rule for LAN (on vtnet0) is configured to route via this as a gateway.  All the firewall NAT holes on the router point to the Virgin link - my dynamic DNS provider (Mythic Beasts) doesn't offer an easy way to assign multiple IPs to a single hostname - the previous A record will simply be overwritten, which is of course the behaviour one usually wants, so it makes sense to keep all the external services on a single WAN interface, although it would be nice if there were a way to fallover these too.

Anyway, if I set the Virgin link to Priority 1 in the gateway group and Vodafone to 2, everything works like a dream.  The local internet transparently flips back and forth if I disconnect Virgin, so I have redundancy and can carry on working in the event of an ISP outage so long as both Virgin and Vodafone aren't down.

Ideally though, I'd like to aggregate the links.   I previously had this working in a test configuration using Wingate with a direct connection of the Virgin link to my PC.   This would show download speeds of 150Mbit/s and upload of 20Mbit/s on speedtest.

If I set both routers to priority 1 on OPNSense, however, things just get weird.  Some sites become unreachable, or have massively long timeouts - Speedtest will only ever use one WAN connection at a time, regardless of whether or not "bind states to interface" is set in firewall rules. 

Link speeds at 110Mbit/s for Virgin, 50-55Mbit/s Vodafone.  Trigger level is set to "Member Down".    Weight is set to 2 for Virgin, 1 for Vodafone.

I'm genuinely not sure where to start in diagnosing the issues I am seeing, and some pointers as to what to start looking at would be much appreciated.

 

[A1Dox]

I've been having the same issues. LAN0 is PPPoE with Zen, LAN1 is Ethernet to a LAN port on a Vodafone Modem/Router. With basic default routing in place everything is great. When I switch to load balancing I experience the same oddness.

I think it's down to DNS as, when I have been able to spot anything, I see both my internal Pi-Hole and  Unbound on the Opnsense box time-out on random lookups and then immediately return a response queried for the same fqdn a second time. I have two Quad9 addresses set in system DNS, one on each gateway, and two Google DNS used as the gateway monitors. DHCP was handing out the Pi-Hole and that was using Unbound which in turn used Quad9. I also tried with just Unbound in DHCP but got the same results.

I tried all sorts of rules ahead of the LB Gateway Group rule, to try and force DNS and some specific hosts over the default route/gateway and that seemed to work but didn't cure the oddness. I also force the Xbox out over Zen while the default was via Vodafone and again, that worked, but I couldn't get the Xbox to show "Open-NAT" no matter what I tried and still saw the issues.

My previous firewall was a Meraki MX60, and I was running Opnsense on an old mini-ITX machine as a test (before buying a small fanless "appliance" device to run it on permanently). Other than the issues, I've been really impressed with the functionality of Opnsense, which far exceeds the capabilities of the Meraki.

Earlier today I had the Vodafone swapped with an EE G.Fast service, which came with a separate modem which allows me to try PPPoE on both uplinks. I'm back on the Meraki for now but, once the EE service has settled down I will try Opnsense again to see if dual PPPoE (with no rfc1918 addressing on a "WAN" port now) makes any difference.

[sesquipedality]

Thanks, this is useful data.   Iniitally I tried setting up two DNS servers for each uplink, but that didn't seem to work at all so I reduced to a single DNS server per uplink (i.e. two in total), which allowed things to work to the extent that they do right now.   DNS problems is certainly consistent with some of the symptoms I have observed, but it's difficult to know what else could be done.  If DNS is the problem, it's unlikely that the physical links are what's causing the difficulty here.  I'm still at somewhat of a loss as to where to start in tracking down or fixing the problem though.