Issues with muti-WAN monitoring

[sesquipedality]

I have two OpnSense routers set up, router1 is connected to Virgin, router 2 to Vodafone. Router1 and router2 are in a CARP group, and router2 takes over from router1 as DNS and DHCP server in the event it goes down.   In addition to LAN links, there is a direct back to back link between the two machines.  The setup for router1 is thus

Code: [Select]

LAN: 192.168.37.5/24
Backbone: 192.168.66.5/24 - point to point with 192.168.66.4
WAN: <wan address>
CARP: 192.168.37.1/24
And router 2 is on .4 with a mirror configuration

I have manually configured a gateway, BACKBONE_STATIC with address 192.168.66.4, it is set as an upstream gateway, but not as a far gateway, and in practice, neither of those settings seems to make a difference.  Priority is the default 255.

The external interface and the backbone link are in a WAN group together, with the external uplink as Tier 1 and the backbone as Tier 2.  The backbone interface has a default firewall rule to allow all traffic, and also, there is a custom outbound NAT rule saying not to NAT the backbone ahead of the auto-generated rules.

This configuration works, and as expected there is failover to the backbone link if the Virgin connection goes down.  The problem is that monitoring seems extremely flakey.   If the WAN on router2 is up when I configure the backbone link, it comes up and continues to show as online even if I take the WAN on router2 down.  If the WAN on router2 is down when I configure monitoring, the backbone link shows as offline and does not come back.

I thought this might be an asymmetric routing issue i.e. that router2 does not send the returning packets back via 192.168.66.5, but enabling the "bind states to interface" firewall option on both routers does not change this behaviour.  Adding a static route to 0.0.0.0/0 via 192.168.66.4 similarly makes no difference.   How do I get this setup to work properly?

As a side note, using Cloudflare DNS (i.e. 1.1.1.1) as a remote target for dpinger does not seem to work.  The OPNsense examples all use Google's Public DNS as a suggested endpoint, but this is only helpful if you only have two WAN links.   Are there any other good IPs to use with gateway monitoring?