Gateway Group not recovering from ISP outage

[9axqe]

I have three gateways, each a wireguard tunnel (IPv4 only for now). Everything goes out the same WAN interface, same ISP. I have 3 wireguard tunnel because the remote side is unreliable.
Each of the three wireguard instances has "Disable routes" checked.

A different monitoring IP is configure on each gateway. If packet loss is happening, that gateway is removed from the group.

If a gateway goes down, it will recover at some point and will be re-added to the group.

My problem is if all three gateway are removed from the group (due to ISP outage for example), they never recover. I have to manually disable a gateway, then the two remaining immediately recover, and I can re-add the third one.

Any reason comes to mind for this? And any elegant fix I could implement?