OPNsense HA issue failing over all interfaces to Backup Firewall

[penley]

Issue: Not all interfaces are failing over to the Backup Firewall. When any interface fails on the Master Firewall, the only interface that switches over to the Backup is the interface that fails. All others stay up on the Master.
However, if the Master Firewall goes completely down then all interfaces fail over to the Backup.

I've tested this by unplugging the WAN cable and saw that it failed over to the Backup, but all other interfaces stay up on the Master.I plugged the WAN back in, it failed back to the Master firewall.
I unplugged the LAN cable and it failed over to the Backup, but all other interfaces remained up on the Master.


Setup: I have an HA setup using two OPNsense virtual machines on 20.7.2. The baremetal OS is Ubuntu 20.04.1.
Both baremetals have 4 ports with a bridge configured on all four ports.
The interfaces for both OPNsense VMs are the same:
1. WAN        vtnet0  VHID1
2. LAN          vtnet1  VHID2
3. pfsync      vtnet2 
4. DMZ         vtnet3  VHID3

The WAN ports are connected to a dumb switch.
The pfsync ports are connected directly.
The LAN and DMZ ports are connected to a managed switch ( The managed switch has no routing capabilities, only configured VLANs).

I have "Disable Preempt* unchecked for both the Master and Backup firewall.

I followed the directions for setting up the high availability using:
- https://www.thomas-krenn.com/en/wiki/OPNsense_HA_Cluster_configuration
- https://docs.opnsense.org/manual/how-tos/carp.html

After reading through the forums (reddit, opnsense, netgate, etc.); I know the HA setup is suppose to work that if one connection fails on the Master then all interfaces fail over to the Backup. However, in my own setup that is not the case. I've looked over the configuration several times to see if I've made a mistake, but nothing pops out. I followed the steps in those links above.

I'll keep researching and see what I can tell in the logs, but I thought I'd post here and ask, has anyone else had this issue?

Kind regards,
penley


EDIT:
I've tested failing over from the Master to the Backup again. I pulled the plug on the WAN and watched the logs. The Master still considers itself the Master of the WAN connection, but when I look at the Backup firewall it now thinks it's the Master of the WAN.
The log showed nothing from the Master firewall when I pulled the WAN cable out. The Backup firewall log showed:
kernel: carp: 1@vtnet0: MASTER -> BACKUP (more frequent advertisement received)
kernel: vtnet0: deletion failed: 3