carp: 1@em1: BACKUP -> MASTER (master timed out)

[dusatvoj]

Hello,
I have a problem with my opnsense setup.

I have 2 firewalls in vmware virtualization and I want to have them in HA (CARP + XMLRPC sync).

My setup:

FW1:
 - WAN IF - some public IPs
 - LAN IF - 10.31.0.0/24
 - PFSYNC IF (in same PVLAN as LAN IF but different network) - 10.31.2.0/24

FW1:
 - WAN IF - some public IPs
 - LAN IF - 10.31.0.0/24
 - PFSYNC IF (in same PVLAN as LAN IF but different network) - 10.31.2.0/24

I have firewall setup on this interface like:
    PASS   IPv4 *    10.31.2.0/24    *    *    *    *    *       
   PASS   IPv4 CARP    *    *    *    *    *    *    

XMLRPC sync works, states sync looks working too (almost same number of states in dashboard even if one firewall has no traffic, there's around 2k states like at master)

but here's problem with CARP IPs - both firewalls switches to master and I can't communicate through CARP IPs and the only thing I have in log is "carp: 1@em1: BACKUP -> MASTER (master timed out)" after disable -> enable CARP in Virtual IP section.

Any suggestions?
Many thanks for any help

[jaredj]

You need your CARP-redounding interfaces to be in a port group that has promiscuous mode, MAC address changes, and forged transmits enabled. I've seen this best documented by the other sense, at https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability-virtual.html?highlight=vmware#hypervisor-users-especially-vmware-esx-esxi; but there are also blog articles you can find if you search for "carp vmware" or so.

If you have multiple physical uplinks for the vswitches in your VMware servers, see the above and also https://kb.vmware.com/s/article/59235, on the /Net/ReversePathFwdCheckPromisc setting.