[SOLVED] 2 HA CARP masters ?!

[tryllz]

Hi,

I have 2 firewalls setup in HA, and both are showing as CARP Masters. The following is whats showing in System > Log Files > General for 192.168.10.7 (supposed to be the Master)

Code Select Expand

2021-03-07T18:20:33 kernel vmx0: promiscuous mode enabled
2021-03-07T18:20:33 kernel carp: demoted by -240 to 0 (pfsync bulk done)
2021-03-07T18:20:33 kernel carp: demoted by 240 to 240 (pfsync bulk start)
2021-03-07T18:20:37 opnsense[35648] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.34 - (3@vmx2)" has resumed the state "MASTER" for vhid 3
2021-03-07T18:20:37 kernel carp: 4@vmx3: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:37 opnsense[57785] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.2 - .
2021-03-07T18:20:37 kernel carp: 3@vmx2: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:37 opnsense[57785] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.2 - (2@vmx1)" has resumed the state "MASTER" for vhid 2
2021-03-07T18:20:37 opnsense[47670] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 192.168.10.6 - .
2021-03-07T18:20:37 kernel carp: 2@vmx1: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:37 opnsense[47670] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.10.6 - (1@vmx0)" has resumed the state "MASTER" for vhid 1
2021-03-07T18:20:36 kernel carp: 1@vmx0: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:34 opnsense[73079] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.194 - .
2021-03-07T18:20:34 opnsense[73079] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.194 - (8@vmx7)" has resumed the state "BACKUP" for vhid 8
2021-03-07T18:20:34 kernel carp: 8@vmx7: INIT -> BACKUP (initialization complete)
2021-03-07T18:20:34 kernel vmx7: promiscuous mode enabled
2021-03-07T18:20:34 opnsense[81403] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.162 - .
2021-03-07T18:20:34 opnsense[81403] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.162 - (7@vmx6)" has resumed the state "BACKUP" for vhid 7
2021-03-07T18:20:34 kernel carp: 7@vmx6: INIT -> BACKUP (initialization complete)
2021-03-07T18:20:34 kernel vmx6: promiscuous mode enabled
2021-03-07T18:20:34 opnsense[48516] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.130 - .
2021-03-07T18:20:34 opnsense[48516] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.130 - (6@vmx5)" has resumed the state "BACKUP" for vhid 6
2021-03-07T18:20:34 kernel carp: 6@vmx5: INIT -> BACKUP (initialization complete)
2021-03-07T18:02:29 opnsense[44335] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.162 - (7@vmx6)" has resumed the state "MASTER" for vhid 7
2021-03-07T18:02:28 opnsense[39921] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.130 - (6@vmx5)" has resumed the state "MASTER" for vhid 6
2021-03-07T18:02:28 opnsense[89999] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.98 - (5@vmx4)" has resumed the state "MASTER" for vhid 5
2021-03-07T18:02:28 kernel carp: 8@vmx7: BACKUP -> MASTER (master timed out)
2021-03-07T18:02:28 opnsense[63214] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.66 - (4@vmx3)" has resumed the state "MASTER" for vhid 4
2021-03-07T18:02:28 kernel carp: 7@vmx6: BACKUP -> MASTER (master timed out)
2021-03-07T18:02:28 opnsense[62132] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.34 - (3@vmx2)" has resumed the state "MASTER" for vhid 3
2021-03-07T18:02:28 kernel carp: 6@vmx5: BACKUP -> MASTER (master timed out)
2021-03-07T18:02:28 kernel carp: 5@vmx4: BACKUP -> MASTER (master timed out)
2021-03-07T18:02:28 kernel carp: 4@vmx3: BACKUP -> MASTER (master timed out)
2021-03-07T18:02:28 kernel
2021-03-07T18:02:28 opnsense[83400] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.2 - (2@vmx1)" has resumed the state "MASTER" for vhid 2
2021-03-07T18:02:28 kernel carp: 3@vmx2: BACKUP -> MASTER (master timed out)
2021-03-07T18:02:28 kernel
2021-03-07T18:02:28 kernel ..
2021-03-07T18:02:28 opnsense[89493] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.10.6 - (1@vmx0)" has resumed the state "MASTER" for vhid 1
2021-03-07T18:02:28 kernel carp: 2@vmx1: BACKUP -> MASTER (master timed out)
2021-03-07T18:02:28 opnsense[5648] /usr/local/etc/rc.bootup: The Workstation_Gateway monitor address is empty, skipping.

And this is whats in System > Log files > General for 192.168.10.8 (Supposed to be the slave)

Code Select Expand

2021-03-07T18:20:38 opnsense[2338] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.194 - (8@vmx7)" has resumed the state "MASTER" for vhid 8
2021-03-07T18:20:37 kernel carp: 8@vmx7: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:37 opnsense[86568] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.162 - .
2021-03-07T18:20:37 opnsense[86568] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.162 - (7@vmx6)" has resumed the state "MASTER" for vhid 7
2021-03-07T18:20:37 kernel carp: 7@vmx6: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:37 opnsense[13457] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.130 - .
2021-03-07T18:20:37 opnsense[13457] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.130 - (6@vmx5)" has resumed the state "MASTER" for vhid 6
2021-03-07T18:20:37 opnsense[91531] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.98 - .
2021-03-07T18:20:37 kernel carp: 6@vmx5: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:37 opnsense[91531] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.98 - (5@vmx4)" has resumed the state "MASTER" for vhid 5
2021-03-07T18:20:37 opnsense[33485] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.66 - .
2021-03-07T18:20:37 kernel carp: 5@vmx4: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:37 opnsense[33485] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.66 - (4@vmx3)" has resumed the state "MASTER" for vhid 4
2021-03-07T18:20:37 opnsense[35648] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.34 - .
2021-03-07T18:20:37 opnsense[35648] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.34 - (3@vmx2)" has resumed the state "MASTER" for vhid 3
2021-03-07T18:20:37 opnsense[35648] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.34 - (3@vmx2)" has resumed the state "MASTER" for vhid 3
2021-03-07T18:20:37 kernel carp: 4@vmx3: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:37 opnsense[57785] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.2 - .
2021-03-07T18:20:37 kernel carp: 3@vmx2: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:37 opnsense[57785] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.2 - (2@vmx1)" has resumed the state "MASTER" for vhid 2
2021-03-07T18:20:37 opnsense[47670] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 192.168.10.6 - .
2021-03-07T18:20:37 kernel carp: 2@vmx1: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:37 opnsense[47670] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.10.6 - (1@vmx0)" has resumed the state "MASTER" for vhid 1
2021-03-07T18:20:36 kernel carp: 1@vmx0: BACKUP -> MASTER (master timed out)
2021-03-07T18:20:34 opnsense[73079] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.194 - .
2021-03-07T18:20:34 opnsense[73079] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.194 - (8@vmx7)" has resumed the state "BACKUP" for vhid 8
2021-03-07T18:20:34 kernel carp: 8@vmx7: INIT -> BACKUP (initialization complete)
2021-03-07T18:20:34 kernel vmx7: promiscuous mode enabled
2021-03-07T18:20:34 opnsense[81403] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.162 - .
2021-03-07T18:20:34 opnsense[81403] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.162 - (7@vmx6)" has resumed the state "BACKUP" for vhid 7
2021-03-07T18:20:34 kernel carp: 7@vmx6: INIT -> BACKUP (initialization complete)
2021-03-07T18:20:34 kernel vmx6: promiscuous mode enabled
2021-03-07T18:20:34 opnsense[48516] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface 10.0.64.130 - .
2021-03-07T18:20:34 opnsense[48516] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.0.64.130 - (6@vmx5)" has resumed the state "BACKUP" for vhid 6
2021-03-07T18:20:34 kernel carp: 6@vmx5: INIT -> BACKUP (initialization complete)

Both of firewalls are running as VM on ESXi 7.0, I have also enabled Promiscous Mode in vSwitch, it did not help. I'm unsure why its timing out, I have restarted the firewalls, and have reloaded all services on 192.168.10.7 followed by 8, that did not help either.

I have 2 other firewalls running as VM in VMware Workstation, both of them did not face any such issue.

Any thoughts.

[FingerlessGloves]

Did you enable Promiscous for the whole vswitch or just the one network?

EDIT: you may also need to enable "MAC Address changes" and "Forged transmits".
Since the carp MAC address moves between the two VMs

[tryllz]

I enabled Promiscous Mode on the whole switch.

I'll enable MAC address changes and Forged Transmits and re check.

[tryllz]

I can confirm, by enabling Promiscuous Mode, MAC Address Changed, and Forged Transmits on the vSwitch of ESXi the Firewall VMs are now showing as Master and Backup.

This issue is now resolved, thanks again FingerlessGloves