[SOLVED] CARP won't become MASTER, both systems always BACKUP

[doug.dimick]

I'm replacing a working pfSense HA setup, so I'm reasonably sure the network is set up properly. I have two OPNsense 18.1.1 VMs running on ESXi 6.5. ESXi is configured to permit promiscuous mode, MAC address changes, and forged transmits. There are no other systems running CARP on my network.

Every three seconds the system log reports:

Code: [Select]

Feb 7 12:08:04 opnsense: /usr/local/etc/rc.carpbackup: Carp cluster member "192.168.0.5 - (1@em1)" has resumed the state "BACKUP" for vhid 1
Feb 7 12:08:04 configd.py: [8b269f1d-df1c-4809-871a-4f3ee75db2ba] Carp backup event
Feb 7 12:08:04 opnsense: /usr/local/etc/rc.carpmaster: Carp cluster member "192.168.0.5 - (1@em1)" has resumed the state "MASTER" for vhid 1
Feb 7 12:08:04 configd.py: [30b8e115-bc25-4339-9f7f-92176a1a471f] Carp master event
Feb 7 12:08:04 kernel: ifa_maintain_loopback_route: deletion failed for interface em1: 3
Feb 7 12:08:04 kernel: carp: 1@em1: MASTER -> BACKUP (more frequent advertisement received)
Feb 7 12:08:04 kernel: carp: 1@em1: BACKUP -> MASTER (master timed out)

Code: [Select]

em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0c:29:30:b0:1d
        hwaddr 00:0c:29:30:b0:1d
        inet 192.168.0.2 netmask 0xffff8000 broadcast 192.168.127.255
        inet 192.168.0.5 netmask 0xffff8000 broadcast 192.168.127.255 vhid 1
        inet6 fe80::20c:29ff:fe30:b01d%em1 prefixlen 64 scopeid 0x2
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        carp: BACKUP vhid 1 advbase 1 advskew 0
        groups: Inside
What am I missing?

Edit: There is no carp0 interface listed when I look at ifconfig output.

[doug.dimick]

I have fixed this issue. I set Net.ReversePathFwdCheckPromisc to 1 in ESXi and the issue is now resolved. I don't understand why I didn't see this problem with pfSense running on the same ESXi host, vSwitch, and port group, but hopefully this will help others in the future.

[franco]

Hi Doug,

Ah, thanks for the quick resolve.


Cheers,
Franco

PS: Welcome!

[phoenix]

I have fixed this issue. I set Net.ReversePathFwdCheckPromisc to 1 in ESXi and the issue is now resolved. I don't understand why I didn't see this problem with pfSense running on the same ESXi host, vSwitch, and port group, but hopefully this will help others in the future.

I assume that you've configured both OPNsense instances on ESXi? I'm thinking of trying this myself, apart from your current (fixed) problem is there any gotchas with this config?

[doug.dimick]

I assume that you've configured both OPNsense instances on ESXi? I'm thinking of trying this myself, apart from your current (fixed) problem is there any gotchas with this config?

Yes, both instances are running on the same ESXi 6.5.0U1 host. I have virtual redundancy, but not physical. I give each OPNsense instace 1 vCPU and 1GB RAM, which is more than enough to push 100mb over OpenVPN. If you want to use Intrusion Detection, add more RAM and maybe another vCPU.

The CARP oddity also caused some behavioral issues with DHCP failover, but fixing one also fixed the other.

There is a cosmetic issue where ESXi will complain that there is a mismatch between the guest OS (Other/FreeBSD 64-bit) configured on the VM, and the actual installed OS (FreeBSD 11.1-RELEASE-p6), but that's the case for any FreeBSD 11 guest system running open-vm-tools and, as far as I can tell, doesn't actually cause any problems.

I haven't seen any other issues yet, but I'm relatively new to OPNsense.

[phoenix]

That sounds encouraging, I'll have to give it a try in the coming weeks. I have two ESXi servers so one on each server will be my set-up in case of any H/W failures.

[b1t_r0t]

I just switched from a single pfSense machine to HA OPNsense on a pair of R610s and saw the exact same behavior with ESXi 6.5.

Another thing I found out, I like to present 1 vNic to the server, while using 2 physical uplinks on the esxi side. This way if fail over occurs, ESXi handles it via the built in NIC teaming and it's transparent to the VM. However the NIC teaming introduces another issue. As soon as I plugged in the 2nd sets of cables the interfaces went hard down and stayed in the Backup state.

Long story short if you are going to run this on ESXi do exactly what was said, set the vSwitches to allow promiscuous mode, MAC address changes, and forged transmits and then change Net.ReversePathFwdCheckPromisc to 1, reboot ESXi, and you should be money.

I have been incredibly happy with this setup since then

Helpful links:
http://www.blissfulidiot.com/2013/11/using-carp-with-vmware-esxi.html
https://kb.vmware.com/s/article/2144849 (<--- Confirms this is an expected behavior of ESXi)

[phoenix]

Thanks for that extra info.