OpnSense Carp Issue - Master status on both firewall

[athisesanr]

Hi Team,

I'm recently installed opnsense in VMware ESXI platform where deployment completed and try to make the CARP between the two machine gets formed but unfortunately I'm receiving master on both machine.

I did TS on Rules where placed correctly and HA SYNC working as expected and I could see the CARP protocal running (224.0.0.1 and 224.0.0.18) in interfaces.

I tried restart, carp disable, persistence mode those are not helping and thoroughly checked  that Virtual IP configuration where placed as it is.

I do have receiving IANS arp from connected top layer physical firewall
      100.100.102.250   7          00:00:5e:00:01:0a >>>> VIP
      100.100.102.252   0          00:50:56:90:01:6a >>>> FW1
      100.100.102.253   1          00:50:56:90:a5:82 >>>> FW2

I did upgrade the system and still issue remain the same.

CARP issue faced version of OPNsense 24.1 and 24.1.5_3-amd64

General Log file that we receiving both machines,

2024-04-17T23:46:57   Notice   opnsense   /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface (100.100.102.250).   
2024-04-17T23:46:57   Notice   opnsense   /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member " (100.100.102.250) (10@vmx0)" has resumed the state "MASTER" for vhid 10   
2024-04-17T23:46:53   Notice   opnsense   /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface (100.100.102.250).   
2024-04-17T23:46:53   Notice   opnsense   /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member " (100.100.102.250) (10@vmx0)" has resumed the state "BACKUP" for vhid 10   
2024-04-17T17:47:42   Notice   opnsense   /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface (100.100.102.250).   
2024-04-17T17:47:42   Notice   opnsense   /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member " (100.100.102.250) (10@vmx0)" has resumed the state "MASTER" for vhid 10   
2024-04-17T17:47:38   Notice   opnsense   /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface (100.100.102.250).   
2024-04-17T17:47:38   Notice   opnsense   /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member " (100.100.102.250) (10@vmx0)" has resumed the state "BACKUP" for vhid 10

So, can anyone help me out to how to resolve this issue ?

Thanks,
Athisesan R

 

[Monviech (Cedrik)]

Since CARP sends broadcasts with the VRRP protocol with forged MAC addresses, you need to allow that in your ESXI vswitch.

[athisesanr]

Hi Monviech,

I'm using dvs with portgroup, so is it required to enable forged transmits from dvs port-group security ?

Thanks,
Athisesan R

[Monviech (Cedrik)]

Sorry I'm not sure on that one. I just know that there cant be mac security enabled, and maybe even promiscious mode is needed. But a little unsure.

[mimugmail]

Hi Monviech,

I'm using dvs with portgroup, so is it required to enable forged transmits from dvs port-group security ?

Thanks,
Athisesan R

Should work, yes, there is a guide in pfsense docs (which is not related to *sense)

[athisesanr]

Hi

I did promiscuous mode enable on connected dvs port group and observing status as both firewall is "Backup" now.

I couldn't get it where the pfsense or vmware vrrp docs where works for here.

https://communities.vmware.com/t5/vSphere-vNetwork-Discussions/Can-t-ping-virtual-router-IP-in-VRRP/td-p/854331
https://www.reddit.com/r/vmware/comments/hh63yd/dvswitch_not_passing_multicast/

Thanks,
Athisesan

[athisesanr]

Hi Team,

Finally, I fixed CARP issue on vmware esxi dvs level with using  vmware mac learning option.

follow the steps.
 - Form the VIP between the Opnsense FWs
 - Edit the DVs port group security from vcenter
 - setting changes likes

       Promiscuous mode - Reject
       MAC address changes - Reject
       Forged transmits - Reject

       MAC Learning
         Status - Enabled
         Allow unicast floodin - Enabled
         MAC limit - 4096
         MAC limit policy - Allow

https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-networking/GUID-E0246B3D-9FB1-4976-8217-5C085863EA9A.html

Thanks,
Athisesan R