Enter Persistent CARP Maintenance Mode - advskew 254 causes problems

[Werner Fischer]

Hi all,

I have some feedback regarding CARP:

• a) pfsync: I think in https://docs.opnsense.org/manual/how-tos/carp.html#setup-ha-sync-xmlrpc-and-pfsync it should be stated that on the backup firewall the "Synchronize States" option should be set, too.
• b) Enter Persistent CARP Maintenance Mode: when clicking this on firwall 1 this sets currently net.inet.carp.demotion=240 and leaves advskew as long there is no reboot. After a reboot of firewall 1, net.inet.carp.demotion=0, but advskew is set to 254 - because of https://github.com/opnsense/core/blob/master/src/etc/inc/interfaces.inc#L1713. When following the steps described at https://docs.opnsense.org/manual/how-tos/carp.html#example-updating-a-carp-ha-cluster the advskew setting is still set to 254 on firewall 1 even after clicking "Leave Persistent CARP Maintenance Mode". When testing a WAN outage (unplug igb1) afterwards, only for the WAN IP, the other firewall gets MASTER, leaving LAN (igb0) as BACKUP there. So the Internet connectivity gets lost for clients. Only rebooting the firewall 1 or manually setting advskew back to 0 solves the issue. I'm not sure what would be the best way to fix this behavior. Any ideas?

Steps to reproduce issue b):

• Build an OPNsense HA cluster with two nodes, firewall 1 as MASTER and firewall 2 as BACKUP
• Click "Enter Persistent CARP Maintenance Mode" on firewall 1. The sysctl "net.inet.carp.demotion" will be set to 240. advskew is still 0 for all configured CARP interfaces.
• Do a reboot of firewall 1.
• After the reboot, on firewall 1 "net.inet.carp.demotion" is now 0 (not 240), but advskew for all CARP interfaces is set to 254 (query by "ifconfig | grep carp"). So advskew is set to 254, but the web interface shows still values of 0 in "Firewall -> Virtual IPs -> Settings".
• Clicking "Leave Persistent CARP Maintenance Mode" on firewall 1 does _not_ switch back the CARP IPs to firewall 1. firewall 2 is still MASTER, although I would expect that now there should be a switch-back to firewall 1 - according to the doc https://docs.opnsense.org/manual/how-tos/carp.html#example-updating-a-carp-ha-cluster
• Only after another reboot of firewall 1, advskew is again set to 0. But in my opinion this additional reboot of firewall 1 is unecessary when updating an OPNsense firewall cluster.

Best regards,
Werner

[mimugmail]

You can track it here: https://github.com/opnsense/core/issues/3671