CARP MASTER during reboot despite maintenance mode

[Andreas_]

When doing regular maintenance on our CARP cluster, I regularly disable CARP on the machine and enter persistent maintenance mode. I'd expect it to never get MASTER until I enable CARP again.

Now, I rebooted the machine (22.1.1), and while it came up I glanced "Timeout on ix2, becoming MASTER" on the console for a second or so until it stepped back to BACKUP.
While I also have layered interfaces (vlan over lagg over 10GBit), this very ix2 interface is just a plain 1GBit onboard Intel NIC, connected to a switch, no VLAN no whistles or bells (upstream internet).

Having double master even for fractions of a second will screw up network traffic more or less badly, so this really isn't good and shouldn't happen, maintenance mode or not.

So how to safely reboot a router without triggering major trouble?

[mimugmail]

Is there spanning-tree on the switch disabled? Otherwise it would not receive packets too early and go to master mode.

[Andreas_]

I think I have seen STP packets, I think I can have them disabled.
But what has spanning tree to do with carp? I'd expect only proto-112 packets to have any impact (and least of all in maintenance mode).

[mimugmail]

Unit1 boots, link goes up but port is due to STP freezed. Firewall doesnt receive carp packets of Unit2 and promotes itself as master

[Andreas_]

Ok did a while to understand, because I was looking for the problem in the router...

To summarize, if STP is configured on the switch, it will not forward traffic for a while directly after the port is physically up; in consequence the freshly rebooted firewall won't receive CARP packets from the master and assume it's dead.

However, this doesn't explain the initial question: why does the firewall do carp at all? I want CARP disabled in the first place to prevent such very glitches.

[mimugmail]

maintenance mode just adds a demotion factor of 240, in this case you might try to disable carp completely.

[Andreas_]

in this case you might try to disable carp completely.

This is what I'd expect from "maintenance mode". I'm not aware of any other means in opnsense to disable carp. Digging FreeBSD docs, I found sysctl net.inet.carp.allow. So am I supposed to use that tuning or am I missing something? (some "enable/disable CARP" opnSense setting)?

[mimugmail]

Please go to Interfaces : Virtual IPs : Status and check the 2 buttons

[Andreas_]

Well these are the very two buttons I always check.
Temporarily disable CARP won't survive the reboot (which is just what I need), and persistent CARP maintenance state doesn't prevent CARP becoming MASTER unconditionally either, which is the very reason for this post.

[mimugmail]

Ah .. indeed. What this button does is:
net.inet.carp.allow: 0

You could also set this in the tunables in order to survive a reboot. But after maintenance you need to set it to 1 again.