OPNsense Forum

Archive => 22.1 Legacy Series => Topic started by: Andreas_ on February 22, 2022, 04:33:59 pm

Title: CARP MASTER during reboot despite maintenance mode
Post by: Andreas_ on February 22, 2022, 04:33:59 pm
When doing regular maintenance on our CARP cluster, I regularly disable CARP on the machine and enter persistent maintenance mode. I'd expect it to never get MASTER until I enable CARP again.

Now, I rebooted the machine (22.1.1), and while it came up I glanced "Timeout on ix2, becoming MASTER" on the console for a second or so until it stepped back to BACKUP.
While I also have layered interfaces (vlan over lagg over 10GBit), this very ix2 interface is just a plain 1GBit onboard Intel NIC, connected to a switch, no VLAN no whistles or bells (upstream internet).

Having double master even for fractions of a second will screw up network traffic more or less badly, so this really isn't good and shouldn't happen, maintenance mode or not.

So how to safely reboot a router without triggering major trouble?
Title: Re: CARP MASTER during reboot despite maintenance mode
Post by: mimugmail on February 22, 2022, 05:49:19 pm
Is there spanning-tree on the switch disabled? Otherwise it would not receive packets too early and go to master mode.
Title: Re: CARP MASTER during reboot despite maintenance mode
Post by: Andreas_ on February 22, 2022, 05:56:28 pm
I think I have seen STP packets, I think I can have them disabled.
But what has spanning tree to do with carp? I'd expect only proto-112 packets to have any impact (and least of all in maintenance mode).
Title: Re: CARP MASTER during reboot despite maintenance mode
Post by: mimugmail on February 23, 2022, 07:09:40 am
Unit1 boots, link goes up but port is due to STP freezed. Firewall doesnt receive carp packets of Unit2 and promotes itself as master
Title: Re: CARP MASTER during reboot despite maintenance mode
Post by: Andreas_ on February 23, 2022, 11:14:13 am
Ok did a while to understand, because I was looking for the problem in the router...

To summarize, if STP is configured on the switch, it will not forward traffic for a while directly after the port is physically up; in consequence the freshly rebooted firewall won't receive CARP packets from the master and assume it's dead.

However, this doesn't explain the initial question: why does the firewall do carp at all? I want CARP disabled in the first place to prevent such very glitches.
Title: Re: CARP MASTER during reboot despite maintenance mode
Post by: mimugmail on February 23, 2022, 04:23:23 pm
maintenance mode just adds a demotion factor of 240, in this case you might try to disable carp completely.
Title: Re: CARP MASTER during reboot despite maintenance mode
Post by: Andreas_ on February 24, 2022, 12:48:50 pm
in this case you might try to disable carp completely.

This is what I'd expect from "maintenance mode". I'm not aware of any other means in opnsense to disable carp. Digging FreeBSD docs, I found sysctl net.inet.carp.allow. So am I supposed to use that tuning or am I missing something? (some "enable/disable CARP" opnSense setting)?
Title: Re: CARP MASTER during reboot despite maintenance mode
Post by: mimugmail on February 24, 2022, 04:04:00 pm
Please go to Interfaces : Virtual IPs : Status and check the 2 buttons :D
Title: Re: CARP MASTER during reboot despite maintenance mode
Post by: Andreas_ on February 24, 2022, 04:10:49 pm
Well these are the very two buttons I always check.
Temporarily disable CARP won't survive the reboot (which is just what I need), and persistent CARP maintenance state doesn't prevent CARP becoming MASTER unconditionally either, which is the very reason for this post.

Title: Re: CARP MASTER during reboot despite maintenance mode
Post by: mimugmail on February 24, 2022, 04:37:29 pm
Ah .. indeed. What this button does is:
net.inet.carp.allow: 0

You could also set this in the tunables in order to survive a reboot. But after maintenance you need to set it to 1 again.