CARP - Preventing individual VLAN Failovers

[browne]

Hey everyone,

I'm currently trying to set up my first CARP HA scenario.
I've installed and licensed the latest OPNsense Business Edition on both nodes.

Both OPNsense instances have two 10G ports configured as a LAG, with all networks configured as VLANs on that interface.
Additionally, each instance has two 1G ports. One is dedicated to pfsync, while the other is used for management access.
All VIPs on the master node have their skew set to 0, while those on the backup node are set to 100. The base is set to 1.
Preemption is enabled on both nodes (the checkbox to disable it is unchecked). I haven't configured anything regarding the demotion level so far.

As far as I understand it, the demotion level can trigger a complete failover if one of the interfaces goes down (assuming it's configured correctly).
However, since all CARP interfaces are VLANs, I assume the demotion mechanism would never be triggered because the VLAN interfaces can't go down individually.
This would mean that CARP failovers could still occur independently for each interface if a particular VLAN experiences connectivity issues or the switch gets misconfigured.

What I'm looking for is a way to prevent the backup node from taking over any CARP VIPs unless the primary node goes offline entirely.
I've found some solutions involving custom scripts that need to be installed via SSH directly in the filesystem. However, I'm a bit concerned that these scripts might break after updates or interfere with the update process itself.

That's why I'd prefer a solution that can be configured entirely through the GUI.
I also came across Monit, but I haven't looked into it in detail yet.

It might sound a bit strange that I'm looking for a solution to this particular scenario, considering that all networks are connected through a single 10G LAG and should theoretically fail together.
However, there are two main reasons for this.

1. Switch firmware bugs
Occasionally, the switches encounter a firmware bug that causes them to stop forwarding traffic for a single VLAN on one switch unit or overload just one particular VLAN. Sometimes even other Devices overload a VLAN which causes an interruption that the switch then can keep isolated in that VLAN.

2. Switch configuration errors
Sometimes, someone makes changes to the switch configuration and somehow manages to remove a VLAN from one of the OPNsense ports.

In both cases, I'd rather have an outage affecting a single network than have the backup node silently take over all CARP IPs, potentially hiding the underlying issue, or taking over only that particular VLAN, sending me in the wrong direction during troubleshooting.

Essentially, I want to avoid individual CARP VIP failovers and ensure that the backup node only takes over when the primary node fails completely.
Does anyone have an idea how to achieve this, preferably using only the OPNsense GUI?

[Monviech (Cedrik)]

An individual VLAN cannot fail over in the default HA configuration.

If the CARP multicasts on a participating interface stop (one that has a CARP VIP configured), the other participant in that Multicast group won't receive them anymore and will assume master state for all VIPs.

So, the failover is always complete if you didn't change settings like "Disable preempt" in the HA settings.

What can also happen is Split Brain if you kill a VLAN transparently on the switch without the OPNsense noticing (e.g. on an interconnect between switches). Then it can happen that both become master for that VLAN.

More failure states or gotchas are described here: https://docs.opnsense.org/manual/how-tos/carp.html#troubleshooting

[browne]

Split Brain is exactly what i am experiencing and what i am trying to avoid in any way.
The Interface itself will stay up, but a single VLAN will fail (overload, switch firmware bug, misconfiguration)

Could i check for a heartbeat from backup to master node, maybe via a ping, on a separate network and only initiate a failover when the heartbeat dies.
Alternatively, could i somehow notify the master that the backup is taking over everything.

Now that i am writing it, notifying the master sounds kinda impossible without scripts and overly complicated.

So is there a way to handle a Split Brain Situation or would this be a limitation of HA?

[Monviech (Cedrik)]

A transient Layer 2 failure is a limitation of HA.

How would you know even with a heartbeat what should happen?

Essentially there is already a heartbeat, its pfsync which most of the time is a direct link configuration. If that link goes down there is also a global demotion happening, for unrelated reasons (firewall state sync not working anymore)... but its also very naive.

But that also doesnt protect from transient failures, no mechanism can know about it. Pings through every vlan are just a worse alternative to the multicasts that already happen through every vlan anyway.

Spanning tree should probably catch vlan paths not working anymore, like Multiple Spanning Tree Protocol. But I don't know... not an expert in that field.

CARP is essentially a Layer3 failover, less so a Layer2 and below failover.

EDIT:
By the way I once looked into a heartbeat where each firewall would send the carp status of individual interfaces to the other firewall so split brain could be detected. But it ended at detection, there isn't awefully much you could do in terms of action. You could set the CARP VIP into INIT mode, but everything you do to improve that situation somehow just makes it worse cause there are too many variables.