CARP - Preventing individual VLAN Failovers

[browne]

Hey everyone,

I'm currently trying to set up my first CARP HA scenario.
I've installed and licensed the latest OPNsense Business Edition on both nodes.

Both OPNsense instances have two 10G ports configured as a LAG, with all networks configured as VLANs on that interface.
Additionally, each instance has two 1G ports. One is dedicated to pfsync, while the other is used for management access.
All VIPs on the master node have their skew set to 0, while those on the backup node are set to 100. The base is set to 1.
Preemption is enabled on both nodes (the checkbox to disable it is unchecked). I haven't configured anything regarding the demotion level so far.

As far as I understand it, the demotion level can trigger a complete failover if one of the interfaces goes down (assuming it's configured correctly).
However, since all CARP interfaces are VLANs, I assume the demotion mechanism would never be triggered because the VLAN interfaces can't go down individually.
This would mean that CARP failovers could still occur independently for each interface if a particular VLAN experiences connectivity issues or the switch gets misconfigured.

What I'm looking for is a way to prevent the backup node from taking over any CARP VIPs unless the primary node goes offline entirely.
I've found some solutions involving custom scripts that need to be installed via SSH directly in the filesystem. However, I'm a bit concerned that these scripts might break after updates or interfere with the update process itself.

That's why I'd prefer a solution that can be configured entirely through the GUI.
I also came across Monit, but I haven't looked into it in detail yet.

It might sound a bit strange that I'm looking for a solution to this particular scenario, considering that all networks are connected through a single 10G LAG and should theoretically fail together.
However, there are two main reasons for this.

1. Switch firmware bugs
Occasionally, the switches encounter a firmware bug that causes them to stop forwarding traffic for a single VLAN on one switch unit or overload just one particular VLAN. Sometimes even other Devices overload a VLAN which causes an interruption that the switch then can keep isolated in that VLAN.

2. Switch configuration errors
Sometimes, someone makes changes to the switch configuration and somehow manages to remove a VLAN from one of the OPNsense ports.

In both cases, I'd rather have an outage affecting a single network than have the backup node silently take over all CARP IPs, potentially hiding the underlying issue, or taking over only that particular VLAN, sending me in the wrong direction during troubleshooting.

Essentially, I want to avoid individual CARP VIP failovers and ensure that the backup node only takes over when the primary node fails completely.
Does anyone have an idea how to achieve this, preferably using only the OPNsense GUI?

[Monviech (Cedrik)]

An individual VLAN cannot fail over in the default HA configuration.

If the CARP multicasts on a participating interface stop (one that has a CARP VIP configured), the other participant in that Multicast group won't receive them anymore and will assume master state for all VIPs.

So, the failover is always complete if you didn't change settings like "Disable preempt" in the HA settings.

What can also happen is Split Brain if you kill a VLAN transparently on the switch without the OPNsense noticing (e.g. on an interconnect between switches). Then it can happen that both become master for that VLAN.

More failure states or gotchas are described here: https://docs.opnsense.org/manual/how-tos/carp.html#troubleshooting

[browne]

Split Brain is exactly what i am experiencing and what i am trying to avoid in any way.
The Interface itself will stay up, but a single VLAN will fail (overload, switch firmware bug, misconfiguration)

Could i check for a heartbeat from backup to master node, maybe via a ping, on a separate network and only initiate a failover when the heartbeat dies.
Alternatively, could i somehow notify the master that the backup is taking over everything.

Now that i am writing it, notifying the master sounds kinda impossible without scripts and overly complicated.

So is there a way to handle a Split Brain Situation or would this be a limitation of HA?

[Monviech (Cedrik)]

A transient Layer 2 failure is a limitation of HA.

How would you know even with a heartbeat what should happen?

Essentially there is already a heartbeat, its pfsync which most of the time is a direct link configuration. If that link goes down there is also a global demotion happening, for unrelated reasons (firewall state sync not working anymore)... but its also very naive.

But that also doesnt protect from transient failures, no mechanism can know about it. Pings through every vlan are just a worse alternative to the multicasts that already happen through every vlan anyway.

Spanning tree should probably catch vlan paths not working anymore, like Multiple Spanning Tree Protocol. But I don't know... not an expert in that field.

CARP is essentially a Layer3 failover, less so a Layer2 and below failover.

EDIT:
By the way I once looked into a heartbeat where each firewall would send the carp status of individual interfaces to the other firewall so split brain could be detected. But it ended at detection, there isn't awefully much you could do in terms of action. You could set the CARP VIP into INIT mode, but everything you do to improve that situation somehow just makes it worse cause there are too many variables.

[browne]

Alright, thank you very much for all the effort you put into it and even looking into it in the first place.

i was thinking of a ping using only the pfsync interface, but i havent thought about what would happen if only the pfsync interface would fail.

This is solved for me now. Again thank you very much.

[Patrick M. Hausen]

So is there a way to handle a Split Brain Situation or would this be a limitation of HA?

There is of course the concept of STONITH - one of my favourite IT acronyms: shoot the other node in the head.

So whichever node first decides that a failover happened and it has become the new primary actively disables (power off, shutdown switch port, ...) the other node to prevent split brain.

Of course this needs operator attention, a diagnose, and a manual restore of the HA state, afterwards. So it is not very common. I cannot think of a mechanism in OPNsense that could do this easily with the tools present today.

"Serious" HA/cluster solutions like e.g. Galera for MariaDB/MySQL or Elasticsearch require at least three nodes to have a reliable quorum without the risk of split brain.

I also know of no implementations for networking that would go that route. HA routing for example is quite easily achieved by some IGP (e.g. OSPF), some EGP (e.g. BGP4) and VRRP/HSRP/CARP for the client facing interfaces. If the links between the two nodes are dedicated ports with simple cables/fibres and are redundant (two of them) to begin with, a split brain situation is highly unlikely.

For example in case you have occasional switch outages you migh want to consider connectiong each OPNsense in your pair via a pair of LCAP interfaces to a pair of two separate switches capable of multi chassis LCAP (MLAG). Then configure every single security zone as a VLAN over that redundant link.

This would give you layer 2 failover - but it's another level of complexity and there are interoperability issues with fast (1s) LACP timeout across vendors, so the most reliable configuration is slow (30s) ones. I don't know if 30s failover time in the worst case is quick enough for you. Then again, STP is 30s in its default configuration, too, so ...

HTH,
Patrick