CARP : Interfaces not in sync when failing over (failing over independently)

[mestafin]

Hi,

I have two OPNsense firewalls running on 2 x dedicated hardware units in a new HA CARP cluster that is working fine, except for one problem - unsynchronised failover of the interfaces.

I have two WAN interfaces and one LAN interface, but with a number of virtual vlan interfaces defined on the LAN interface.

Each WAN interface has a CARP VIP, with the other public ip's of the WAN sub-net defined as an IP Alias on top of the CARP VIP of the WAN sub-net (same VHID number).

On the LAN side, each vlan has a CARP VIP and two device ip's.
For the 10.1.1.0/24 subnet and vlan, I have defined:

   OPNsense1  10.1.1.1.91, OPNsense2 10.1.1.92 and CARP VIP 10.1.1.1

The same for the other vlans subnets.

Each WAN interface and each vlan virtual interface has a unique VHID.

The problem is that during a failover, the WAN and virtual interfaces do not fail over at the same time. If I reboot the active unit, I may find that one unit has the WAN interfaces as active with the LAN interfaces as backup and the other unit the reverse. It looks like there is a timing difference between the WAN and LAN interfaces when a decision is made to failover or not. Sometimes even the WAN interfaces are split  or the LAN interfaces are split between the two units.

From my understanding of CARP, each VHID is handled individually and will failover independently of the other VHID's.

This will also be a problem if one interface fails. It will not help if ony that interface fails over to the other fw.

Obviously, this is not going to work.

The main reason for the HA cluster, is failure of the OPNsense hardware units, as I already have dual uplinks to the ISP and dual switches.

How do I ensure that the units fail all interfaces over at the same time?

[mestafin]

Hi,

I have done some more research with Google, from OpenBSD.....

It looks like you have to create "CARP Groups" and that will force all the CARP interfaces to fail over as a group when one of the CARP Group interfaces fail.

In FreeBSD 11 and OPNsense, I can find no mention of CARP Groups, so how do I force all CARP interfaces to failover together.

Also, in another thread on this forum, it was advised to add a tuneable

  net.inet.carp.senderr_demotion_factor=0

Is this still advised?

[franco]

Hi mestafin,

If it fixes your issue yes, otherwise no.

I'm not an expert. I only know that most installs do not need this anymore so it was removed.


Cheers,
Franco