HA Multiple Addresses on single iface: Multi-carp or carp+ipalias?

[Wolfspyre]

Howdy all!

I'm trying to identify the recommended path forward.

I have two opnsense hosts.
 - Each host has a single non-vlan-ified interface to ISP1 (igb1) and ISP2 (igb2)
 - Each host has a heartbeat nic (igb0)
 - Each host has dual 10G nics LAGGed (lagg0) and then vlan-scoped interfaces for internal traffic.
   - lagg0_vlanN1
   - lagg0_vlanN2
   - lagg0_vlanN3
   - ...
I have a few internal VIPs, and multiple public static addresses..

in each of these scenarios, I'm wondering what the cleanest / most elegant way to have one host be "active" and the other "inactive"

I've had some issues where some carp addresses stick to an active state on one node, when everything else is active on the other...  using fewer carp addresses (and leveraging IP Aliases on top of a single-carp-per-interface smells like it might be less prone to wonky behavior, as fewer moving parts.

however on the other hand, I've run into issues on interfaces which have a carp address, and IP Aliases on top, where I start seeing invalid vhid messages for $carp address spewed across the consoles of both hosts:

assuming I had a carp vip using VHID 99 on vlan 99:
192.168.99.1

and ipaliases 192.168.99.11, and 192.168.99.12... atop the .1 carp vip (using vhid 99) I'd regularly see error messages like this:

Code: [Select]

arprequest_internal: cannot find matching address
arprequest_internal: cannot find matching address
carp_input_c: incorrect hash for VHID 99@lagg0_vlan99
arprequest_internal: cannot find matching address
carp_input_c: incorrect hash for VHID 99@lagg0_vlan99
arprequest_internal: cannot find matching address
arprequest_internal: cannot find matching address

I've seen guidance in both directions.... (paraphrased below)
- 'use one carp address per HA vip, even with multiple vips in the same broadcast domain / subnet'
- 'use one carp address per broadcast domain/subnet, and use ip aliases on top of that carp for additional vips'


This blurb in the docs:
https://wiki.opnsense.org/manual/how-tos/carp.html#adding-multiple-carp-ips

Since adding a VHID for every IP would make the CARP traffic very noisy, you can also add a new IP Alias and choose the correct VHID where the first CARP IP is configured

seems to steer the user towards one carp per interface/broadcast domain/subnet  and using ip aliases on top of that..

but I'd like to clarify this if possible.

I've seen several questions about this on the forums:
  - https://forum.opnsense.org/index.php?topic=15938.msg74048#msg74048
  - https://forum.opnsense.org/index.php?topic=5765.msg23663#msg23663

these two pop up, but I've seen others (just cant find the links to them now)


however I've not seen a clear "this is what we have found to work the most reliably, and is the community's recommended implementation"

which ... consensus... I get it...
but I'd like to TRY to get a bit more clarity if possible?