With your assistance in previous topics, I got HA in working condition, but...
To describe my setup:
- 2x Opnsense instances in high availability mode with carp vip interfaces on single pve host. I know it's not full HA but I want software HA and also simply to test it.
- VMs are connected through 3 bridges: 1 on WAN side, the other on LAN side (and further trunk physical link to switch) and pfsync bridge.
- IGMP snooping, storm control are disabled in (UniFi) switches.
In order to change above configuration and (trying to) test my issue, I created additional LAN bridge for backup instance and instead of having them (2x opnsense) connected over single linux bridge - within proxmox, I connected them over physical switch.
This of course requires second downlink:
- master/regular LAN bridge would remain connected as it is now
- backup/new LAN bridge is connected to switch via additional downlink
But problem I'm facing is duplicated communication/data flow to and from both VMs; both instances have same looking graphs in proxmox webgui - network flow and also cpu. Despite they don't change their master/backup status (no flapping at carp status) I have something similar to split brain situation, for example if I communicate with opnsense webgui or ssh on carp vip interface, reply comes either from one of those two and toggles every few seconds. If I ping them, reply is duplicated ("DUP!"). Communication to other hosts and WAN is ok. I have already set Mac filter to "no" in proxmox VM's firewall options (pve firewall is disabled). I tried ovs and Linux bridges with same results.
To me, it is something related to MAC and network switches; is it possible to set it up correctly?
One more thing to add is when I reach my LAN over VPN (either Wireguard or OpenVPN) I can't communicate with backup instance (its physical interface addresses) at all while FW rules allow them to send requests to any hosts...
So, for those who may experience similar issues:
- WG (only in HA CARP mode) requires additional outbound rule for LAN interfaces and source as WG net
- Very unexpected reason of duplicated data was my set up of CARP VIPs in unicast mode. After removing peers (back to multicast CARP), all of sudden there's no more duplicated data flow and pings plus no webgui/ssh flapping... I wonder why?