Second CARP interface has connection drops

[Calypso]

Having the feeling that I'm overlooking something simple, but I've got the following situation:

* 2 OpnSense VM's on Proxmox (doesn't matter if they are on different nodes or on the same one)
* WAN interface has IPv6 address on each box; no IPv4 address configured, but I've tried with 'fake addresses' and even with a temporary address from the public /24
* Both boxes have the IPv4 gateway set - tried both with far gateway on and off
* I have 2 CARP interfaces on the WAN side with public addresses in the same /24. Configured exactly the same on different VHD

What happens is that the 2nd CARP address is giving me connection problems; not all the time, but at around 40 to 50% of the time. Does that with ICMP ping or TCP connection to a port. It's always the 2nd CARP IP that does this, so when I switch both addresses, it's still the address on the 2nd CARP. When I change the 2nd CARP to an IP Alias everything is fine.

But ofcourse I don't want that because I want to be able to move that address to the 2nd OPNSense VM. I've also tried to put CARP in maintenance mode, stopped the second OPNSense box, etc.

Anyone have any idea what's happening?

Added:
* Boxes are on OPNsense 22.7.1
* When I want the IP Alias (that works) to transfer to box #2 and I set the VHID from "none" to the number of the CARP interface, the associated addres of the IP Alias is unreachable.