OPNsense running in HA mode in Azure

[fsebera]

Is there anyone running OPNsense setup in High Availability (HA) mode within Azure.

I run an evaluation test lab with OPNsense setup in HA mode in Oracle VirtualBox 6 and find that without the use of promiscuous mode, Proxy ARP or gratuitous arp-ing, OPNsense does not work effectively as an HA failover pair.

The 2 OPNsense HA boxes do forward end-user data traffic but ingress and egress traffic flows are not manageable. The only way I can get this to work in HA mode is to use IP Aliases but as I mentioned, no way to control which box is forwarding the actual end-user data traffic. As soon as you refresh the ARP tables, the end-user data traffic may or may not follow the same path. When one of the HA pair fails, if you don't refresh the ARP tables on the adjacent routers, end-user data traffic stops flowing until the ARP timer expires on that path (think hours).  If you set the adjacent routers ARP timeout to a low number, every time ARP is refreshed, end-user data traffic flip-flops back and forth between the two HA pairs. Sometimes egress traffic uses the Master LAN interface while the return ingress traffic uses the WAN interface of Backup and this changes pretty much evert time ARP refreshes.

If you run OPNsense in HA mode within Azure, would you share? PLEASE!

..... And if I have something configured incorrectly, PLEASE point out my mistake!!!!!!!!

PIC of flip-flop data flows attached jpg file.

Thank you
Frank