High Availability (on VirtualBox)

[being]

Hello!

First of all my humble thank you to everyone who has put in their time for this project. It's amazing! <3

I've been fiddling around with OPNsense for the last couple of days. My experience so far has been with enterprise firewalls (Cisco & Fortigate mostly), but I'm considering using OPNsense for a side-project.

My main worry is seamless HA, so I labbed it up on VirtualBox with 2x OPNsense and a client host.
When I finally got everything working (had to enable Promiscuous mode in VirtualBox for CARP), I first tested what happens when I shut down or reboot the Master of the cluster.

I found that the sessions will not be transferred and I will lose quite a bunch of pings into a black hole (just guessing, but probably because the master is not telling the backup, that it's no longer routing the packets, until CARP detects it when the master has finally shut down).

Secondly I tested what happens when I just disconnect the ethernet cables from the master, while there's a download going and ping running. I did not lose any pings (nice!), but the session was still not transferred to the backup OPNsense.

I documented these tests into a video, which can be found here
It's a too long video, but I only discovered in the 2nd part of the video, that I could see what was happening with the sessions with the pfTop feature. Anyways you can see everything in the first minutes.

Do you know if the sessions not being transferred to the other member of the cluster is working as intended or is there something wrong with my setup or it's a bug?

[mimugmail]

Screenshot of your HA settings please :)
Normally this should work.

[being]

Screenshot of your HA settings please :)
Normally this should work.

HA is working correctly when I disconnect the cables, but the sessions are not transferred.
Settings on the master. All settings possible are synced with the backup.

[mimugmail]

Firewall in HA is everything allowed? Do you see something blocked?

[being]

Firewall in HA is everything allowed? Do you see something blocked?

Yes. Anything and everything allowed on HA.

[mimugmail]

Firewall : Settings : Advanced, can you check if you have something with kill states enabled?
Firewall : Diagnostics : States Dump: Open on both machines an check if the states are equal.

[being]

Firewall : Settings : Advanced, can you check if you have something with kill states enabled?
Firewall : Diagnostics : States Dump: Open on both machines an check if the states are equal.

Kill states    "Disable State Killing on Gateway Failure" is ticked.

States are not equal tho.
But I do see constant pfsync traffic from Master to Backup from both sides! And the increase of packets picks up speed a lot when I download a big file!
Which should imply that the states are actually sent from Master to Backup.
10.0.0.1 is Master and 10.0.0.2 is backup.

It could be, that since it's not a standard protocol, VirtualBox might have problems with forwarding it correctly?

[being]

Firewall : Settings : Advanced, can you check if you have something with kill states enabled?
Firewall : Diagnostics : States Dump: Open on both machines an check if the states are equal.

Kill states    "Disable State Killing on Gateway Failure" is ticked.

States are not equal tho.
But I do see constant pfsync traffic from Master to Backup from both sides! And the increase of packets picks up speed a lot when I download a big file!
Which should imply that the states are actually sent from Master to Backup.
10.0.0.1 is Master and 10.0.0.2 is backup.

It could be, that since it's not a standard protocol, VirtualBox might have problems with forwarding it correctly?

Changed Promiscuous Mode on the HA interface from "Allow All" to "Allow VMs" and the sessions were transferred. Only to encounter a new issue.

Seems like packets are sent twice or more times from OPNsense towards the client after the Master joins back the cluster. And this applies only for the sessions that were transferred. New sessions work fine.