[solved] pfSync not syncing states. CARP limitations ? vtnet ? VLAN ? Broadcast?

[toxic]

Hello,

I've followed the guide at https://docs.opnsense.org/manual/how-tos/carp.html and I do get my nodes switching from backup to master and reverse.

But I don't manage to get my states synced...

The only difference I see is that they communicate on a dedicated network, but they are not alone on that network, other nodes are present that will not participate in CARP, but they are my switches and so on on their management interface, so I have no fear for security on this VLAN, but it is a VLAN and not a direct cable between the 2 boxes...

Is that a known limit of CARP that I couldn't see ?

Or am I missing something ??

On master, I can get several thousands of states when the backup only has less than a hundred...
And when I start an ssh session between 2 machines and filter the states, I do see it on the active master but never on the backup, therefore upon shutdown of master the ssh session breaks... But I can start a new one, so the CARP IP for the gateway has indeed changed.

InInterface/ VIP/Status I see this on the master :

Code: [Select]

pfSync nodes
0c4b8edd
80503395
b47f8193
b95fd8c4
8f126435
7f757b24
a87f1071
42def9a3
01de2c4c
fb7c4474
786810a3
9aa1708f
8c3ee092
bc06947f
fb97c1f7
5d2d9f63
and only this on the backup :

Code: [Select]

pfSync nodes
f5c1236f
41242b9c
eb4db7be
d965f991
05676a28
89c378c8
d7408ebe
f8903274
d662c183
c64890b4
My pfsync interface is opt9 (vlan 9 on vtnet0).
Both master and backup are using virtual NICs passed by proxmox, physically they are all Intel I211 and on both proxmox hosts they are bridged together in a bond with strictly the same setup (4 ports static LAG, tested and working iperf3, or smb file transfers...)

Thanks in advance for any pointer as to what could cause this.

Regards,
Toxic.

Edit: I just checked, I understand CARP is doing broadcast, so I verified but all IGMP snooping is disabled from my switches.

[toxic]

Ok, I think it's not realy explicit in the doc, but in fact, just enabling "sync states" on the backup firewall and giving it the IP of the master seems to have solved my issue !

It merits more tests, but as of now I was able to open an ssh session, reboot the master, see the backup taking over the gateway IP, and master coming back up, allthewhile the ssh session was not closed and is still usable !

So yes, "sync states" in System/HighAvailability needs to be set on the 2 notes...

[guest28717]

I had the same issue. I'm running two instances of Opnsense in Proxmox and have a VLAN for the sync interfaces. When I had "Synchronize Peer IP" set to the default multicast address of 224.0.0.240 the states on master where not synchronizing to the states on backup. Changing the "Synchronize Peer IP" to the explicit IP address of the other Opnsense sync interface on each instance fixed the issue. Maybe I don't have some configuration correct on my switch that is blocking multicast traffic?