Problem with pfSync

[danbet]

I went through the following document: https://docs.opnsense.org/manual/how-tos/carp.html.
Unfortunately pfSync doesn't work:
carp: demoted by -240 to 0 (pfsync bulk fail)

I may have missed something, but I can't find anything. I left out DHCP.

However, I don't have a direct connection between the two devices master and backup because they are virtual machines. I still created my own interfaces that are on the same network as the LAN.
Would it be better if I selected the LAN interface directly for synchronization?

[Monviech (Cedrik)]

pfsync only works if on both VMs the interface names (LAN, WAN, OPT1 etc...) and have the exact same (physical or virtual) interface associations on both sides. Have you made sure of that?

Depending on the size of the state table thats expected, it might be best to isolate the pfsync traffic into a seperate VLAN with their own vNICs because it is very multicast heavy.

Also make sure, that the chosen interfaces that communicate over pfsync, have a firewall rule that allow the "Protocol PFSYNC" direction in from any source to any destination.

Also as an addendum:
- CARP
- xmlrpc sync
- pfsync

Are totally independant from each other. If you don't need firewall state syncronisation, you can safely leave pfsync off. If there is a failover, sessions will be newly initiated. Just using CARP is enough if theres just failover requirement without the need of sessions being carried over seamlessly.

[danbet]

The interfaces were correct, the problem was the rule of the pfSync interface. I had to allow any protocol on this, otherwise it won't work. I now only allow the pfSync network as a source. Is that right?

With VMware ESXi you can create port groups and assign them a VLAN ID. Is that enough, or does this VLAN ID also have to be specified on the switch?

The point about complete independence is very valuable to me. I actually don't need status synchronization. But isn't the configuration also transferred via pfSync?

[danbet]

Can I use Hybrid creation of outgoing NAT rules (automatically generated rules are applied after the manual rules), instead of manual outbound nat rules?

[Monviech (Cedrik)]

No the configuration is transferred via xmlrpc sync. It's a different mechanism that uses the REST API to transfer the configuration. PFSYNC is its own protocol, used by the pf packet filter to synchronize states. https://man.freebsd.org/cgi/man.cgi?pfsync%284%29

And yes you can use hybrid creation of outbound NAT.
- The automatic rules will all still exist
- You can create your own rules too.

[danbet]

Does that mean, if I just want to synchronize the configuration, that I only fill out the lower part of the HA Settings page and leave the upper part empty?

[Monviech (Cedrik)]

Yes that is correct. The configuration sync is done with the checkboxes below. State synchronization doesn't synchronize the configuration.

The configuration sync is done manually. You can create a cronjob though if you want.

[danbet]

OK, and for this I dont use a separate pfSync connection. Good.

[mccasian]

Hello there,

Although the problem is solved for you, more people will visit this thread looking for a solution on this problem, therefore I want to raise another important aspect which can prevent the sync between the firewalls.

Please be aware that in order for the sync to work properly, the slave machine must have the interface used for pfSync on the list of interfaces where WebGUI is listening: System -> Settings -> Administration -> Listen Interfaces(under Web GUI). Although the master can ping the slave, the FW is not blocking anything, without the port 443 being open on the pfSync interface on slave, it will simply not be able to connect.

before:
Error:/usr/local/etc/rc.filter_synchronize: An error occurred while attempting XMLRPC sync with username root and https://10.9.8.3/xmlrpc.php fetch error. remote host down?
nc -vz 10.9.8.3 443
nc: connect to 10.9.8.3 port 443 (tcp) failed: Connection refused

after:
nc -vz 10.9.8.3 443
Connection to 10.9.8.3 443 port [tcp/https] succeeded!
Notice:/usr/local/etc/rc.filter_synchronize: Filter sync successfully completed with https://10.9.8.3/xmlrpc.php.

That solved the problem for me.