pfsync through VPN

[ctmarc]

Hello, I'm trying to get a failover group to work, similar to the article https://docs.opnsense.org/manual/how-tos/carp.html. I'm afraid I'm not very experienced with networking, but am trying to catch up.

The situation is a bit different from the manual because the 2 VMs that we use both have only 1 WAN network interface, and no LAN interface. What I'm trying to do is to group both firewalls in a VPN and have pfsync use that tunnel for synchronisation. Later there would be a second VPN that connects to the CARP virtual IP, the final goal would be to have a highly-available VPN server. But my problems start already in the first part.

pfsync is not fully synchronizing and both the main, and the backup firewall claim "MASTER" status on the VIP.
I suspect that my interface setup is the root of the problem:

Master node:
WAN: vtnet0
SYNC: ovpns1 (the VPN server)

Backup node:
WAN: vtnet0
SYNC: ovpnc1 (the VPN client)

On the master firewall, the synchronisation appears to work, as I get "pfsync bulk done" in the master's logfile, and the XMLRPC sync is correctly replicating most of the configuration on the backup node. The WAN starts to advertise the VIP with "skew 0" on the master node and "skew 100" on the backup node.

But the backup FW is not assuming backup state, both ifconfig and on the Firewall/VIP/status page are saying "MASTER", and the "settings > high availability" page shows the default blue box ("no backup node configured").

When I "enter persistent CARP maintenance" on the master FW, the log shows the following output:

Code: [Select]

Jul 17 09:30:00 opnsense: /firewall_virtual_ip.php: The command `/sbin/ifconfig 'vtnet0' -alias '111.222.333.45'' failed to execute
Jul 17 09:29:02 kernel: carp: demoted by -240 to 0 (sysctl)
Jul 17 09:28:10 kernel: carp: demoted by 240 to 240 (sysctl)
Jul 17 09:26:17 kernel: vtnet0: promiscuous mode disabled
Jul 17 09:26:17 kernel: carp: 192@vtnet0: MASTER -> INIT (hardware interface up)
Jul 17 09:26:17 kernel: ifa_maintain_loopback_route: deletion failed for interface vtnet0: 3
Jul 17 09:26:17 kernel: ifa_maintain_loopback_route: deletion failed for interface vtnet0: 3

And afterwards, vtnet0 has lost it's VIP alias, and the interface stops advertising the CARP address. The backup firewall still advertises with unchanged skew (100). After leaving persistent CARP maintenance, the virtual IP remains lost, and it's only restored after a reboot.

I suspected that the problem is the difference between the SYNC interfaces on master and backup, because one is linked to the interface ovpns1 and the other to ovpnc1. But then again, the problem appears to come from vtnet0 which is the WAN interface, and which is exactly the same on both nodes.

I'm trying all sorts of things but so far nothing would really work. Currently a bit stuck and if anybody has a tip I'd greatly appreciate it. Does anybody have a tip on how to fix that pfsync / CARP installation?

Greetings
marc

[ctmarc]

Problem solved by upgrading from 19.1 to 19.7.
I can enter and leave persistent CARP maintenance mode and the skew is changing as expected.