High Availability won't come up?

[loganx1121]

Going to link to a reddit post since screenshots here are so limited.  If anyone has any ideas I'd appreciate it. 

https://www.reddit.com/r/OPNsenseFirewall/comments/xwigt7/cant_get_ha_to_come_up_but_carp_masterslave_works/

[Patrick M. Hausen]

Do you have an "allow all" rule on the synchronisation interface? The master needs to access the UI/API of the backup.

[loganx1121]

Sure do.  Sync interface has an any/any rule for IPv4 on both firewalls.  I am occasionally seeing traffic go from the master to the backup on the backup firewall's sync IP, dest port 443.  It's maybe once every 15-20 minutes

[loganx1121]

I just threw 2 new opnsense appliances into the same GNS3 map.  Did the preliminary CLI config the exact same way on both and only setup the LAN interface.  Didn't setup a WAN interface at all. 
Setup the SYNC interface through the GUI, made the FW rule for it on both FW's, pinged across the HA link and tried to set it up again, still can't get it to come up. 

[loganx1121]

Partial output from pcap on sync interface.

Code: [Select]

Interface Capture output
SYNC
vtnet7 20:27:48.724343 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 112
    update compressed count 1
    eof count 1
    update compressed count 1
    eof count 1
    delete compressed count 33
    eof count 1
    delete compressed count 32
    eof count 1
    delete compressed count 1
    eof count 1
    insert count 5
    update compressed count 1
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 3
    eof count 1
    insert count 2
    eof count 1
    delete compressed count 1
    eof count 1
    delete compressed count 1
    eof count 1
    delete compressed count 32
    eof count 1
    delete compressed count 1
    eof count 1
    insert count 1
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 2
    eof count 1
    insert count 2
    eof count 1
    delete compressed count 32
    eof count 1
    delete compressed count 32
    eof count 1
    delete compressed count 1
    eof count 1
    delete compressed count 33
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 2
    eof count 1
    insert count 2
    eof count 1
    delete compressed count 32
    eof count 1
    delete compressed count 32
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 2
    eof count 1
    insert count 3
    eof count 1
    delete compressed count 32
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 2
    eof count 1
    insert count 2
    eof count 1
    delete compressed count 1
    eof count 1
    delete compressed count 32
    eof count 1
    delete compressed count 32
    eof count 1
    delete compressed count 32
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 2
    eof count 1
    insert count 2
    eof count 1
    insert count 1
    eof count 1
    delete compressed count 32
    eof count 1
    delete compressed count 32
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
    insert count 6
    eof count 1
SYNC
vtnet7 20:27:49.773014 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 112
SYNC
vtnet7 20:27:57.373136 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 424
SYNC
vtnet7 20:28:17.602986 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 412
SYNC
vtnet7 20:28:17.603010 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 40
SYNC
vtnet7 20:28:20.906353 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1326
SYNC
vtnet7 20:28:20.907436 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:28:20.910140 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:28:20.911269 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:28:20.915713 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:28:20.917060 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:28:20.919516 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:28:20.920972 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:28:20.923339 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:28:20.926045 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:28:21.102935 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 754
SYNC
vtnet7 20:28:21.902933 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 512
SYNC
vtnet7 20:28:37.813036 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 40
SYNC
vtnet7 20:28:47.912856 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 40
SYNC
vtnet7 20:28:58.012996 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 412
SYNC
vtnet7 20:29:08.113119 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 40
SYNC
vtnet7 20:29:24.812853 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 270
SYNC
vtnet7 20:29:24.958817 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:29:24.959370 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:29:24.964922 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:29:24.965462 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:29:24.968374 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:29:24.968875 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:29:24.974115 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:29:24.975257 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:29:24.979211 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:29:24.979784 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:29:26.009223 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 512
SYNC
vtnet7 20:29:26.009235 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 512
SYNC
vtnet7 20:29:28.313069 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 412
SYNC
vtnet7 20:29:58.613263 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 412
SYNC
vtnet7 20:29:58.613298 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 40
SYNC
vtnet7 20:30:28.912973 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 424
SYNC
vtnet7 20:30:29.017923 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:30:29.020077 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:30:29.022984 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:30:29.023887 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:30:29.027202 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:30:29.028249 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:30:29.031411 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:30:29.032259 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:30:29.034820 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:30:29.035469 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:30:30.012837 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 512
SYNC
vtnet7 20:30:30.012872 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 512
SYNC
vtnet7 20:31:09.372959 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 412
SYNC
vtnet7 20:31:29.572960 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 412
SYNC
vtnet7 20:31:33.078201 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:31:33.078816 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:31:33.081767 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:31:33.082276 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:31:33.087036 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:31:33.087712 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:31:33.090695 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:31:33.091188 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:31:33.094053 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:31:33.094633 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480
SYNC
vtnet7 20:31:34.073928 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 512
SYNC
vtnet7 20:31:34.073953 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 754
SYNC
vtnet7 20:32:09.973051 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 412
SYNC
vtnet7 20:32:37.177911 IP 10.0.10.1 > 10.0.10.2: PFSYNCv5 len 1480


[loganx1121]

Switched IP's of the sync interface. Still doesn't work...but for the first time I'm seeing pfsync as the protocol in the master firewall logs. Added screenshot

[loganx1121]

I downloaded the iso's and moved them to vmware workstation.  Same thing.  I can ping across my sync interface but HA shows "The backup firewall is not accessible or not configured".

Also saw the attached screenshot in the VMware workstation console output

[loganx1121]

So, out of pure desperation, I downloaded version 22.1.2_2 (was trying to do this with 22.7), loaded it up in VMware and it worked immediately.  So I guess don't use 22.7 if you want HA.

 My test firewalls didn't have 'real" internet access to update to 22.7.x so, in case anyone else tries this, maybe set that up and update to something other than base 22.7, or use an older version.

[vernetzen]

So, out of pure desperation, I downloaded version 22.1.2_2 (was trying to do this with 22.7), loaded it up in VMware and it worked immediately.  So I guess don't use 22.7 if you want HA.

 My test firewalls didn't have 'real" internet access to update to 22.7.x so, in case anyone else tries this, maybe set that up and update to something other than base 22.7, or use an older version.

Seems to be the case. HA was working with ZERO issues until the upgrade. Standard failover/upgrade/carp-maintenance/upgrade/fail-back ... the OS gets updated and everything except HA is working.

I've triple checked firewall rules, pfsync account password, XMLRPC sync is allowed via HTTPS, etc etc.. all the usual step. Doesn't matter. Then the two will fight over which one is primary because they're both sitting around at either 0 or 240, and occasionally the logs will have "pfsync bulk start" followed by "pfsync bulk failed".

There's nothing wrong with the settings, nothing wrong with the physical hardware (two R630 with CARP going over a direct physical link that has been functionally validated), the only thing that changed was upgrading the two boxes to 22.7.5

Not a great situation. Have already burned six hours wasting time checking all of the settings that were exactly the same as before this upgrade. If I have to downgrade the hosts to a different release this is going to be a huge PITA. Bug fix please!

[loganx1121]

So wait.  Is it working on 22.7.5?

What version were you on when you realized it wasn't working for you?

I assumed it was some bug in the 22.7 base version.  My actual FW at my house is 22.7.2 but I don't have another firewall to test HA with that version.