Messages

14 hours laters and at least 3 years older ... I have a perfectly working HA router finally.
I started to rollback all my changes (done in last let's say 7 days) this morning. And I discovered my environment is simply incompatible with "IP Alias" bound to CARP VIP.

As soon as I reconfigured all WAN IP Aliases (19 in total) back to separate WAN CARP VIPs (each with its own VHID group, subnet equal to /26, unicast, no xmlrpc sync at the end) everything started to work flawlessly. I can reboot nodes, initiate Maintenance modes as I wish, everything behaves nice and smooth.

My HA routers created as VMs in OVH private cloud work fine - expect of one problem. And I don't know what to do.
Both VMs run 25.1.3, same config, same interfaces, names etc.
Interface   Identifier
[DMZ]      opt2
[DMZ2]      opt3
[LAN]      lan
[PFSYNC]   opt1
[VPN]      opt4
[WAN]      wan

All interfaces use default 224.0.0.18 peer address in virtual IP definitions - except of WAN. For WAN I have to use unicast IPv4 of the second VM (and vice versa) with "No XMLRPC Sync" checked on.

Each node uses its own public IPv4/26 with correct DGW. I use "Manual outbound NAT rule generation", each VM initiates its own communication from its own public and dedicated IPv4/26 (and not incorrectly from VIP as described in some tickets I found in this forum). Confirmed by "curl https://ifconfig.me/ip" from both boxes.

Based on another recommendation found on this forum I created only one CARP VIP on WAN. Another 19 public IPs for WAN have been created as IP aliases with /32 subnet and with the same VHID group number created for CARP VIP. It should minimize CARP traffic. There was a catch with /26 used for IP aliases, in my case it has to be /32. I believe it's correct configuration.

So CARP VIP on WAN has "No XMLRPC Sync" check on. Another 19 IP Aliases have "No XMLRPC Sync" unchecked. Synchronization is working from primary to secondary VM without any problem.

Now I want to test router failover.
1) Let's press "Enter Persistent CARP Maintenance Mode" on primary node.
2) Primary node becomes BACKUP, secondary node becomes MASTER. So far so good.
3) I'll initiate primary node's reboot while pinging WAN CARP VIP and/or any IP Alias from the Internet.
4) All pings work ... until primary node finishes reboot. Pings to dedicated WAN IPs work for both VMs but nothing replies when pinging WAN CARP VIP or IP Alias.
5) Primary node is still BACKUP, secondary node is still MASTER. But WAN communication does not work.
6) WAN communication is restored when I press "Leave Persistent CARP Maintenance Mode" on primary node.

Do you have any idea what may be wrong, what should I check again?

Based on this article (https://forum.opnsense.org/index.php?topic=39906.0) I planned to test the same procedure described above (steps 1-6) with an additional step:
2a) On node1 (master) - Go to "Interfaces: Virtual IPs: Settings" and look at one of the CARP Vips, expand advanced mode, look at the "advskew" - it should be something like 0 or 1. Set this around 100 higher than node2.

The problem is I am not able to find "advskew" in 25.1.3 GUI :) Clicking "Advanced" mode shows/hides Gateway form field only. Is it expected to not see "advskew" in 25.1.3?

Hello,
I am happy to confirm that the fix provided in two patches above is working. I synced the initial Virtual IP configuration from master to backup, checked "No XMLRPC Sync" on master for all unicast-based Virtual IPs. After that I checked "No XMLRPC Sync" on backup for all unicast-based Virtual IPs and also changed "Peer (ipv4)" for all unicast-based Virtual IPs.

I've just initiated "Synchronize and reconfigure all" and everything is ok on backup node.

Thanks the whole team for a quick fix!

Sure, I was just checking how to get to the patch because milestone 25.7 (which the solution was assigned too) is so far :)

Thank you for a quick test and confirmation that I am not doing any mistake. I also tried to create the same record on the backup node with the same results - it's deleted during the nearest sync from master to backup.

Let's see what devs will come up with. For me it would be great if it would work exactly as described in help. Right now I have 24 VIPs, 20 of them on WAN when I am forced to use unicast.

Meanwhile I am going to study other small but important differences between OPNsense and pfSense.

Hello,
our company decided to move from pfSense to OPNsense. So I am quite new to OPNsense. I've created CARP-based redundant fw to compare with long years used CARP-based redundat pfSense box. We use OVH, I am able to use directed multicast on all interfaces except of WAN where I am force to use unicast because of OVH cloud provider.

I use the latest OPNsense 25.1.2 on both nodes. There is no problem with CARP as such, I am able to failover and failback with all IP addresses on all interfaces. The problem is with syncing the configuration (including Virtual IPs) via pfsync from master to slave (or backup node).

Inline help describes exactly what I want to achieve:
"Exclude this item from the HA synchronization process. An already existing item with the same UUID on the synchronization target will not be altered or deleted as long as this is active. This option can be helpful when using Unicast CARP. After the initial synchronization, enable this option and adjust the Unicast IPs on the backup firewall. Additional IP aliases in the same VHID group can now be synced without overwriting their parent CARP VIP."

Let me describe the steps:
1) create unicast-based Virtual IP on master, do not check No XMLRPC Sync
2) sync configuration to backup node
3) reconfigure "Peer (ipv4)" so it refers to master's IP and to backup's IP
4) test CARP, failover, failback, all good
5) on master, check "No XMLRPC Sync". I tried to check this on backup node too, nothing changes for the result.
6) sync configuration to backup node

Unfortunately the whole Virtual IP address configuration using unicast disappears on backup node after finishing step 6.

Am I doing something wrong?

In System / High Availability / Settings I've chosen the following services to be synchronized via XMLRPC Sync:
Aliases, Certificates, DHCPD, Firewall Categories, Firewall Groups, Firewall Log Templates, Firewall Rules, Firewall Schedules, NAT, Static Routes, Unbound DNS, Users and Groups, Virtual IPs, Web GUI.

I didn't test it on 25.1.1, I've just finished the configuration this morning.

Best regards,

Radek