Messages

Hi,
the problem seems to be fixed.
The solution was a new installation of the OPNSense and an import from the configuration backup.

Hi,
after an update to the current Version (21.7.1) from some versions behind, I get some times a CARP demotion level of -240 on backup OPNSense.

On an error the demotion level is increased by 240. So far, so good but, if the demotion level is decreased it doesn't stop at 0 instead went down to -240. This causes the backup firewall to take over the carp IPs

Code Select Expand

vtnet0: deletion failed: 3
vtnet0: deletion failed: 3
vtnet0: deletion failed: 3
carp: 27@vtnet0: BACKUP -> INIT (hardware interface up)
carp: 27@vtnet0: INIT -> BACKUP (initialization complete)
vtnet2: deletion failed: 3
vtnet2: deletion failed: 3
vtnet2: deletion failed: 3
carp: 200@vtnet2: BACKUP -> INIT (hardware interface up)
carp: 200@vtnet2: INIT -> BACKUP (initialization complete)
pflog0: permanently promiscuous mode enabled
carp: demoted by -480 to 0 (sysctl)
carp: demoted by -240 to -240 (pfsync bulk fail)
carp: 24@vtnet0: BACKUP -> MASTER (preempting a slower master)
carp: 200@vtnet2: BACKUP -> MASTER (preempting a slower master)
carp: 17@vtnet0: BACKUP -> MASTER (preempting a slower master)


How can I fix this behavior or further investigate the cause?

Thanks

Hi,
I have a OPNSense master (FW1) backup (FW2) setup with CARP IPs on the LAN and WAN site (preempt enabled).

Under normal conditions everything works as expected.
If I disconnect the LAN or WAN cable from the master (FW1). All CARP IPs are switched over to the backup OPNSense (FW2) which becomes the CARP master.

But if in the situation that the master OPNSense (FW1) is in "Persistent CARP Maintenance Mode" (CARP demotion level 240) and the backup OPNSense (FW2) is therefore CARP master, an interface e.g. WAN from the backup OPNSense (FW2) gets faulty (or disconnected -> CARP demotion level 240) only the WAN IP switches back to the FW1 (which is in Maintenance Mode) and the LAN IP stays on FW". Therfor CARP WAN and LAN IPs are not on the same OPNSense.

Is that behaviour intended?

Is it possible to change CARP environment variables to prevent the IP split in this situation. In other words mack the LAN IP switch with the WAN IP on FW1 although it is in Maintenance Mode?

The reason for that is that FW1 was in "Persistent CARP Maintenance Mode" after a bust of interface errors. And I wound like still have the possibility for a fallback if an interface on FW2 went relay down.

OPNsense Version 21.1.6

net.inet.carp.ifdown_demotion_factor: 240
net.inet.carp.senderr_demotion_factor: 240
net.inet.carp.demotion: 0
net.inet.carp.log: 1
net.inet.carp.preempt: 1
net.inet.carp.dscp: 56
net.inet.carp.allow: 1
net.pfsync.carp_demotion_factor: 240

(net.inet.carp.demotion change to 240 in the described situations)

Hello,
we had the problem that some private CAPT IPs were switched over from the master to the slave. But none of the public IP-addresses where switched over. Which resulted in inaccessible VMs behind the OPNSense.

On the Setup we have 233 CARP IPs (100+ on the wan interface and the rest on the lan site distributed over 60+ VLAN interfaces).

As fare as I understand the CARP preempt setting should result in a switch over from all CARP IPs at once

Code Select Expand

# sysctl net.inet.carp.preempt
net.inet.carp.preempt: 1

What can be the reason for that behavior?

Version:
OPNsense 19.7.2-amd64
FreeBSD 11.2-RELEASE-p12-HBSD
OpenSSL 1.0.2s 28 May 2019

Currently, CARP is disabled on the slave to prevent that from happening again.
ifconfig curently reports a advskew value of 254 for all CARP IPs. That may be curelated to that bug:
https://github.com/opnsense/core/issues/3671
but not a explenation for the problem.

Code Select Expand

# sysctl -a | grep carp
<6>carp: 19@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 96@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 83@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 10@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 106@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 53@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 16@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 60@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 25@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 44@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 10@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 83@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 96@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 19@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 75@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 44@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 25@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 60@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 16@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 53@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 106@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 106@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 53@igb0: BACKUP -> MASTER (master timed out)
....
<6>carp: 107@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 37@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 95@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 2@igb1_vlan209: BACKUP -> MASTER (master timed out)
<6>carp: 2@igb1_vlan228: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan209: BACKUP -> MASTER (master timed out)
<6>carp: 2@igb1_vlan210: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan260: BACKUP -> MASTER (master timed out)
<6>carp: 9@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 2@igb1_vlan246: BACKUP -> MASTER (master timed out)
<6>carp: 84@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan238: BACKUP -> MASTER (master timed out)
<6>carp: 4@igb1_vlan201: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan231: BACKUP -> MASTER (master timed out)
<6>carp: 2@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan248: BACKUP -> MASTER (master timed out)
<6>carp: 67@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 36@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan235: BACKUP -> MASTER (master timed out)
<6>carp: 63@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 34@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 54@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan258: BACKUP -> MASTER (master timed out)
....
<6>carp: 104@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 93@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 89@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 100@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 99@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 90@igb0: BACKUP -> MASTER (master timed out)
net.inet.carp.ifdown_demotion_factor: 240
net.inet.carp.senderr_demotion_factor: 240
net.inet.carp.demotion: 0
net.inet.carp.log: 1
net.inet.carp.preempt: 1
net.inet.carp.allow: 1
net.pfsync.carp_demotion_factor: 240

system.log

Code Select Expand

Oct 15 03:12:47 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x0 - CARP IP HA Proxy (75@igb0)" has resumed the state "MASTER" for vhid 75
Oct 15 03:12:47 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x0 - CARP IP HA Proxy.
Oct 15 03:12:48 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x01 - CARP IP HA Proxy (10@igb0)" has resumed the state "MASTER" for vhid 10
Oct 15 03:12:48 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x01 - CARP IP HA Proxy.
Oct 15 03:12:49 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x51 - CARP IP HA Proxy (60@igb0)" has resumed the state "MASTER" for vhid 60
Oct 15 03:12:49 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x51 - CARP IP HA Proxy.
Oct 15 03:12:50 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (25@igb0)" has resumed the state "MASTER" for vhid 25
Oct 15 03:12:50 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:12:51 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x07 - CARP IP HA Proxy (16@igb0)" has resumed the state "BACKUP" for vhid 16
Oct 15 03:12:51 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x07 - CARP IP HA Proxy.
Oct 15 03:12:53 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (106@igb0)" has resumed the state "BACKUP" for vhid 106
Oct 15 03:12:53 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:12:54 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x44 - CARP IP HA Proxy (53@igb0)" has resumed the state "BACKUP" for vhid 53
Oct 15 03:12:54 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x44 - CARP IP HA Proxy.
Oct 15 03:12:55 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (66@igb0)" has resumed the state "BACKUP" for vhid 66
Oct 15 03:12:55 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:12:56 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x1 - CARP IP HA Proxy (109@igb0)" has resumed the state "BACKUP" for vhid 109
Oct 15 03:12:56 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x1 - CARP IP HA Proxy.
Oct 15 03:12:57 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x1 - CARP IP HA Proxy (108@igb0)" has resumed the state "BACKUP" for vhid 108
Oct 15 03:12:57 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x1 - CARP IP HA Proxy.
Oct 15 03:12:58 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x54 - CARP IP HA Proxy (63@igb0)" has resumed the state "BACKUP" for vhid 63
Oct 15 03:12:58 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x54 - CARP IP HA Proxy.
Oct 15 03:13:00 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x19 - CARP IP HA Proxy (31@igb0)" has resumed the state "BACKUP" for vhid 31
Oct 15 03:13:00 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x19 - CARP IP HA Proxy.
Oct 15 03:13:01 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x1 - MGMT_TRAFFIC CARP (64@igb0)" has resumed the state "BACKUP" for vhid 64
Oct 15 03:13:01 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x1 - MGMT_TRAFFIC CARP.
Oct 15 03:13:02 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (26@igb0)" has resumed the state "BACKUP" for vhid 26
Oct 15 03:13:02 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:13:03 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (105@igb0)" has resumed the state "BACKUP" for vhid 105
Oct 15 03:13:03 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:13:04 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x26 - CARP IP HA Proxy (35@igb0)" has resumed the state "BACKUP" for vhid 35
Oct 15 03:13:04 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x26 - CARP IP HA Proxy.
Oct 15 03:13:05 os1 kernel: carp: 60@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 25@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 44@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 75@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 19@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 96@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 83@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 10@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 106@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 53@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 16@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (101@igb0)" has resumed the state "BACKUP" for vhid 101
Oct 15 03:13:05 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:13:06 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x40 - CARP IP HA Proxy (49@igb0)" has resumed the state "BACKUP" for vhid 49
Oct 15 03:13:06 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x40 - CARP IP HA Proxy.
Oct 15 03:13:08 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x45 - CARP IP HA Proxy (54@igb0)" has resumed the state "BACKUP" for vhid 54
Oct 15 03:13:08 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x45 - CARP IP HA Proxy.
Oct 15 03:13:09 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x28 - CARP IP HA Proxy (37@igb0)" has resumed the state "BACKUP" for vhid 37
Oct 15 03:13:09 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x28 - CARP IP HA Proxy.
Oct 15 03:13:10 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x8 - CARP WAN (1@igb0)" has resumed the state "BACKUP" for vhid 1
Oct 15 03:13:10 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x8 - CARP WAN.
Oct 15 03:13:11 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x1 - CARP IP (86@igb0)" has resumed the state "BACKUP" for vhid 86
Oct 15 03:13:11 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x1 - CARP IP.
Oct 15 03:13:12 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x0 - CARP IP HA Proxy (75@igb0)" has resumed the state "BACKUP" for vhid 75
Oct 15 03:13:12 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x0 - CARP IP HA Proxy.
Oct 15 03:13:13 os1 kernel: carp: 16@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 53@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 106@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 10@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 83@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 96@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 19@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 75@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 44@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 25@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 60@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x01 - CARP IP HA Proxy (10@igb0)" has resumed the state "BACKUP" for vhid 10
Oct 15 03:13:13 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x01 - CARP IP HA Proxy.
Oct 15 03:13:15 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x51 - CARP IP HA Proxy (60@igb0)" has resumed the state "BACKUP" for vhid 60
Oct 15 03:13:15 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x51 - CARP IP HA Proxy.
Oct 15 03:13:16 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (25@igb0)" has resumed the state "BACKUP" for vhid 25