Unattended partly CAPT IPs switched over

[timo-fc]

Hello,
we had the problem that some private CAPT IPs were switched over from the master to the slave. But none of the public IP-addresses where switched over. Which resulted in inaccessible VMs behind the OPNSense.

On the Setup we have 233 CARP IPs (100+ on the wan interface and the rest on the lan site distributed over 60+ VLAN interfaces).

As fare as I understand the CARP preempt setting should result in a switch over from all CARP IPs at once

Code Select Expand

# sysctl net.inet.carp.preempt
net.inet.carp.preempt: 1

What can be the reason for that behavior?

Version:
OPNsense 19.7.2-amd64
FreeBSD 11.2-RELEASE-p12-HBSD
OpenSSL 1.0.2s 28 May 2019

Currently, CARP is disabled on the slave to prevent that from happening again.
ifconfig curently reports a advskew value of 254 for all CARP IPs. That may be curelated to that bug:
https://github.com/opnsense/core/issues/3671
but not a explenation for the problem.

Code Select Expand

# sysctl -a | grep carp
<6>carp: 19@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 96@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 83@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 10@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 106@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 53@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 16@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 60@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 25@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 44@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 10@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 83@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 96@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 19@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 75@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 44@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 25@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 60@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 16@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 53@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 106@igb0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 106@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 53@igb0: BACKUP -> MASTER (master timed out)
....
<6>carp: 107@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 37@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 95@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 2@igb1_vlan209: BACKUP -> MASTER (master timed out)
<6>carp: 2@igb1_vlan228: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan209: BACKUP -> MASTER (master timed out)
<6>carp: 2@igb1_vlan210: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan260: BACKUP -> MASTER (master timed out)
<6>carp: 9@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 2@igb1_vlan246: BACKUP -> MASTER (master timed out)
<6>carp: 84@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan238: BACKUP -> MASTER (master timed out)
<6>carp: 4@igb1_vlan201: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan231: BACKUP -> MASTER (master timed out)
<6>carp: 2@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan248: BACKUP -> MASTER (master timed out)
<6>carp: 67@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 36@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan235: BACKUP -> MASTER (master timed out)
<6>carp: 63@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 34@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 54@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 1@igb1_vlan258: BACKUP -> MASTER (master timed out)
....
<6>carp: 104@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 93@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 89@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 100@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 99@igb0: BACKUP -> MASTER (master timed out)
<6>carp: 90@igb0: BACKUP -> MASTER (master timed out)
net.inet.carp.ifdown_demotion_factor: 240
net.inet.carp.senderr_demotion_factor: 240
net.inet.carp.demotion: 0
net.inet.carp.log: 1
net.inet.carp.preempt: 1
net.inet.carp.allow: 1
net.pfsync.carp_demotion_factor: 240

system.log

Code Select Expand

Oct 15 03:12:47 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x0 - CARP IP HA Proxy (75@igb0)" has resumed the state "MASTER" for vhid 75
Oct 15 03:12:47 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x0 - CARP IP HA Proxy.
Oct 15 03:12:48 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x01 - CARP IP HA Proxy (10@igb0)" has resumed the state "MASTER" for vhid 10
Oct 15 03:12:48 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x01 - CARP IP HA Proxy.
Oct 15 03:12:49 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x51 - CARP IP HA Proxy (60@igb0)" has resumed the state "MASTER" for vhid 60
Oct 15 03:12:49 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x51 - CARP IP HA Proxy.
Oct 15 03:12:50 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (25@igb0)" has resumed the state "MASTER" for vhid 25
Oct 15 03:12:50 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:12:51 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x07 - CARP IP HA Proxy (16@igb0)" has resumed the state "BACKUP" for vhid 16
Oct 15 03:12:51 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x07 - CARP IP HA Proxy.
Oct 15 03:12:53 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (106@igb0)" has resumed the state "BACKUP" for vhid 106
Oct 15 03:12:53 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:12:54 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x44 - CARP IP HA Proxy (53@igb0)" has resumed the state "BACKUP" for vhid 53
Oct 15 03:12:54 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x44 - CARP IP HA Proxy.
Oct 15 03:12:55 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (66@igb0)" has resumed the state "BACKUP" for vhid 66
Oct 15 03:12:55 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:12:56 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x1 - CARP IP HA Proxy (109@igb0)" has resumed the state "BACKUP" for vhid 109
Oct 15 03:12:56 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x1 - CARP IP HA Proxy.
Oct 15 03:12:57 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x1 - CARP IP HA Proxy (108@igb0)" has resumed the state "BACKUP" for vhid 108
Oct 15 03:12:57 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x1 - CARP IP HA Proxy.
Oct 15 03:12:58 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x54 - CARP IP HA Proxy (63@igb0)" has resumed the state "BACKUP" for vhid 63
Oct 15 03:12:58 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x54 - CARP IP HA Proxy.
Oct 15 03:13:00 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x19 - CARP IP HA Proxy (31@igb0)" has resumed the state "BACKUP" for vhid 31
Oct 15 03:13:00 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x19 - CARP IP HA Proxy.
Oct 15 03:13:01 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x1 - MGMT_TRAFFIC CARP (64@igb0)" has resumed the state "BACKUP" for vhid 64
Oct 15 03:13:01 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x1 - MGMT_TRAFFIC CARP.
Oct 15 03:13:02 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (26@igb0)" has resumed the state "BACKUP" for vhid 26
Oct 15 03:13:02 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:13:03 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (105@igb0)" has resumed the state "BACKUP" for vhid 105
Oct 15 03:13:03 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:13:04 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x26 - CARP IP HA Proxy (35@igb0)" has resumed the state "BACKUP" for vhid 35
Oct 15 03:13:04 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x26 - CARP IP HA Proxy.
Oct 15 03:13:05 os1 kernel: carp: 60@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 25@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 44@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 75@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 19@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 96@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 83@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 10@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 106@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 53@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 kernel: carp: 16@igb0: MASTER -> BACKUP (more frequent advertisement received)
Oct 15 03:13:05 os1 kernel: ifa_maintain_loopback_route: deletion failed for interface igb0: 3
Oct 15 03:13:05 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (101@igb0)" has resumed the state "BACKUP" for vhid 101
Oct 15 03:13:05 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x - CARP IP HA Proxy.
Oct 15 03:13:06 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x40 - CARP IP HA Proxy (49@igb0)" has resumed the state "BACKUP" for vhid 49
Oct 15 03:13:06 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x40 - CARP IP HA Proxy.
Oct 15 03:13:08 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x45 - CARP IP HA Proxy (54@igb0)" has resumed the state "BACKUP" for vhid 54
Oct 15 03:13:08 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x45 - CARP IP HA Proxy.
Oct 15 03:13:09 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x28 - CARP IP HA Proxy (37@igb0)" has resumed the state "BACKUP" for vhid 37
Oct 15 03:13:09 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x28 - CARP IP HA Proxy.
Oct 15 03:13:10 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x8 - CARP WAN (1@igb0)" has resumed the state "BACKUP" for vhid 1
Oct 15 03:13:10 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x8 - CARP WAN.
Oct 15 03:13:11 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x1 - CARP IP (86@igb0)" has resumed the state "BACKUP" for vhid 86
Oct 15 03:13:11 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x1 - CARP IP.
Oct 15 03:13:12 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x0 - CARP IP HA Proxy (75@igb0)" has resumed the state "BACKUP" for vhid 75
Oct 15 03:13:12 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x0 - CARP IP HA Proxy.
Oct 15 03:13:13 os1 kernel: carp: 16@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 53@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 106@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 10@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 83@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 96@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 19@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 75@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 44@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 25@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 kernel: carp: 60@igb0: BACKUP -> MASTER (master timed out)
Oct 15 03:13:13 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x01 - CARP IP HA Proxy (10@igb0)" has resumed the state "BACKUP" for vhid 10
Oct 15 03:13:13 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x01 - CARP IP HA Proxy.
Oct 15 03:13:15 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x51 - CARP IP HA Proxy (60@igb0)" has resumed the state "BACKUP" for vhid 60
Oct 15 03:13:15 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface xxx.xxx.xxx.x51 - CARP IP HA Proxy.
Oct 15 03:13:16 os1 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "xxx.xxx.xxx.x - CARP IP HA Proxy (25@igb0)" has resumed the state "BACKUP" for vhid 25