CARP Master randomly gets set to 1200 demotion level

[mmaxwell]

OS: OPNsense 23.1.7_3-amd64
Device: DEC3840

We have HA setup on two OPNSense firewalls w/ CARP configured. Once every 2 or 3 weeks, our master firewall will get demoted to a level of 1200. I have looked at the logs but do not see a reason why the demotion of the master firewall. I will attach a screenshot and logs from when the demotion happened below. Please let me know if I can provide any other information to help. I have the two firewalls in separate server rooms connected directly to each other via 10Gb fiber.

System > Log Files > General

Code: [Select]

2023-05-06T03:04:00-05:00 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure route_reload (execute task : core_routing_batch(1,[]))
2023-05-06T03:04:00-05:00 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure route_reload (1,[])
2023-05-06T03:04:00-05:00 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure ipsec (execute task : ipsec_configure_do(1))
2023-05-06T03:04:00-05:00 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure ipsec (1)
2023-05-06T03:03:21-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface .
2023-05-06T03:03:21-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "  (1@igb0_vlan168)" has resumed the state "BACKUP" for vhid 1
2023-05-06T03:03:21-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Accesscontrol - Devices - just Internet .
2023-05-06T03:03:21-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Accesscontrol - Devices - just Internet  (13@igb0_vlan20)" has resumed the state "BACKUP" for vhid 13
2023-05-06T03:03:20-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface PH-GUEST - just Internet .
2023-05-06T03:03:20-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "PH-GUEST - just Internet  (12@igb0_vlan999)" has resumed the state "BACKUP" for vhid 12
2023-05-06T03:03:20-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface IOT - Devices - just Internet.
2023-05-06T03:03:20-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "IOT - Devices - just Internet  (10@igb0_vlan250)" has resumed the state "BACKUP" for vhid 10
2023-05-06T03:03:20-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface AV-Equipment - just Internet .
2023-05-06T03:03:20-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "AV-Equipment - just Internet  (11@igb0_vlan144)" has resumed the state "BACKUP" for vhid 11
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface .
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "  (2@igb1)" has resumed the state "BACKUP" for vhid 2
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface .
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "  (3@igb2)" has resumed the state "BACKUP" for vhid 3
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface mail - IPS2 .
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "mail - IPS2 (5@igb2)" has resumed the state "BACKUP" for vhid 5
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface mail - IPS1 .
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "mail - IPS1  (4@igb1)" has resumed the state "BACKUP" for vhid 4
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface ldap - IPS1 .
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "ldap - IPS1 (6@igb1)" has resumed the state "BACKUP" for vhid 6
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface ldap - IPS2.
2023-05-06T03:03:19-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "ldap - IPS2  (8@igb2)" has resumed the state "BACKUP" for vhid 8
2023-05-06T03:03:18-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface sp - IPS.
2023-05-06T03:03:18-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "sp - IPS1 (7@igb1)" has resumed the state "BACKUP" for vhid 7
2023-05-06T03:03:18-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface sp - IPS2 .
2023-05-06T03:03:18-05:00 Notice opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "sp - IPS2 (9@igb2)" has resumed the state "BACKUP" for vhid 9
2023-05-06T03:02:00-05:00 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure route_reload (execute task : core_routing_batch(1,[]))
2023-05-06T03:02:00-05:00 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure route_reload (1,[])
2023-05-06T03:01:59-05:00 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure ipsec (execute task : ipsec_configure_do(1))
2023-05-06T03:01:59-05:00 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure ipsec (1)

[bimbar]

There are quite a few ways in which CARP can get demoted, there are for example some routing configurations which hook into the CARP process.

[mmaxwell]

Do you know if there is documentation on what would trigger a CARP demotion? 

Also, does a demotion of 1200 seem strange? 

[mimugmail]

Can you post a dmesg -a

[mmaxwell]

Code: [Select]

root@SRVUSFW01:~ # dmesg -a
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-RELEASE-p7 stable/23.1-n250430-7eb6eb035df SMP amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.                                                                                                                                                             0-0-gd7b669b3a303)
VT(vga): resolution 640x480
CPU: AMD EPYC 3101 4-Core Processor                  (2096.17-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x800f12  Family=0x17  Model=0x1  Stepping=2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,C                                                                                                                                                             MOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POP                                                                                                                                                             CNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x35c233ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,                                                                                                                                                             SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX>
  Structured Extended Features=0x209c01a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,AD                                                                                                                                                             X,SMAP,CLFLUSHOPT,SHA>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  AMD Extended Feature Extensions ID EBX=0x1007<CLZERO,IRPerf,XSaveErPtr,IBPB>
  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8118956032 (7742 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INSYDE WALLABY>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 cache groups x 2 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0: MADT APIC ID 128 != hw id 0
ioapic1: MADT APIC ID 129 != hw id 0
ioapic0 <Version 2.1> irqs 0-23
ioapic1 <Version 2.1> irqs 24-55
Launching APs: 2 3 1
wlan: mac acl policy registered
random: entropy device external interface
kbd0 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
vtvga0: <VT VGA driver>
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x7945e000-0x7945e01e
smbios0: Version: 3.0, BCD Revision: 3.0
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <INSYDE WALLABY>
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 450
Event timer "HPET2" frequency 14318180 Hz quality 450
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
apei0: <ACPI Platform Error Interface> on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 1.3 on pci0
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) I210 Flashless (Copper)> port 0x5000-0x501f mem 0x80800000-0x808                                                                                                                                                             1ffff,0x80820000-0x80823fff at device 0.0 on pci1
igb0: NVM V0.6 imgtype6
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 4 RX queues 4 TX queues
igb0: Using MSI-X interrupts with 5 vectors
igb0: Ethernet address: f4:90:ea:00:63:67
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
pcib2: <ACPI PCI-PCI bridge> at device 1.4 on pci0
pci2: <ACPI PCI bus> on pcib2
igb1: <Intel(R) I210 Flashless (Copper)> port 0x4000-0x401f mem 0x80700000-0x807                                                                                                                                                             1ffff,0x80720000-0x80723fff at device 0.0 on pci2
igb1: NVM V0.6 imgtype6
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 4 RX queues 4 TX queues
igb1: Using MSI-X interrupts with 5 vectors
igb1: Ethernet address: f4:90:ea:00:63:68
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
pcib3: <ACPI PCI-PCI bridge> at device 1.5 on pci0
pci3: <ACPI PCI bus> on pcib3
igb2: <Intel(R) I210 Flashless (Copper)> port 0x3000-0x301f mem 0x80600000-0x806                                                                                                                                                             1ffff,0x80620000-0x80623fff at device 0.0 on pci3
igb2: NVM V0.6 imgtype6
igb2: Using 1024 TX descriptors and 1024 RX descriptors
igb2: Using 4 RX queues 4 TX queues
igb2: Using MSI-X interrupts with 5 vectors
igb2: Ethernet address: f4:90:ea:00:63:69
igb2: netmap queues/slots: TX 4/1024, RX 4/1024
pcib4: <ACPI PCI-PCI bridge> at device 1.6 on pci0
pci4: <ACPI PCI bus> on pcib4
igb3: <Intel(R) I210 Flashless (Copper)> port 0x2000-0x201f mem 0x80500000-0x805                                                                                                                                                             1ffff,0x80520000-0x80523fff at device 0.0 on pci4
igb3: NVM V0.6 imgtype6
igb3: Using 1024 TX descriptors and 1024 RX descriptors
igb3: Using 4 RX queues 4 TX queues
igb3: Using MSI-X interrupts with 5 vectors
igb3: Ethernet address: f4:90:ea:00:63:6a
igb3: netmap queues/slots: TX 4/1024, RX 4/1024
pcib5: <ACPI PCI-PCI bridge> at device 7.1 on pci0
pci5: <ACPI PCI bus> on pcib5
pci5: <unknown> at device 0.0 (no driver attached)
pci5: <encrypt/decrypt> at device 0.2 (no driver attached)
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x80200000-0x802fffff at device 0                                                                                                                                                             .3 on pci5
xhci0: 64 bytes context size, 64-bit DMA
usbus0: waiting for BIOS to give up control
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pcib6: <ACPI PCI-PCI bridge> at device 8.1 on pci0
pci6: <ACPI PCI bus> on pcib6
pci6: <unknown> at device 0.0 (no driver attached)
pci6: <encrypt/decrypt> at device 0.1 (no driver attached)
ahci0: <AMD KERNCZ AHCI SATA controller> mem 0x8018e000-0x8018efff at device 0.2                                                                                                                                                              on pci6
ahci0: AHCI v1.31 with 1 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
hdac0: <AMD X370 HDA Controller> mem 0x80180000-0x80187fff at device 0.3 on pci6
ax0: <AMD 10 Gigabit Ethernet Driver> mem 0x80160000-0x8017ffff,0x80140000-0x801                                                                                                                                                             5ffff,0x80188000-0x80189fff at device 0.4 on pci6
ax0: Using 512 TX descriptors and 512 RX descriptors
ax0: Using 4 RX queues 4 TX queues
ax0: Using MSI-X interrupts with 8 vectors
ax0: Ethernet address: f4:90:ea:00:63:6b
ax0: xgbe_config_sph_mode: SPH disabled in channel 0
ax0: xgbe_config_sph_mode: SPH disabled in channel 1
ax0: xgbe_config_sph_mode: SPH disabled in channel 2
ax0: xgbe_config_sph_mode: SPH disabled in channel 3
ax0: RSS Enabled
ax0: Receive checksum offload Enabled
ax0: VLAN filtering Enabled
ax0: VLAN Stripping Enabled
ax0: Checking GPIO expander validity
ax0: Input port registers: 0x0
ax0: Output port registers: 0x7777
ax0: Polarity port registers: 0x0
ax0: Configuration port registers: 0x77ff
ax0: SFP detected:
ax0:   vendor:   ADDON
ax0:   part number:    JD092B-AO
ax0:   revision level: A4
ax0:   serial number:  ADD8095E6349
ax0: netmap queues/slots: TX 4/512, RX 4/512
ax1: <AMD 10 Gigabit Ethernet Driver> mem 0x80120000-0x8013ffff,0x80100000-0x801                                                                                                                                                             1ffff,0x8018a000-0x8018bfff at device 0.5 on pci6
ax1: Using 512 TX descriptors and 512 RX descriptors
ax1: Using 4 RX queues 4 TX queues
ax1: Using MSI-X interrupts with 8 vectors
ax1: Ethernet address: f4:90:ea:00:63:6c
ax1: xgbe_config_sph_mode: SPH disabled in channel 0
ax1: xgbe_config_sph_mode: SPH disabled in channel 1
ax1: xgbe_config_sph_mode: SPH disabled in channel 2
ax1: xgbe_config_sph_mode: SPH disabled in channel 3
ax1: RSS Enabled
ax1: Receive checksum offload Enabled
ax1: VLAN filtering Enabled
ax1: VLAN Stripping Enabled
ax1: Checking GPIO expander validity
ax1: Input port registers: 0x0
ax1: Output port registers: 0x7777
ax1: Polarity port registers: 0x0
ax1: Configuration port registers: 0x77ff
ax1: xgbe_phy_sfp_signals: port_sfp_inputs: 0x7
ax1: xgbe_phy_sfp_detect: mod absent
ax1: netmap queues/slots: TX 4/512, RX 4/512
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
uart0: <8250 or 16450 or compatible> port 0x3f8-0x3ff irq 3 flags 0x10 on acpi0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounter "TSC" frequency 2096061728 Hz quality 1000
Timecounters tick every 1.000 msec
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
ugen0.1: <AMD XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <TS256GMTS952T2 02J0T4GB> ACS-2 ATA SATA 3.x device
ada0: Serial Number G426270039
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 244198MB (500118192 512 byte sectors)
uhub0: 8 ports with 8 removable, self powered
ax0: Link is UP - 10Gbps/Full - flow control off
ax0: link state changed to UP
carp: demoted by 240 to 240 (sysctl)
igb0: link state changed to UP
igb1: link state changed to UP
igb2: link state changed to UP
intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0
smbus0: <System Management Bus> on intsmb0
lo0: link state changed to UP
pflog0: permanently promiscuous mode enabled
igb0: link state changed to DOWN
vlan0: changing name to 'igb0_vlan144'
vlan1: changing name to 'igb0_vlan168'
vlan2: changing name to 'igb0_vlan20'
vlan3: changing name to 'igb0_vlan250'
vlan4: changing name to 'igb0_vlan999'
igb0: promiscuous mode enabled
igb0_vlan144: promiscuous mode enabled
carp: demoted by 240 to 480 (interface down)
carp: demoted by 240 to 720 (pfsync bulk start)
igb0_vlan20: promiscuous mode enabled
carp: demoted by 240 to 960 (interface down)
igb0: link state changed to UP
igb0_vlan168: link state changed to UP
igb0_vlan250: link state changed to UP
carp: 13@igb0_vlan20: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 720 (interface up)
igb0_vlan20: link state changed to UP
carp: 11@igb0_vlan144: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 480 (interface up)
igb0_vlan144: link state changed to UP
igb0_vlan999: link state changed to UP
ax0: Link is DOWN
ax0: xgbe_config_sph_mode: SPH disabled in channel 0
ax0: xgbe_config_sph_mode: SPH disabled in channel 1
ax0: xgbe_config_sph_mode: SPH disabled in channel 2
ax0: xgbe_config_sph_mode: SPH disabled in channel 3
ax0: RSS Enabled
ax0: Receive checksum offload Disabled
ax0: VLAN filtering Disabled
ax0: VLAN Stripping Disabled
ax0: Checking GPIO expander validity
ax0: Input port registers: 0x702
ax0: Output port registers: 0x7777
ax0: Polarity port registers: 0x0
ax0: Configuration port registers: 0x77ff
ax0: SFP detected:
ax0:   vendor:   ADDON
ax0:   part number:    JD092B-AO
ax0:   revision level: A4
ax0:   serial number:  ADD8095E6349
ax0: link state changed to DOWN
igb0_vlan250: promiscuous mode enabled
carp: 10@igb0_vlan250: INIT -> BACKUP (initialization complete)
igb0_vlan999: promiscuous mode enabled
carp: 12@igb0_vlan999: INIT -> BACKUP (initialization complete)
igb0_vlan168: promiscuous mode enabled
carp: 1@igb0_vlan168: INIT -> BACKUP (initialization complete)
igb1: link state changed to DOWN
igb1: promiscuous mode enabled
carp: demoted by 240 to 720 (interface down)
carp: demoted by 240 to 960 (interface down)
carp: demoted by 240 to 1200 (interface down)
carp: demoted by 240 to 1440 (interface down)
igb2: link state changed to DOWN
igb2: promiscuous mode enabled
carp: demoted by 240 to 1680 (interface down)
carp: demoted by 240 to 1920 (interface down)
carp: demoted by 240 to 2160 (interface down)
carp: demoted by 240 to 2400 (interface down)
tun1: changing name to 'ovpns1'
ax0: Link is UP - 10Gbps/Full - flow control off
ax0: link state changed to UP
carp: demoted by -240 to 2160 (pfsync bulk done)
carp: 13@igb0_vlan20: BACKUP -> MASTER (master timed out)
carp: 11@igb0_vlan144: BACKUP -> MASTER (master timed out)
carp: 10@igb0_vlan250: BACKUP -> MASTER (master timed out)
carp: 12@igb0_vlan999: BACKUP -> MASTER (master timed out)
carp: 1@igb0_vlan168: BACKUP -> MASTER (master timed out)
carp: 2@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1920 (interface up)
carp: 4@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1680 (interface up)
carp: 6@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1440 (interface up)
carp: 7@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1200 (interface up)
igb1: link state changed to UP
carp: 3@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 960 (interface up)
carp: 5@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 720 (interface up)
carp: 8@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 480 (interface up)
carp: 9@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 240 (interface up)
igb2: link state changed to UP
carp: 2@igb1: BACKUP -> MASTER (master timed out)
carp: 4@igb1: BACKUP -> MASTER (master timed out)
carp: 6@igb1: BACKUP -> MASTER (master timed out)
carp: 7@igb1: BACKUP -> MASTER (master timed out)
carp: 3@igb2: BACKUP -> MASTER (master timed out)
carp: 5@igb2: BACKUP -> MASTER (master timed out)
carp: 8@igb2: BACKUP -> MASTER (master timed out)
carp: 9@igb2: BACKUP -> MASTER (master timed out)
ovpns1: link state changed to UP
WARNING: attempt to domain_add(netgraph) after domainfinalize()
carp: 1@igb0_vlan168: MASTER -> BACKUP (more frequent advertisement received)
carp: 11@igb0_vlan144: MASTER -> BACKUP (more frequent advertisement received)
carp: 13@igb0_vlan20: MASTER -> BACKUP (more frequent advertisement received)
carp: 10@igb0_vlan250: MASTER -> BACKUP (more frequent advertisement received)
carp: 12@igb0_vlan999: MASTER -> BACKUP (more frequent advertisement received)
carp: 8@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 7@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 6@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 9@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 5@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 4@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 2@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 3@igb2: MASTER -> BACKUP (more frequent advertisement received)
510.149137 [ 851] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                              2048
510.250652 [ 851] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                              2048
carp: 2@igb1: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 480 (interface down)
carp: 4@igb1: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 720 (interface down)
carp: 6@igb1: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 960 (interface down)
carp: 7@igb1: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 1200 (interface down)
igb1: link state changed to DOWN
510.558695 [ 851] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                              2048
510.834917 [ 851] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                              2048
510.845526 [ 851] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                              2048
carp: 3@igb2: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 1440 (interface down)
carp: 5@igb2: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 1680 (interface down)
carp: 8@igb2: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 1920 (interface down)
carp: 9@igb2: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 2160 (interface down)
igb2: link state changed to DOWN
511.135701 [ 851] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                              2048
511.411583 [ 851] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                              2048
511.421996 [ 851] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                              2048
igb0: link state changed to DOWN
carp: 1@igb0_vlan168: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 2400 (interface down)
igb0_vlan168: link state changed to DOWN
carp: 10@igb0_vlan250: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 2640 (interface down)
igb0_vlan250: link state changed to DOWN
carp: 13@igb0_vlan20: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 2880 (interface down)
igb0_vlan20: link state changed to DOWN
carp: 11@igb0_vlan144: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 3120 (interface down)
igb0_vlan144: link state changed to DOWN
carp: 12@igb0_vlan999: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 3360 (interface down)
igb0_vlan999: link state changed to DOWN
511.750382 [ 851] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                              2048
carp: 2@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 3120 (interface up)
carp: 4@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 2880 (interface up)
carp: 6@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 2640 (interface up)
carp: 7@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 2400 (interface up)
igb1: link state changed to UP
carp: 3@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 2160 (interface up)
carp: 5@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1920 (interface up)
carp: 8@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1680 (interface up)
carp: 9@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1440 (interface up)
igb2: link state changed to UP
igb0: link state changed to UP
carp: 1@igb0_vlan168: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1200 (interface up)
igb0_vlan168: link state changed to UP
carp: 10@igb0_vlan250: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 960 (interface up)
igb0_vlan250: link state changed to UP
carp: 13@igb0_vlan20: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 720 (interface up)
igb0_vlan20: link state changed to UP
carp: 11@igb0_vlan144: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 480 (interface up)
igb0_vlan144: link state changed to UP
carp: 12@igb0_vlan999: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 240 (interface up)
igb0_vlan999: link state changed to UP
carp: 2@igb1: BACKUP -> MASTER (master timed out)
carp: 4@igb1: BACKUP -> MASTER (master timed out)
carp: 6@igb1: BACKUP -> MASTER (master timed out)
carp: 7@igb1: BACKUP -> MASTER (master timed out)
arprequest_internal: cannot find matching address
carp: 3@igb2: BACKUP -> MASTER (master timed out)
carp: 5@igb2: BACKUP -> MASTER (master timed out)
carp: 8@igb2: BACKUP -> MASTER (master timed out)
carp: 9@igb2: BACKUP -> MASTER (master timed out)
carp: 1@igb0_vlan168: BACKUP -> MASTER (master timed out)
carp: 10@igb0_vlan250: BACKUP -> MASTER (master timed out)
carp: 13@igb0_vlan20: BACKUP -> MASTER (master timed out)
carp: 11@igb0_vlan144: BACKUP -> MASTER (master timed out)
carp: 12@igb0_vlan999: BACKUP -> MASTER (master timed out)
carp: 2@igb1: MASTER -> INIT (hardware interface up)
carp: 4@igb1: MASTER -> INIT (hardware interface up)
carp: 6@igb1: MASTER -> INIT (hardware interface up)
carp: 7@igb1: MASTER -> INIT (hardware interface up)
igb1: promiscuous mode disabled
igb1: promiscuous mode enabled
carp: 2@igb1: INIT -> BACKUP (initialization complete)
carp: 4@igb1: INIT -> BACKUP (initialization complete)
carp: 6@igb1: INIT -> BACKUP (initialization complete)
carp: 7@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 2@igb1: BACKUP -> MASTER (master timed out)
carp: 4@igb1: BACKUP -> MASTER (master timed out)
carp: 6@igb1: BACKUP -> MASTER (master timed out)
carp: 7@igb1: BACKUP -> MASTER (master timed out)
carp: 3@igb2: MASTER -> INIT (hardware interface up)
carp: 5@igb2: MASTER -> INIT (hardware interface up)
carp: 8@igb2: MASTER -> INIT (hardware interface up)
carp: 9@igb2: MASTER -> INIT (hardware interface up)
igb2: promiscuous mode disabled
igb2: promiscuous mode enabled
carp: 3@igb2: INIT -> BACKUP (initialization complete)
carp: 5@igb2: INIT -> BACKUP (initialization complete)
carp: 8@igb2: INIT -> BACKUP (initialization complete)
carp: 9@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 3@igb2: BACKUP -> MASTER (master timed out)
carp: 5@igb2: BACKUP -> MASTER (master timed out)
carp: 8@igb2: BACKUP -> MASTER (master timed out)
carp: 9@igb2: BACKUP -> MASTER (master timed out)
carp: 1@igb0_vlan168: MASTER -> INIT (hardware interface up)
igb0_vlan168: promiscuous mode disabled
igb0_vlan168: promiscuous mode enabled
carp: 1@igb0_vlan168: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 10@igb0_vlan250: MASTER -> INIT (hardware interface up)
igb0_vlan250: promiscuous mode disabled
igb0_vlan250: promiscuous mode enabled
carp: 10@igb0_vlan250: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: 1@igb0_vlan168: BACKUP -> MASTER (master timed out)
carp: 13@igb0_vlan20: MASTER -> INIT (hardware interface up)
igb0_vlan20: promiscuous mode disabled
igb0_vlan20: promiscuous mode enabled
carp: 13@igb0_vlan20: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 10@igb0_vlan250: BACKUP -> MASTER (master timed out)
carp: 11@igb0_vlan144: MASTER -> INIT (hardware interface up)
igb0_vlan144: promiscuous mode disabled
igb0_vlan144: promiscuous mode enabled
carp: 11@igb0_vlan144: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 13@igb0_vlan20: BACKUP -> MASTER (master timed out)
carp: 7@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 6@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 12@igb0_vlan999: MASTER -> INIT (hardware interface up)
igb0_vlan999: promiscuous mode disabled
igb0_vlan999: promiscuous mode enabled
carp: 12@igb0_vlan999: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: 2@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 3@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 4@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 11@igb0_vlan144: BACKUP -> MASTER (master timed out)
carp: 5@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 9@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 8@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 12@igb0_vlan999: BACKUP -> MASTER (master timed out)
carp: 12@igb0_vlan999: MASTER -> BACKUP (more frequent advertisement received)
carp: 10@igb0_vlan250: MASTER -> BACKUP (more frequent advertisement received)
carp: 11@igb0_vlan144: MASTER -> BACKUP (more frequent advertisement received)
carp: 13@igb0_vlan20: MASTER -> BACKUP (more frequent advertisement received)
carp: 1@igb0_vlan168: MASTER -> BACKUP (more frequent advertisement received)
carp: demoted by -240 to 0 (sysctl)
carp: 12@igb0_vlan999: BACKUP -> MASTER (preempting a slower master)
carp: 5@igb2: BACKUP -> MASTER (preempting a slower master)
carp: 9@igb2: BACKUP -> MASTER (preempting a slower master)
carp: 8@igb2: BACKUP -> MASTER (preempting a slower master)
carp: 6@igb1: BACKUP -> MASTER (preempting a slower master)
carp: 7@igb1: BACKUP -> MASTER (preempting a slower master)
carp: 11@igb0_vlan144: BACKUP -> MASTER (preempting a slower master)
carp: 1@igb0_vlan168: BACKUP -> MASTER (preempting a slower master)
carp: 13@igb0_vlan20: BACKUP -> MASTER (preempting a slower master)
carp: 10@igb0_vlan250: BACKUP -> MASTER (preempting a slower master)
carp: 4@igb1: BACKUP -> MASTER (preempting a slower master)
carp: 3@igb2: BACKUP -> MASTER (preempting a slower master)
carp: 2@igb1: BACKUP -> MASTER (preempting a slower master)
ax0: xgbe_phy_sfp_signals: port_sfp_inputs: 0x2
ax0: Link is DOWN
ax0: link state changed to DOWN
ax0: xgbe_phy_sfp_signals: port_sfp_inputs: 0x0
ax0: Link is UP - 10Gbps/Full - flow control off
ax0: link state changed to UP
ax0: xgbe_phy_sfp_signals: port_sfp_inputs: 0x2
ax0: Link is DOWN
ax0: link state changed to DOWN
ax0: xgbe_phy_sfp_signals: port_sfp_inputs: 0x0
ax0: Link is UP - 10Gbps/Full - flow control off
ax0: link state changed to UP
ax0: xgbe_phy_sfp_signals: port_sfp_inputs: 0x2
ax0: Link is DOWN
ax0: link state changed to DOWN
ax0: xgbe_phy_sfp_signals: port_sfp_inputs: 0x0
ax0: Link is UP - 10Gbps/Full - flow control off
ax0: link state changed to UP


[mimugmail]

You are running Suricata or Zenarmor and it resets the interface which is causing a demote. Disable both and Lookif its stable to rule out the error

[mmaxwell]

I have not had this issue for several months.  After upgrading to 24.7.X it started happening again.  We are running Suricata and would preferably not want to disable it. 

Code: [Select]

igb2: link state changed to DOWN
igb2: promiscuous mode enabled
carp: demoted by 240 to 1920 (interface down)
carp: demoted by 240 to 2160 (interface down)
carp: demoted by 240 to 2400 (interface down)
carp: demoted by 240 to 2640 (interface down)
carp: demoted by 240 to 2880 (interface down)
tun1: changing name to 'ovpns1'
carp: demoted by -240 to 2640 (pfsync bulk done)
carp: 10@igb0_vlan250: BACKUP -> MASTER (master timed out)
carp: 12@igb0_vlan999: BACKUP -> MASTER (master timed out)
carp: 1@igb0_vlan168: BACKUP -> MASTER (master timed out)
carp: 2@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 2400 (interface up)
carp: 4@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 2160 (interface up)
carp: 6@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1920 (interface up)
carp: 7@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1680 (interface up)
carp: 16@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1440 (interface up)
igb1: link state changed to UP
carp: 3@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1200 (interface up)
carp: 5@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 960 (interface up)
carp: 8@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 720 (interface up)
carp: 9@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 480 (interface up)
carp: 17@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 240 (interface up)
igb2: link state changed to UP
carp: 16@igb1: BACKUP -> MASTER (master timed out)
carp: 7@igb1: BACKUP -> MASTER (master timed out)
carp: 6@igb1: BACKUP -> MASTER (master timed out)
carp: 4@igb1: BACKUP -> MASTER (master timed out)
carp: 2@igb1: BACKUP -> MASTER (master timed out)
carp: 17@igb2: BACKUP -> MASTER (master timed out)
carp: 9@igb2: BACKUP -> MASTER (master timed out)
carp: 8@igb2: BACKUP -> MASTER (master timed out)
carp: 5@igb2: BACKUP -> MASTER (master timed out)
carp: 3@igb2: BACKUP -> MASTER (master timed out)
carp: 14@igb0_vlan90: MASTER -> BACKUP (more frequent advertisement received)
carp: 13@igb0_vlan20: MASTER -> BACKUP (more frequent advertisement received)
carp: 15@igb0_vlan91: MASTER -> BACKUP (more frequent advertisement received)
carp: 1@igb0_vlan168: MASTER -> BACKUP (more frequent advertisement received)
carp: 11@igb0_vlan144: MASTER -> BACKUP (more frequent advertisement received)
carp: 10@igb0_vlan250: MASTER -> BACKUP (more frequent advertisement received)
carp: 12@igb0_vlan999: MASTER -> BACKUP (more frequent advertisement received)
carp: 16@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 7@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 4@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 2@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 6@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 17@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 9@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 3@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 5@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 8@igb2: MASTER -> BACKUP (more frequent advertisement received)
389.341838 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
389.350139 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
389.358574 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
389.430896 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
carp: 2@igb1: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 480 (interface down)
carp: 4@igb1: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 720 (interface down)
carp: 6@igb1: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 960 (interface down)
carp: 7@igb1: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 1200 (interface down)
carp: 16@igb1: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 1440 (interface down)
igb1: link state changed to DOWN
390.547490 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
390.555772 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
390.564231 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
391.831272 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
391.839587 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
391.847998 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
391.858452 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
carp: 3@igb2: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 1680 (interface down)
carp: 5@igb2: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 1920 (interface down)
carp: 8@igb2: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 2160 (interface down)
carp: 9@igb2: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 2400 (interface down)
carp: 17@igb2: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 2640 (interface down)
igb2: link state changed to DOWN
393.010303 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
393.018581 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
393.027020 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
394.246335 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
394.254629 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
394.263043 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
394.273549 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
igb0: link state changed to DOWN
carp: 1@igb0_vlan168: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 2880 (interface down)
igb0_vlan168: link state changed to DOWN
carp: 10@igb0_vlan250: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 3120 (interface down)
igb0_vlan250: link state changed to DOWN
carp: 13@igb0_vlan20: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 3360 (interface down)
igb0_vlan20: link state changed to DOWN
carp: 11@igb0_vlan144: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 3600 (interface down)
igb0_vlan144: link state changed to DOWN
carp: 12@igb0_vlan999: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 3840 (interface down)
igb0_vlan999: link state changed to DOWN
carp: 15@igb0_vlan91: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 4080 (interface down)
igb0_vlan91: link state changed to DOWN
395.454663 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
395.462976 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
395.471367 [ 850] iflib_netmap_config       txr 4 rxr 4 txd 1024 rxd 1024 rbufsz                                                                                                                                                                                                                                              2048
carp: 14@igb0_vlan90: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 4320 (interface down)
igb0_vlan90: link state changed to DOWN
carp: 2@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 4080 (interface up)
carp: 4@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 3840 (interface up)
carp: 6@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 3600 (interface up)
carp: 7@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 3360 (interface up)
carp: 16@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 3120 (interface up)
igb1: link state changed to UP
carp: 3@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 2880 (interface up)
carp: 5@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 2640 (interface up)
carp: 8@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 2400 (interface up)
carp: 9@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 2160 (interface up)
carp: 17@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1920 (interface up)
igb2: link state changed to UP
carp: 16@igb1: BACKUP -> MASTER (master timed out)
carp: 7@igb1: BACKUP -> MASTER (master timed out)
carp: 6@igb1: BACKUP -> MASTER (master timed out)
carp: 4@igb1: BACKUP -> MASTER (master timed out)
carp: 2@igb1: BACKUP -> MASTER (master timed out)
igb0: link state changed to UP
carp: 1@igb0_vlan168: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1680 (interface up)
igb0_vlan168: link state changed to UP
carp: 10@igb0_vlan250: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1440 (interface up)
igb0_vlan250: link state changed to UP
carp: 13@igb0_vlan20: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 1200 (interface up)
igb0_vlan20: link state changed to UP
carp: 11@igb0_vlan144: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 960 (interface up)
igb0_vlan144: link state changed to UP
carp: 12@igb0_vlan999: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 720 (interface up)
igb0_vlan999: link state changed to UP
carp: 15@igb0_vlan91: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 480 (interface up)
igb0_vlan91: link state changed to UP
carp: 14@igb0_vlan90: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 240 (interface up)
igb0_vlan90: link state changed to UP
carp: 17@igb2: BACKUP -> MASTER (master timed out)
carp: 9@igb2: BACKUP -> MASTER (master timed out)
carp: 8@igb2: BACKUP -> MASTER (master timed out)
carp: 5@igb2: BACKUP -> MASTER (master timed out)
carp: 3@igb2: BACKUP -> MASTER (master timed out)
carp: 1@igb0_vlan168: BACKUP -> MASTER (master timed out)
carp: 13@igb0_vlan20: BACKUP -> MASTER (master timed out)
carp: 10@igb0_vlan250: BACKUP -> MASTER (master timed out)
carp: 12@igb0_vlan999: BACKUP -> MASTER (master timed out)
carp: 11@igb0_vlan144: BACKUP -> MASTER (master timed out)
carp: 14@igb0_vlan90: BACKUP -> MASTER (master timed out)
carp: 15@igb0_vlan91: BACKUP -> MASTER (master timed out)
carp: 2@igb1: MASTER -> INIT (hardware interface up)
carp: 4@igb1: MASTER -> INIT (hardware interface up)
carp: 6@igb1: MASTER -> INIT (hardware interface up)
carp: 7@igb1: MASTER -> INIT (hardware interface up)
carp: 16@igb1: MASTER -> INIT (hardware interface up)
igb1: promiscuous mode disabled
igb1: promiscuous mode enabled
carp: 2@igb1: INIT -> BACKUP (initialization complete)
carp: 4@igb1: INIT -> BACKUP (initialization complete)
carp: 6@igb1: INIT -> BACKUP (initialization complete)
carp: 7@igb1: INIT -> BACKUP (initialization complete)
carp: 16@igb1: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 2@igb1: BACKUP -> MASTER (master timed out)
carp: 4@igb1: BACKUP -> MASTER (master timed out)
carp: 6@igb1: BACKUP -> MASTER (master timed out)
carp: 7@igb1: BACKUP -> MASTER (master timed out)
carp: 16@igb1: BACKUP -> MASTER (master timed out)
carp: 3@igb2: MASTER -> INIT (hardware interface up)
carp: 5@igb2: MASTER -> INIT (hardware interface up)
carp: 8@igb2: MASTER -> INIT (hardware interface up)
carp: 9@igb2: MASTER -> INIT (hardware interface up)
carp: 17@igb2: MASTER -> INIT (hardware interface up)
igb2: promiscuous mode disabled
igb2: promiscuous mode enabled
carp: 3@igb2: INIT -> BACKUP (initialization complete)
carp: 5@igb2: INIT -> BACKUP (initialization complete)
carp: 8@igb2: INIT -> BACKUP (initialization complete)
carp: 9@igb2: INIT -> BACKUP (initialization complete)
carp: 17@igb2: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: 3@igb2: BACKUP -> MASTER (master timed out)
carp: 5@igb2: BACKUP -> MASTER (master timed out)
carp: 8@igb2: BACKUP -> MASTER (master timed out)
carp: 9@igb2: BACKUP -> MASTER (master timed out)
carp: 17@igb2: BACKUP -> MASTER (master timed out)
carp: 1@igb0_vlan168: MASTER -> INIT (hardware interface up)
igb0_vlan168: promiscuous mode disabled
igb0_vlan168: promiscuous mode enabled
carp: 1@igb0_vlan168: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 10@igb0_vlan250: MASTER -> INIT (hardware interface up)
igb0_vlan250: promiscuous mode disabled
igb0_vlan250: promiscuous mode enabled
carp: 10@igb0_vlan250: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 1@igb0_vlan168: BACKUP -> MASTER (master timed out)
carp: 13@igb0_vlan20: MASTER -> INIT (hardware interface up)
igb0_vlan20: promiscuous mode disabled
igb0_vlan20: promiscuous mode enabled
carp: 13@igb0_vlan20: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 10@igb0_vlan250: BACKUP -> MASTER (master timed out)
carp: 11@igb0_vlan144: MASTER -> INIT (hardware interface up)
igb0_vlan144: promiscuous mode disabled
igb0_vlan144: promiscuous mode enabled
carp: 11@igb0_vlan144: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 13@igb0_vlan20: BACKUP -> MASTER (master timed out)
carp: 12@igb0_vlan999: MASTER -> INIT (hardware interface up)
igb0_vlan999: promiscuous mode disabled
igb0_vlan999: promiscuous mode enabled
carp: 12@igb0_vlan999: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: 11@igb0_vlan144: BACKUP -> MASTER (master timed out)
carp: 15@igb0_vlan91: MASTER -> INIT (hardware interface up)
igb0_vlan91: promiscuous mode disabled
igb0_vlan91: promiscuous mode enabled
carp: 15@igb0_vlan91: INIT -> BACKUP (initialization complete)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 12@igb0_vlan999: BACKUP -> MASTER (master timed out)
carp: 16@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 7@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 4@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 2@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 6@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 14@igb0_vlan90: MASTER -> INIT (hardware interface up)
igb0_vlan90: promiscuous mode disabled
igb0_vlan90: promiscuous mode enabled
carp: 14@igb0_vlan90: INIT -> BACKUP (initialization complete)
carp: demoted by 240 to 480 (pfsync bulk start)
carp: demoted by -240 to 240 (pfsync bulk done)
carp: 15@igb0_vlan91: BACKUP -> MASTER (master timed out)
carp: 17@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 8@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 5@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 3@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 9@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 14@igb0_vlan90: BACKUP -> MASTER (master timed out)
carp: 12@igb0_vlan999: MASTER -> BACKUP (more frequent advertisement received)
carp: 14@igb0_vlan90: MASTER -> BACKUP (more frequent advertisement received)
carp: 13@igb0_vlan20: MASTER -> BACKUP (more frequent advertisement received)
carp: 11@igb0_vlan144: MASTER -> BACKUP (more frequent advertisement received)
carp: 15@igb0_vlan91: MASTER -> BACKUP (more frequent advertisement received)
carp: 10@igb0_vlan250: MASTER -> BACKUP (more frequent advertisement received)
carp: 1@igb0_vlan168: MASTER -> BACKUP (more frequent advertisement received)
carp: demoted by -240 to 0 (sysctl)
carp: 14@igb0_vlan90: BACKUP -> MASTER (preempting a slower master)
carp: 16@igb1: BACKUP -> MASTER (preempting a slower master)
carp: 12@igb0_vlan999: BACKUP -> MASTER (preempting a slower master)
carp: 13@igb0_vlan20: BACKUP -> MASTER (preempting a slower master)
carp: 7@igb1: BACKUP -> MASTER (preempting a slower master)
carp: 17@igb2: BACKUP -> MASTER (preempting a slower master)
carp: 6@igb1: BACKUP -> MASTER (preempting a slower master)
carp: 11@igb0_vlan144: BACKUP -> MASTER (preempting a slower master)
carp: 1@igb0_vlan168: BACKUP -> MASTER (preempting a slower master)
carp: 10@igb0_vlan250: BACKUP -> MASTER (preempting a slower master)
carp: 4@igb1: BACKUP -> MASTER (preempting a slower master)
carp: 5@igb2: BACKUP -> MASTER (preempting a slower master)
carp: 15@igb0_vlan91: BACKUP -> MASTER (preempting a slower master)
carp: 2@igb1: BACKUP -> MASTER (preempting a slower master)
carp: 3@igb2: BACKUP -> MASTER (preempting a slower master)
carp: 9@igb2: BACKUP -> MASTER (preempting a slower master)
carp: 8@igb2: BACKUP -> MASTER (preempting a slower master)
carp: demoted by 240 to 240 (send error 55 on igb0_vlan250)
carp: demoted by 240 to 480 (send error 55 on igb0_vlan90)
carp: demoted by 240 to 720 (send error 55 on igb0_vlan91)
carp: demoted by 240 to 960 (send error 55 on igb0_vlan168)
carp: demoted by 240 to 1200 (send error 55 on igb0_vlan20)
carp: 2@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 17@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 7@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 9@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 16@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 8@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 5@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 3@igb2: MASTER -> BACKUP (more frequent advertisement received)
carp: 6@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 4@igb1: MASTER -> BACKUP (more frequent advertisement received)
carp: 10@igb0_vlan250: MASTER -> BACKUP (more frequent advertisement received)
carp: 13@igb0_vlan20: MASTER -> BACKUP (more frequent advertisement received)
carp: 14@igb0_vlan90: MASTER -> BACKUP (more frequent advertisement received)
carp: 1@igb0_vlan168: MASTER -> BACKUP (more frequent advertisement received)
carp: 15@igb0_vlan91: MASTER -> BACKUP (more frequent advertisement received)
carp: 11@igb0_vlan144: MASTER -> BACKUP (more frequent advertisement received)
carp: 12@igb0_vlan999: MASTER -> BACKUP (more frequent advertisement received)