CARP Problem - HP ProLiant DL360p Gen8

[Sven-J]

Hi,

I wrote already in the german part of this awesome forum: https://forum.opnsense.org/index.php?topic=11687.0

But maybe someone of you have an Idea:

I got the following message all the time:

CARP has detected a problem and this unit has been demoted to BACKUP status.
Check link status on all interfaces with configured CARP VIPs.

Following Setup: 2x DL360p Gen8 as Firewall OPNsense 19.1 installed

LAGG0 / LACP - bge0 / bge1 for WAN (HP Ethernet 1Gb 4-port 331FLR Adapter)
LAGG1 / LACP - bxe0 / bxe1 for all vlans (10GB) (HP Ethernet 10Gb 2-port 530T Adapter)

LAGG0 with a transfernet works as expected. Everything is fine here:

Switch Bridge Configuration: LAGG0 / LACP / WAN

LAGG0: FW-NODE01
<DEHAM01-CORE-01>display interface Bridge-Aggregation 1
Bridge-Aggregation1
Current state: UP
IP packet frame type: Ethernet II, hardware address: 5c8a-3850-2332
Description: LACP-FW-1-INET
Bandwidth: 2000000 kbps
2Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
PVID: 15
Port link-type: Access
 Tagged VLANs:   None
 Untagged VLANs: 15
Last clearing of counters: Never
Last 300 seconds input:  112 packets/sec 91282 bytes/sec 0%
Last 300 seconds output:  113 packets/sec 88820 bytes/sec 0%
Input (total):  69182321 packets, 15868738090 bytes
        68822555 unicasts, 88195 broadcasts, 271571 multicasts, 0 pauses
Input (normal):  69182321 packets, - bytes
        68822555 unicasts, 88195 broadcasts, 271571 multicasts, 0 pauses
Input:  0 input errors, 0 runts, 0 giants, 0 throttles
        0 CRC, 0 frame, - overruns, 0 aborts
        - ignored, - parity errors
Output (total): 683819046 packets, 988166268219 bytes
        682716902 unicasts, 224279 broadcasts, 877865 multicasts, 0 pauses

LAGG0: FW-NODE02
<DEHAM01-CORE-01>display interface Bridge-Aggregation 3
Bridge-Aggregation3
Current state: UP
IP packet frame type: Ethernet II, hardware address: 5c8a-3850-2334
Description: LACP-FW-2-INET
Bandwidth: 2000000 kbps
2Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
PVID: 15
Port link-type: Access
 Tagged VLANs:   None
 Untagged VLANs: 15
Last clearing of counters: Never
Last 300 seconds input:  1 packets/sec 78 bytes/sec 0%
Last 300 seconds output:  2 packets/sec 232 bytes/sec 0%
Input (total):  1173615 packets, 484490037 bytes
        1051210 unicasts, 87942 broadcasts, 34463 multicasts, 0 pauses
Input (normal):  1173615 packets, - bytes
        1051210 unicasts, 87942 broadcasts, 34463 multicasts, 0 pauses
Input:  0 input errors, 0 runts, 0 giants, 0 throttles
        0 CRC, 0 frame, - overruns, 0 aborts
        - ignored, - parity errors
Output (total): 3160504 packets, 1886456666 bytes
        1821496 unicasts, 224600 broadcasts, 1114408 multicasts, 0 pauses
Output (normal): 3160504 packets, - bytes
        1821496 unicasts, 224600 broadcasts, 1114408 multicasts, 0 pauses
Output: 0 output errors, - underruns, - buffer failures
        0 aborts, 0 deferred, 0 collisions, 0 late collisions
        0 lost carrier, - no carrier

Switch Bridge Configuration: LAGG1 / LACP / VLANS

LAGG1: FW-NODE1
<DEHAM01-CORE-01>display interface Bridge-Aggregation 2
Bridge-Aggregation2
Current state: UP
IP packet frame type: Ethernet II, hardware address: 5c8a-3850-2333
Description: LACP-FW-1-trunk
Bandwidth: 20000000 kbps
20Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
PVID: 1
Port link-type: Trunk
 VLAN Passing:   1(default vlan), 10, 40, 42-44, 47, 150, 500-506, 547, 551-552, 1000-1003, 1011, 1020, 1150, 4000-4001
 VLAN permitted: 1(default vlan), 10, 40-4094
 Trunk port encapsulation: IEEE 802.1q
Last clearing of counters: Never
Last 300 seconds input:  23 packets/sec 4297 bytes/sec 0%
Last 300 seconds output:  19 packets/sec 6576 bytes/sec 0%
Input (total):  1305255099 packets, 1730191247711 bytes
        1302081181 unicasts, 7510 broadcasts, 3166408 multicasts, 0 pauses
Input (normal):  1305255099 packets, - bytes
        1302081181 unicasts, 7510 broadcasts, 3166408 multicasts, 0 pauses
Input:  0 input errors, 0 runts, 0 giants, 0 throttles
        0 CRC, 0 frame, - overruns, 0 aborts
        - ignored, - parity errors
Output (total): 690755112 packets, 753149602631 bytes
        688259838 unicasts, 154091 broadcasts, 2341183 multicasts, 0 pauses
Output (normal): 690755112 packets, - bytes
        688259838 unicasts, 154091 broadcasts, 2341183 multicasts, 0 pauses
Output: 0 output errors, - underruns, - buffer failures
        0 aborts, 0 deferred, 0 collisions, 0 late collisions
        0 lost carrier, - no carrier

LAGG1: FW-NODE2
<DEHAM01-CORE-01>display interface Bridge-Aggregation 4
Bridge-Aggregation4
Current state: UP
IP packet frame type: Ethernet II, hardware address: 5c8a-3850-2335
Description: LACP-FW-2-trunk
Bandwidth: 20000000 kbps
20Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
PVID: 3
Port link-type: Trunk
 VLAN Passing:   3, 10, 40, 42-44, 47, 150, 500-506, 547, 551-552, 1000-1003, 1011, 1020, 1150, 4000-4001
 VLAN permitted: 3, 10, 40-4094
 Trunk port encapsulation: IEEE 802.1q
Last clearing of counters: Never
Last 300 seconds input:  0 packets/sec 126 bytes/sec 0%
Last 300 seconds output:  12 packets/sec 2172 bytes/sec 0%
Input (total):  2361306 packets, 2234309304 bytes
        1517489 unicasts, 1323 broadcasts, 842494 multicasts, 0 pauses
Input (normal):  2361306 packets, - bytes
        1517489 unicasts, 1323 broadcasts, 842494 multicasts, 0 pauses
Input:  0 input errors, 0 runts, 0 giants, 0 throttles
        0 CRC, 0 frame, - overruns, 0 aborts
        - ignored, - parity errors
Output (total): 5115311 packets, 1190861340 bytes
        776382 unicasts, 141679 broadcasts, 4197250 multicasts, 0 pauses
Output (normal): 5115311 packets, - bytes
        776382 unicasts, 141679 broadcasts, 4197250 multicasts, 0 pauses
Output: 0 output errors, - underruns, - buffer failures
        0 aborts, 0 deferred, 0 collisions, 0 late collisions
        0 lost carrier, - no carrier

TCPDUMP LAGG0: FW-NODE1
root@DEHAM01-FW01:~ # tcpdump -i lagg0 -ttt -n proto CARP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
 00:00:00.000000 IP xxx.xxx.142.179 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 240, authtype none, intvl 1s, length 36
 00:00:00.000023 IP xxx.xxx.142.179 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 240, authtype none, intvl 1s, length 36
 00:00:02.009931 IP xxx.xxx.142.179 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 240, authtype none, intvl 1s, length 36
 00:00:00.000022 IP xxx.xxx.142.179 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 240, authtype none, intvl 1s, length 36

TCPDUMP LAGG0: FW-NODE2
root@DEHAM01-FW02:~ # tcpdump -i lagg0 -ttt -n proto CARP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
 00:00:00.000000 IP xxx.xxx.142.179 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 240, authtype none, intvl 1s, length 36
 00:00:02.008065 IP xxx.xxx.142.179 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 240, authtype none, intvl 1s, length 36
 00:00:02.011843 IP xxx.xxx.142.179 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 240, authtype none, intvl 1s, length 36

TCPDUMP LAGG1: FW-NODE1
root@DEHAM01-FW01:~ # tcpdump -i lagg1_vlan10 -ttt -n proto CARP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lagg1_vlan10, link-type EN10MB (Ethernet), capture size 262144 bytes
 00:00:00.000000 IP 10.100.10.251 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 240, authtype none, intvl 1s, length 36
 00:00:01.941046 IP 10.100.10.251 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 240, authtype none, intvl 1s, length 36
 00:00:01.944357 IP 10.100.10.251 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 240, authtype none, intvl 1s, length 36
 00:00:02.019615 IP 10.100.10.251 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 240, authtype none, intvl 1s, length 36

TCPDUMP LAGG1: FW-NODE2:
root@DEHAM01-FW02:~ # tcpdump -i lagg1_vlan10 -ttt -n proto CARP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lagg1_vlan10, link-type EN10MB (Ethernet), capture size 262144 bytes
 00:00:00.000000 IP 10.100.10.251 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 240, authtype none, intvl 1s, length 36
 00:00:01.942515 IP 10.100.10.251 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 240, authtype none, intvl 1s, length 36
 00:00:01.975992 IP 10.100.10.251 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 240, authtype none, intvl 1s, length 36
 00:00:01.956651 IP 10.100.10.251 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 240, authtype none, intvl 1s, length 36

Master after reboot:

Master nach einem Reboot:
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 2@lagg1_vlan10: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 4@lagg1_vlan42: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 6@lagg1_vlan44: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 3@lagg1_vlan40: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 5@lagg1_vlan43: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 7@lagg1_vlan150: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 10@lagg1_vlan1002: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 9@lagg1_vlan1001: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 11@lagg1_vlan1003: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 8@lagg1_vlan1000: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 13@lagg1_vlan1020: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 16@lagg1_vlan4001: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 15@lagg1_vlan4000: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 12@lagg1_vlan1011: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 17@lagg1_vlan47: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:00:50 DEHAM01-FW01 kernel: carp: 14@lagg1_vlan1150: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:06:28 DEHAM01-FW01 kernel: carp: 1@lagg0: MASTER -> BACKUP (more frequent advertisement received)
Feb 18 22:06:28 DEHAM01-FW01 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "XXXX.142.178 -  (1@lagg0)" has resumed the state "BACKUP" for vhid 1
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 1@lagg0: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 2@lagg1_vlan10: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "XXX.142.178 -  (1@lagg0)" has resumed the state "BACKUP" for vhid 1
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 3@lagg1_vlan40: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 4@lagg1_vlan42: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 5@lagg1_vlan43: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 6@lagg1_vlan44: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 7@lagg1_vlan150: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 8@lagg1_vlan1000: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 9@lagg1_vlan1001: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 10@lagg1_vlan1002: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 11@lagg1_vlan1003: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 12@lagg1_vlan1011: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 13@lagg1_vlan1020: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 14@lagg1_vlan1150: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 15@lagg1_vlan4000: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 16@lagg1_vlan4001: INIT -> BACKUP (initialization complete)
Feb 18 22:10:01 DEHAM01-FW01 kernel: carp: 17@lagg1_vlan47: INIT -> BACKUP (initialization complete)
Feb 18 22:10:02 DEHAM01-FW01 kernel: carp: demoted by 240 to 240 (pfsync bulk start)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 2@lagg1_vlan10: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 3@lagg1_vlan40: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 4@lagg1_vlan42: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 5@lagg1_vlan43: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 6@lagg1_vlan44: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 7@lagg1_vlan150: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 8@lagg1_vlan1000: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 9@lagg1_vlan1001: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 10@lagg1_vlan1002: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 11@lagg1_vlan1003: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 12@lagg1_vlan1011: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 13@lagg1_vlan1020: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 14@lagg1_vlan1150: BACKUP -> MASTER (master timed out)
Feb 18 22:10:04 DEHAM01-FW01 kernel: carp: 15@lagg1_vlan4000: BACKUP -> MASTER (master timed out)
Feb 18 22:10:05 DEHAM01-FW01 kernel: carp: 16@lagg1_vlan4001: BACKUP -> MASTER (master timed out)
Feb 18 22:10:05 DEHAM01-FW01 kernel: carp: 17@lagg1_vlan47: BACKUP -> MASTER (master timed out)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 480 (send error 50 on lagg1_vlan40)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 720 (send error 50 on lagg1_vlan10)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 960 (send error 50 on lagg1_vlan47)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 1200 (send error 50 on lagg1_vlan4001)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 1440 (send error 50 on lagg1_vlan4000)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 1680 (send error 50 on lagg1_vlan1150)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 1920 (send error 50 on lagg1_vlan1020)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 2160 (send error 50 on lagg1_vlan1011)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 2400 (send error 50 on lagg1_vlan1003)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 2640 (send error 50 on lagg1_vlan1002)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 2880 (send error 50 on lagg1_vlan1001)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 3120 (send error 50 on lagg1_vlan1000)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 3360 (send error 50 on lagg1_vlan150)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 3600 (send error 50 on lagg1_vlan44)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 3840 (send error 50 on lagg1_vlan43)
Feb 18 22:10:07 DEHAM01-FW01 kernel: carp: demoted by 240 to 4080 (send error 50 on lagg1_vlan42)
Feb 18 22:11:07 DEHAM01-FW01 kernel: carp: demoted by -240 to 3840 (pfsync bulk fail)

Maybe someone here have an Idea what happens.

Thank you!

[Sven-J]

No one an Idea?