Problems with CARP - bug?

[Perun]

Hi

I have problems since 23.x with one CARP interface. I've 4 of them and one starts on the fallback opnsense initiali in BACKUP status and then after 4-5s it changes to MASTER (with active master on the primary opnsense).
3 other CARP interfaces doesnt have this problem.

I dont have changed the configuration, only update from last 22.x to 22.1 and after this to 22.1_6
It is a known bug?

Greetz

[mimugmail]

Different nics? Any logs? Console? HA config screenshot?

[ThomasHamel]

I have the same issue, came with update to 22.7.11.
I have a lagg0 with four interfaces on it. Three VIP's seem to be OK, one is making problems.

Updated to 23.1_6.
The VIP's are swapping frequently. The lagg0_vlan80 is having problems.
But also all other VIP's are effected, no traffic possible for about 10 minutes.

2023-02-06T14:06:43   Notice   configd.py   [fa17137f-adfd-4cd1-9cec-d6f691056785] Carp event on subsystem 12@lagg0_vlan80 for type MASTER   
2023-02-06T14:06:39   Notice   configd.py   [0f4181cb-d43d-4e8c-b076-d2cfac7f3ec7] Carp event on subsystem 12@lagg0_vlan80 for type BACKUP   
2023-02-06T14:06:38   Notice   configd.py   [6c0cda9a-238e-426f-8e04-3c29aa86719d] Carp event on subsystem 12@lagg0_vlan80 for type MASTER   
2023-02-06T14:05:38   Notice   configd.py   [b287b466-1a9b-42e4-bc23-6c04bcd75188] Carp event on subsystem 12@lagg0_vlan80 for type BACKUP   
2023-02-06T13:45:23   Notice   configd.py   [c4eb3166-afec-4386-b8d7-0f129641af6f] Carp event on subsystem 12@lagg0_vlan80 for type MASTER   
2023-02-06T13:45:20   Notice   configd.py   [f9a85c84-eb9a-4a63-9ef2-5e1df97e0327] Carp event on subsystem 12@lagg0_vlan80 for type BACKUP   
2023-02-06T13:39:53   Notice   configd.py   [bc11800c-46f6-49b7-82cc-c846a5a5d695] Carp event on subsystem 12@lagg0_vlan80 for type MASTER   
2023-02-06T13:39:50   Notice   configd.py   [1385fd0c-9709-4de2-bce9-31c80f2a1816] Carp event on subsystem 12@lagg0_vlan80 for type BACKUP   
2023-02-06T13:25:40   Notice   configd.py   [d050812f-a205-40ae-9f66-2c681383f95d] Carp event on subsystem 12@lagg0_vlan80 for type MASTER   
2023-02-06T13:25:36   Notice   configd.py   [9eca6fd6-8085-49ad-827b-c1c8c3544068] Carp event on subsystem 12@lagg0_vlan80 for type BACKUP   
2023-02-06T13:18:17   Notice   configd.py   [bf29afba-6f22-49ef-a55b-823473621420] Carp event on subsystem 12@lagg0_vlan80 for type MASTER   
2023-02-06T13:18:14   Notice   configd.py   [68780d24-4c6f-44e7-8a01-c572ec49bd22] Carp event on subsystem 12@lagg0_vlan80 for type BACKUP   
2023-02-06T13:18:13   Notice   configd.py   [2c36a472-cb02-446f-91a2-97440eb897b1] Carp event on subsystem 12@lagg0_vlan80 for type MASTER   
2023-02-06T13:17:36   Notice   configd.py   [2da78693-92a1-4ad1-ba31-c75082c7692e] Carp event on subsystem 12@lagg0_vlan80 for type BACKUP   
2023-02-06T13:16:43   Notice   configd.py   [a7f6f843-38a5-4709-95db-d4c5d29dda2e] Carp event on subsystem 12@lagg0_vlan80 for type MASTER   
2023-02-06T13:16:40   Notice   configd.py   [c49bbc96-11c6-4828-bebe-f3ff9c5ffc32] Carp event on subsystem 12@lagg0_vlan80 for type BACKUP

[dkanzlemar]

I'm not sure of your complete setup, but I had the same thing happen on my setup this weekend. I have 10 VLANs, and 1 out of the 10 was showing active on both nodes. Even though I had a firewall rule to allow subnet communication in that VLAN, my MASTER and BACKUP nodes couldn't ping each other on that subnet. My issue turned out to be that VLAN wasn't properly defined on my Unifi switch. Once I added the VLAN to my Unifi switch and made sure it was being broadcast to the ports for both routers, then I was able to communicate on that subnet, and my CARP started working correctly for that VLAN. Again, not sure of your entre setup, but I spent a good two hours trying to figure that one out this weekend. Hopefully this helps.

[ThomasHamel]

In my case the two OPNsense boxes were running for more than one year now, connected to the same switch. No issue at all.

Only after the update to 22.7.11 I'm facing the problem. Nothing changed at the switch.

[ThomasHamel]

Hi Perun,
do you see any ARP issues in your OPNsense terminal ?
I have seen a lot them, looks like no ARP resolution can be performed for some minutes, then it is settling down again to normal.

[amw]

There might be a problem....

Same as me. Running a 4 node cluster. After Upgrade from 22.7 only 2 of 25 VLAN's have problems. Changed the VHID and then it works... Change back - same again.

[bubbagump]

I'll ask the obvious dumb question - what does the config of the offending VIPs look like on each box? Are base and skew set correctly and match the "good" VIPs? I know you said the config didn't change, but old configs that worked despite themselves until a bug is fixed is not unheard of.

[ThomasHamel]

Hi bubbagump,

in my case all four VIP's have a different VHID, they match on both boxes.
At the first box I have an advbase of 1 and an advskew of 1 for all VIP's.
At the second box I have an advbase of 1 and an advskew of 101 for all VIP's.

These four VLAN's are on a trunk interface at the OPNsense.
These trunk interfaces from both boxes go to a switch where also trunks are defined with all four VLAN's enabled as tagged.
The communication via the switch is working well.
I also tried a different switch from another vendor. Also there the communication works, but also the same CARP issues.

[mimugmail]

Carp issues are very hard to diagnose in a forum.

You need to check if multicast packets are seen/arriving on thise interfaces.

[ThomasHamel]

Hi mimugmail,

I started a tcpdump at the first box.

I changed the advbase to 2 seconds.

Just spotted now that at 12:58:53 something happened. Instead of the 2-second-interval, the multicast messages are send in about 10 nanosecond intervals.

12:58:31.545214 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:33.554059 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:35.575059 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:37.593496 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:39.601832 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:41.611017 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:43.620565 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:45.664436 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:47.675805 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:49.681860 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:51.692869 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.697689 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.697807 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704257 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704272 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704282 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704294 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704306 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704317 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704329 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704343 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704357 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704374 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704384 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.704399 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36

[ThomasHamel]

That behaviour went on until 12:58:59.

Just counted the multicast messages within 12:58:53: 978 messages sent.

12:58:53.714240 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.714250 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.714259 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:53.714267 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:58:59.738948 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:59:01.743682 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:59:03.761903 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36
12:59:05.772651 IP 172.16.13.253 > 224.0.0.18: VRRPv2, Advertisement, vrid 12, prio 1, authtype none, intvl 2s, length 36

[mimugmail]

Looks like a look, are those virtual Firewalls?

[ThomasHamel]

These are physical boxes, Supermicro 1 HU servers.