CARP Maintenance mode doesn't do anything?

[loganx1121]

2 Opnsense firewalls with VIPs configured for the WAN interfaces and the backend VLANS. All VLANs live on the firewall and I'm trunking over a LAGG from a cisco switch. I can't seem to get CARP maintenance mode to do anything. I see the CARP demotion level increase to 240, but the primary firewall still shows as the master.

I was checking the traffic logs and I saw the IGMP traffic from the VLAN interface being blocked, so I made a rule to allow that but that didn't seem to help. I also globally disabled IGMP snooping on the cisco switch, but that had no effect either.

Just wondering if anyone else has run into this. If I disable CARP it seems to failover fine, and I've already tried recreating all the virtual IP's.  I also confirmed I can ping from the VLAN interface of 1 firewall to the same VLAN on the other firewall, so it doesn't seem to be a connectivity problem. 

[mimugmail]

You should test with a different switch. If you enable mnt and its still Master, node 2 doesnt get the packets or are filtered

[loganx1121]

You should test with a different switch. If you enable mnt and its still Master, node 2 doesnt get the packets or are filtered

Sorry?  Enable mnt?

[Patrick M. Hausen]

"maintenance"

[loganx1121]

"maintenance"

Ah.  Unfortunately I only have the one switch to use.  Well, it's 2 in a stack.  I did globally disable igmp snooping though, which according to cisco documentation here:

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/68131-cat-multicast-prob.html

Code: [Select]

If you disable IGMP snooping, all switches treat multicast traffic as a broadcast traffic. This floods the traffic to all the ports in that VLAN, regardless of whether the ports have interested receivers for that multicast stream.
So by disabling, it should get where it needs to go, but the behavior is still the same.

[loganx1121]

I deleted all of the virtual IP’s and configured just 1 for a single VLAN.  Here is a packet capture from that VLAN interface on the secondary firewall.  After I started the capture I tried to enable maintenance mode on the primary

Code: [Select]

VLAN_10_Active_Directory
vlan01 09:31:25.810669 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    10.110.10.2 > 224.0.0.18: vrrp 10.110.10.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 160.210.4.23,235.105.4.180,113.194.71.99,176.7.125.190,245.170.7.212,120.225.232.142,10.65.27.171
    10.110.10.2 > 224.0.0.18: vrrp 10.110.10.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 13.99.217.19,131.170.210.220,10.55.161.162,117.164.121.126,229.17.234.117,101.70.133.79,12.136.252.26
    10.110.10.2 > 224.0.0.18: vrrp 10.110.10.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 240, authtype none, intvl 1s, length 36, addrs(7): 158.236.0.177,106.209.71.251,204.211.6.131,53.175.7.127,178.250.195.244,6.252.140.171,36.40.171.150
    10.110.10.2 > 224.0.0.18: vrrp 10.110.10.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 240, authtype none, intvl 1s, length 36, addrs(7): 227.229.191.192,21.166.15.213,19.133.21.216,251.37.38.71,65.1.102.92,83.36.20.109,142.201.180.76
    10.110.10.2 > 224.0.0.18: vrrp 10.110.10.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 240, authtype none, intvl 1s, length 36, addrs(7): 59.210.190.198,13.88.70.116,219.49.97.178,37.201.88.145,70.7.92.84,238.130.171.236,243.47.41.238
    10.110.10.2 > 224.0.0.18: vrrp 10.110.10.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 108.180.155.137,155.144.229.212,9.124.122.114,79.49.195.250,125.226.65.230,24.228.173.183,26.75.230.227
    10.110.10.2 > 224.0.0.18: vrrp 10.110.10.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 137.44.149.172,67.170.162.10,248.22.34.249,125.82.231.76,60.179.78.110,235.132.159.181,71.132.92.170
    10.110.10.2 > 224.0.0.18: vrrp 10.110.10.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 254.102.126.151,229.62.112.146,227.23.161.59,120.249.23.183,196.151.35.87,50.189.145.49,60.92.216.168
VLAN_10_Active_Directory
vlan01 09:31:26.873849 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
VLAN_10_Active_Directory
vlan01 09:31:26.938052 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
VLAN_10_Active_Directory
vlan01 09:31:28.892743 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
VLAN_10_Active_Directory
vlan01 09:31:30.926398 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
VLAN_10_Active_Directory
vlan01 09:31:32.049355 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
VLAN_10_Active_Directory
vlan01 09:31:33.064451 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
VLAN_10_Active_Directory
vlan01 09:31:34.124511 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)


[loganx1121]

I find it interesting that I don't see any new traffic on either firewall when I enter maintenance mode on the primary. 

[loganx1121]

If I manually change the skew of the virtual IP on the primary to something above 100 (which is what the secondary is set to) then the primary becomes the backup.  I also found this github post that seems relevant but it looks like this was already patched back in 2019. https://github.com/opnsense/core/issues/3671

I recreated the same topology in GNS3 and I'm seeing the same issue there.  GNS3 is using version 22.1.2_2 and my "real" version is 22.10

[mimugmail]

Is there a second Cluster un this network?

[loganx1121]

Is there a second Cluster un this network?

Just 2 firewalls in a single HA pair.  I actually resolved this by disabling pre-emption on the secondary.