Get DUP! when I ping VIP address

[ednt]

We have several interfaces in CARP.
If we ping the VIP address from one of them (no VLAN)
I get sporadic a DUP! and I can see that the slave, which is in backup mode, get the request and replies.
Not on every request, only sometimes.
We already cleared the arp cache of the involved switches.

As you can see, both opnsense send a reply:

Code Select Expand

17:19:32.363137 1c:c1:de:06:46:a2 > a4:bf:01:16:b1:db, ethertype IPv4 (0x0800), length 98: 192.168.254.248 > 192.168.254.167: ICMP echo reply, id 3595
64
17:19:32.363335 1c:c1:de:06:38:42 > a4:bf:01:16:b1:db, ethertype IPv4 (0x0800), length 98: 192.168.254.248 > 192.168.254.167: ICMP echo reply, id 3595
64

But this happens only on the interface without VLAN and on an other interface.
And only when we ping from a server which has also an interface in the same network.

In dmesg I can not see any change of the CARP mode, so it is not 'flipping'.

We are running out of ideas.

Any other idea?

[meyergru]

Are you mixing tagged and untagged VLANs on the same OpnSense interface by any chance?

[Monviech (Cedrik)]

We have some hints here:

https://docs.opnsense.org/manual/how-tos/carp.html#known-limitations

Last two times I saw dups with pings in customer support was firmware bug in a switch that caused the CAM table to misbehave, and the other time it was switches that were not stacked (even though they should have been)

Most of the time its the switch being weird.

Specifically read this:
https://docs.opnsense.org/manual/how-tos/carp.html#stacking

[ednt]

This interface is on a dedicated 1GB port. No other stuff (VLANs) are involved.

Both opnsense are connected to the same (none stacked) switch.
As already written: clearing the mac address cache changed nothing.

It is very strange.
Even if the packet receives the slave, since the interface tells us 'backup', why is it sending a reply?

[Monviech (Cedrik)]

It sounds like your switch does not know where to find the virtual CARP MAC address, so it falls back to "Unicast Flooding" and duplicates your packets to all ports.

Both nodes have the VIP configured, but only the MASTER should be attracting traffic for the CARP virtual MAC in the first place. If the switch floods or missdelivers the unicast frame to the BACKUP anyway, the BACKUP may still see an IP packet addressed to a locally configured VIP and answer it.

[ednt]

Hm ...

as written: the DUP! happens only when I ping from a server which is in the same net:

Code Select Expand

64 bytes from 192.168.254.248: icmp_seq=537 ttl=64 time=0,400 ms
64 bytes from 192.168.254.248: icmp_seq=538 ttl=64 time=0,338 ms
64 bytes from 192.168.254.248: icmp_seq=539 ttl=64 time=0,286 ms
64 bytes from 192.168.254.248: icmp_seq=540 ttl=64 time=0,202 ms
64 bytes from 192.168.254.248: icmp_seq=541 ttl=64 time=0,261 ms
64 bytes from 192.168.254.248: icmp_seq=541 ttl=64 time=0,350 ms (DUP!)
64 bytes from 192.168.254.248: icmp_seq=542 ttl=64 time=0,275 ms
64 bytes from 192.168.254.248: icmp_seq=542 ttl=64 time=0,328 ms (DUP!)
64 bytes from 192.168.254.248: icmp_seq=543 ttl=64 time=0,389 ms
64 bytes from 192.168.254.248: icmp_seq=544 ttl=64 time=0,421 ms
64 bytes from 192.168.254.248: icmp_seq=544 ttl=64 time=0,477 ms (DUP!)
64 bytes from 192.168.254.248: icmp_seq=545 ttl=64 time=0,305 ms
64 bytes from 192.168.254.248: icmp_seq=546 ttl=64 time=0,319 ms
64 bytes from 192.168.254.248: icmp_seq=547 ttl=64 time=0,395 ms
64 bytes from 192.168.254.248: icmp_seq=548 ttl=64 time=0,319 ms
64 bytes from 192.168.254.248: icmp_seq=548 ttl=64 time=0,373 ms (DUP!)
64 bytes from 192.168.254.248: icmp_seq=549 ttl=64 time=0,267 ms
64 bytes from 192.168.254.248: icmp_seq=550 ttl=64 time=0,336 ms

So it happens not for every ping.

If I ping 248 from my local PC which is not in the same net, it works correct, no DUPs


Master:

Code Select Expand

ifconfig bce3
bce3: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: MANAGEMENT (opt1)
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 1c:c1:de:06:46:a2
        inet 192.168.254.252 netmask 0xfffffc00 broadcast 192.168.255.255
        inet 192.168.254.248 netmask 0xfffffc00 broadcast 192.168.255.255 vhid 16
        groups: CARP_Group
        carp: MASTER vhid 16 advbase 1 advskew 0
              peer 224.0.0.18 peer6 ff02::12
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Slave:

Code Select Expand

ifconfig bce3
bce3: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: MANAGEMENT (opt1)
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 1c:c1:de:06:38:42
        inet 192.168.254.253 netmask 0xfffffc00 broadcast 192.168.255.255
        inet 192.168.254.248 netmask 0xfffffc00 broadcast 192.168.255.255 vhid 16
        groups: CARP_Group
        carp: BACKUP vhid 16 advbase 1 advskew 100
              peer 224.0.0.18 peer6 ff02::12
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


[ednt]

Btw. switching master in maintenance and back works correct.

How can I see which OPNsense owns the 00:00:5E mac address?

[Monviech (Cedrik)]

You have to check the CAM tables of your switch(es).
https://en.wikipedia.org/wiki/Forwarding_information_base


The 00:00:5E MAC address(es) should always point to the port the current MASTER is connected to, inside your switch(es).

[ednt]

Strange that it does not happen always, only sometimes.

I solved the problem now by putting the slave bce3 interface on an other switch.

But how can I see/show the 00:00:5E address on the opnsense?

ifconfig doesn't show it.

[Monviech (Cedrik)]

It's a synthetic address that's not directly displayed via ifconfig.

It should be something like this:

00:00:5e:00:01:<vhid in hex>

tcpdump would be easiest, here an example from my HA peer:

Code Select Expand

root@opn01:~ # tcpdump -eni vlan0.1 proto 112
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vlan0.1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:42:54.727829 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 172.16.0.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36
00:00:5e:00:01:01 -> CARP mac address for vhid 1
01:00:5e:00:00:12 -> Multicast MAC for 224.0.0.18