Unexpected CARP ARP Reply

[modzilla]

Hi there,

I am trying to use Opnsense in OpenStack, which works great with a single instance and no HA. But when I try to add a second instance to use HA with CARP I am never able to use the VIP.

I suspect that the issue is that the arp-reply is always done with the MAC of the Opnsense Node and not with a 00:00:5e... MAC, which will result in a DROP in our OVN SDN. I quickly tested the same with Linux + FRR in containerlab and can see that it does reply with the correct 00:00:5e.. MAC.

Code Select Expand

08:13:33.416505 fa:16:3e:0e:36:1b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 100.64.56.254 tell 100.64.56.227, length 28
08:13:33.416593 fa:16:3e:61:76:50 > fa:16:3e:0e:36:1b, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 100.64.56.254 is-at 00:00:5e:00:01:01, length 28


Code Select Expand

root@ovn-db1:~# ovn-trace --minimal 'inport == "c9d850ee-a534-4a52-8c16-b866d0a3a6d2"
&& arp.op == 2
&& eth.src == 00:00:5e:00:01:01
&& arp.sha == 00:00:5e:00:01:01

&& eth.dst == fa:16:3e:0e:36:1b
&& arp.tha == fa:16:3e:0e:36:1b

&& arp.spa == 100.64.56.254
&& arp.tpa == 100.64.56.227'
# arp,reg14=0x5,vlan_tci=0x0000,dl_src=00:00:5e:00:01:01,dl_dst=fa:16:3e:0e:36:1b,arp_spa=100.64.56.254,arp_tpa=100.64.56.227,arp_op=2,arp_sha=00:00:5e:00:01:01,arp_tha=fa:16:3e:0e:36:1b
output("84886d");
root@ovn-db1:~# ovn-trace --summary 'inport == "c9d850ee-a534-4a52-8c16-b866d0a3a6d2"
&& arp.op == 2
&& eth.src == fa:16:3e:61:76:50
&& arp.sha == 00:00:5e:00:01:01

&& eth.dst == fa:16:3e:0e:36:1b
&& arp.tha == fa:16:3e:0e:36:1b

&& arp.spa == 100.64.56.254
&& arp.tpa == 100.64.56.227'
# arp,reg14=0x5,vlan_tci=0x0000,dl_src=fa:16:3e:61:76:50,dl_dst=fa:16:3e:0e:36:1b,arp_spa=100.64.56.254,arp_tpa=100.64.56.227,arp_op=2,arp_sha=00:00:5e:00:01:01,arp_tha=fa:16:3e:0e:36:1b
ingress(dp="wan", inport="opnsense1-wan") {
    reg0[15] = check_in_port_sec();
    next;
    drop;
};



Is there anything I can do to make it work, is it expected behavior or a bug? I was not able to find any info on CARP beyond the basic man pages and tutorials on setting it up.

Thanks in advance!

[Maurice]

It is expected behaviour. The correct MAC address (00:00:5e:00:01:01) is in the payload of the ARP reply. The source MAC address is irrelevant.

If your infrastructure requires these MAC addresses to match, that's where the problem lies. There have been discussions in the FreeBSD community about making CARP more compatible with broken network equipment for years, but as far as I'm aware there's currently no workaround.

Cheers
Maurice

[modzilla]

Thank you for the quick response!
I already expected as much...

Best regards,
modzilla