CARP arp reply with wrong src mac

[athurdent]

Hi, did a quick search in the forums and on github but did not find an answer, is this FreeBSD/pfSense CARP problem also an issue in OPNsense?

https://redmine.pfsense.org/issues/6957

[franco]

Hi arthurdent,

It would seem so. The patch noted in the ticket hasn't been in OPNsense for a long time, yet on the other hand I haven't heard about this from our users in the forum.

There was some confusion about CARP IP usage for traffic originating from the firewall, but only answering over the CARP IP yields the correct ARP there. One cannot use the CARP IP for originating a connection. Maybe this is related?


Cheers,
Franco

[athurdent]

Hi Franco,

this problem is mostly about traffic passing through the firewall. Here is an example, just one ping packet, pinging a host on the Internet through the firewall:

Code: [Select]

14:09:17.236464 b0:10:41:71:**:30 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 40521, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.***.38 > 194.***.***.230: ICMP echo request, id 47905, seq 6, length 64
14:09:17.258560 92:38:7e:91:**:2a > b0:10:41:71:**:30, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 244, id 26065, offset 0, flags [none], proto ICMP (1), length 84)
    194.***.***.230 > 192.168.***.38: ICMP echo reply, id 47905, seq 6, length 64
As you can see, the 192.168 IP correctly sends the ICMP packet to the CARP virtual MAC address, but the pfSense gateway answers with the hardware MAC address of it's interface.
Most people won't notice that, but at least here the switches connected to the switch the pfSense gw is connected to now have no idea where to send traffic to the CARP MAC 00:00:5e:00:01:01. Hence they flood it to all ports, which again normally no one will really notice. But once you start wondering why the WLAN accesspoints connected to your switches start flooding ACK packets not destined to any WLAN device when someone downloads from a wired machine, this becomes a potential performance problem for your WLAN. It eats up airtime

[franco]

Alright, same behaviour here with a little mock setup:

% sudo tcpdump -eni em0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:27:22.173986 78:31:c1:d1:04:2c > f4:90:ea:00:19:92, ethertype IPv4 (0x0800), length 98: 10.0.99.1 > 10.0.0.1: ICMP echo request, id 26365, seq 3, length 64
15:27:22.174024 f4:90:ea:00:19:92 > 78:31:c1:d1:04:2c, ethertype IPv4 (0x0800), length 98: 10.0.0.1 > 10.0.99.1: ICMP echo reply, id 26365, seq 3, length 64
15:27:23.185447 78:31:c1:d1:04:2c > f4:90:ea:00:19:92, ethertype IPv4 (0x0800), length 98: 10.0.99.1 > 10.0.0.1: ICMP echo request, id 26365, seq 4, length 64
15:27:23.185486 f4:90:ea:00:19:92 > 78:31:c1:d1:04:2c, ethertype IPv4 (0x0800), length 98: 10.0.0.1 > 10.0.99.1: ICMP echo reply, id 26365, seq 4, length 64
15:27:24.184692 78:31:c1:d1:04:2c > f4:90:ea:00:19:92, ethertype IPv4 (0x0800), length 98: 10.0.99.1 > 10.0.0.1: ICMP echo request, id 26365, seq 5, length 64
15:27:24.184727 f4:90:ea:00:19:92 > 78:31:c1:d1:04:2c, ethertype IPv4 (0x0800), length 98: 10.0.0.1 > 10.0.99.1: ICMP echo reply, id 26365, seq 5, length 64

^^^^ Ping of real gateway IP

15:27:26.345690 78:31:c1:d1:04:2c > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: 10.0.99.1 > 10.0.0.2: ICMP echo request, id 49425, seq 0, length 64
15:27:26.345735 f4:90:ea:00:19:92 > 78:31:c1:d1:04:2c, ethertype IPv4 (0x0800), length 98: 10.0.0.2 > 10.0.99.1: ICMP echo reply, id 49425, seq 0, length 64
15:27:27.353656 78:31:c1:d1:04:2c > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: 10.0.99.1 > 10.0.0.2: ICMP echo request, id 49425, seq 1, length 64
15:27:27.353694 f4:90:ea:00:19:92 > 78:31:c1:d1:04:2c, ethertype IPv4 (0x0800), length 98: 10.0.0.2 > 10.0.99.1: ICMP echo reply, id 49425, seq 1, length 64
15:27:28.354216 78:31:c1:d1:04:2c > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: 10.0.99.1 > 10.0.0.2: ICMP echo request, id 49425, seq 2, length 64
15:27:28.354250 f4:90:ea:00:19:92 > 78:31:c1:d1:04:2c, ethertype IPv4 (0x0800), length 98: 10.0.0.2 > 10.0.99.1: ICMP echo reply, id 49425, seq 2, length 64

^^^^ Ping of CARP Gateway IP

15:28:27.763075 78:31:c1:d1:04:2c > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: 10.0.99.1 > 8.8.8.8: ICMP echo request, id 64913, seq 0, length 64
15:28:27.801367 f4:90:ea:00:19:92 > 78:31:c1:d1:04:2c, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 10.0.99.1: ICMP echo reply, id 64913, seq 0, length 64
15:28:28.766366 78:31:c1:d1:04:2c > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: 10.0.99.1 > 8.8.8.8: ICMP echo request, id 64913, seq 1, length 64
15:28:28.804608 f4:90:ea:00:19:92 > 78:31:c1:d1:04:2c, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 10.0.99.1: ICMP echo reply, id 64913, seq 1, length 64

^^^^ Ping of Google DNS

It looks like FreeBSD 11 is also having this issue and judging by the code in 12-CURRENT no change there as well.

The IP handling, however, looks correct so there may be a chance to derive from CARP handling in the network layer to fix the link layer? I can take a closer look...


Cheers,
Franco

[franco]

https://github.com/opnsense/src/blob/master/sys/netinet/ip_carp.c#L1411-L1413

So, after crashing a few debug kernels it looks like the code to replace the MAC address is right there and working, but PACKET_TAG_CARP is never set for the IPv4 cases where traffic is passing through.


Cheers,
Franco

[athurdent]

Thanks for your efforts! So, no luck in fixing this?

[marzougui]

Hello
Actualy i have the same problème, the carp src adress is respending with wrong mac adresse only for the master FW

the CARP MAC ADDRESS: 00:00:5e:00:01:01

the LAN INTERFACE MAC ADRESSE : 00:50:56:b3:63:29
the MACHINE MAC ADDRESS: 00:50:56:8c:9c:27

LAN
vmx2   10:59:09.437543 00:50:56:8c:9c:27 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 23845, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.99.100 > 8.8.8.8: ICMP echo request, id 12950, seq 782, length 64
LAN:
vmx2   10:59:09.450479 00:50:56:b3:63:29 > 00:50:56:8c:9c:27, ethertype IPv4 (0x0800), length 98: (tos 0xf0, ttl 53, id 0, offset 0, flags [none], proto ICMP (1), length 84)
LAN
vmx2   10:59:10.500023 00:50:56:b3:63:29 > 00:50:56:8c:9c:27, ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 228, id 13513, offset 0, flags [none], proto TCP (6), length 1500)
LAN
vmx2   10:59:16.486125 00:50:56:8c:9c:27 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 24777, offset 0, flags [DF], proto ICMP (1), length 84)
LAN
vmx2   10:59:16.499240 00:50:56:b3:63:29 > 00:50:56:8c:9c:27, ethertype IPv4 (0x0800), length 98: (tos 0xf0, ttl 53, id 0, offset 0, flags [none], proto ICMP (1), length 84)
LAN
vmx2   10:59:17.488025 00:50:56:8c:9c:27 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 24829, offset 0, flags [DF], proto ICMP (1), length 84)
LAN
vmx2   10:59:17.500968 00:50:56:b3:63:29 > 00:50:56:8c:9c:27, ethertype IPv4 (0x0800), length 98: (tos 0xf0, ttl 53, id 0, offset 0, flags [none], proto ICMP (1), length 84)
LAN
vmx2   10:59:30.585845 00:50:56:8c:9c:27 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 26215, offset 0, flags [DF], proto ICMP (1), length 84)

is there any solution please

thank you

[Jacob-]

Does anyone know of a solution that works with equipment that reads the source MAC address in the frame instead of parsing the ARP reply/announcement for the VIP MAC address?

I understand that from


VRRP April 2004
https://datatracker.ietf.org/doc/html/rfc3768#section-8.2

To

VRRPv3 March 2010
https://datatracker.ietf.org/doc/html/rfc5798#page-29

That it looks like this note has been added to clarify

“Note that the source address of the Ethernet frame
   of this ARP response is the physical MAC address of the physical
   router.“


But there is some equipment from other manufactures, ex. Nokia, Cisco, Juniper inspect the source mac of an ARP response to determine the MAC address associated with the IP.


At a previous point

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=141023

The option to modify the source mac was an option

net.link.ether.inet.carp_mac is set to 1

But it was removed as it deviated from the protocol.

The question is, what other options exist for interoperability with these vendors?

[naltalef]

Hi.
I'm not totally sure, but think that the same behavior is in OpenBSD original carp implementation.
It would be good if the MAC Address of the CARP interface was used, but I do not know how it could affect it in case of master failure.
The backup would start using the same MAC, but that the switch have it learned in another port (the one of the master).
Does it have to do with this?

Regards
Norberto

Hi, did a quick search in the forums and on github but did not find an answer, is this FreeBSD/pfSense CARP problem also an issue in OPNsense?

https://redmine.pfsense.org/issues/6957

[franco]

In OpenBSD you can select the MAC behaviour. In FreeBSD you can not.


Cheers,
Franco