Backup CARP member using CARP IPv6 address as source for ping6

[greg124816]

I have a HA firewall setup that has been working in production with IPv4 for many years.

I'm adding IPv6 now and ran into an issue with source address selection on the Backup CARP interface.

ping6 to any IPv6 address the subnet works correctly from either firewall except for one case:

ping6 to the CARP IP from the Backup firewall fails

For some reason the Backup firewall uses the CARP IP as the source address, even though it is in BACKUP state. If I force ping6 to use the permanent IP assigned to the backup firewall it works fine.

I can see with tcpdump that the frames come with both src and dest IP as the CARP ip. Even the Neighbor Solicitation has incorrect src IP, and is also not sent to ff02::1:ff00:1 (Solicited-node Multicast address).

Here is tcpdump on the master firewall using a regular ping6 to CARP IP from the backup firewall ( ping6 2001:db8:d::1 )

Code: [Select]

root@dmzfwa:~ # tcpdump -ni igb2_vlan4 ip6 and not proto 112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2_vlan4, link-type EN10MB (Ethernet), capture size 262144 bytes
05:54:09.722690 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 0, length 16
05:54:10.784383 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 1, length 16
05:54:11.821819 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 2, length 16
05:54:12.831525 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 3, length 16
05:54:13.845976 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 4, length 16
05:54:14.768000 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32
05:54:14.909059 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 5, length 16
05:54:15.768636 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32
05:54:15.963281 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 6, length 16
05:54:16.768648 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32
05:54:17.026216 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 7, length 16

Things work if I force the source IP selection of ping6  (ping6 -S 2001:db8:d::3 2001:db8:d::1)

Code: [Select]

root@dmzfwa:~ # tcpdump -ni igb2_vlan4 ip6 and not proto 112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2_vlan4, link-type EN10MB (Ethernet), capture size 262144 bytes
06:03:25.427573 IP6 2001:db8:d::3 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32
06:03:25.427666 IP6 2001:db8:d::2 > 2001:db8:d::3: ICMP6, neighbor advertisement, tgt is 2001:db8:d::1, length 32
06:03:25.427740 IP6 2001:db8:d::3 > 2001:db8:d::1: ICMP6, echo request, seq 0, length 16
06:03:25.427776 IP6 2001:db8:d::1 > 2001:db8:d::3: ICMP6, echo reply, seq 0, length 16
Pinging the IPv4 CARP master IP works fine still. It's also not just ping6 having issues, ssh to the CARP master IPv6 ip has the same symptoms(tcpdump looks the same with src+dst as CARP IP).

Has any one seen anything like this? I've rebooted multiple times, built and rebuilt the IPv6 CARP as it's own CARP item in opnsense with different VHID and also as an IP alias on the same VHID. I get the same results both ways. Never any problems with IPv4.

With tcpdump -e option I did verify that the ping6 and NS frames had the proper SRC MAC of the backup firewall interface.

Here are ifconfig details for the interface on both firewalls:

Master firewall(oops, edited to change IPv6 first part to 2001:db8 like the rest):

Code: [Select]

root@dmzfwa:~ # ifconfig igb2_vlan4
igb2_vlan4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether ac:1f:6b:67:01:b0
        inet6 fe80::ae1f:6bff:fe67:1b0%igb2_vlan4 prefixlen 64 scopeid 0xd
        inet6 2001:db8:d::2 prefixlen 64
        inet6 2001:db8:d::1 prefixlen 64 vhid 1
        inet 10.10.144.2 netmask 0xffffffc0 broadcast 10.10.144.63
        inet 10.10.144.1 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        inet 10.10.144.58 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        inet 10.10.144.54 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        inet 10.10.144.55 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        inet 10.10.144.56 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        inet 10.10.144.57 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 4 vlanpcp: 0 parent interface: igb2
        carp: MASTER vhid 1 advbase 1 advskew 0
        groups: vlan
root@dmzfwa:~ #Backup firewall:

Code: [Select]

root@dmzfwb:~ # ifconfig igb2_vlan4
igb2_vlan4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether ac:1f:6b:67:01:fe
        inet6 fe80::ae1f:6bff:fe67:1fe%igb2_vlan4 prefixlen 64 scopeid 0xd
        inet6 2001:db8:d::3 prefixlen 64
        inet6 2001:db8:d::1 prefixlen 64 vhid 1
        inet 10.10.144.3 netmask 0xffffffc0 broadcast 10.10.144.63
        inet 10.10.144.1 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        inet 10.10.144.58 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        inet 10.10.144.54 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        inet 10.10.144.55 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        inet 10.10.144.56 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        inet 10.10.144.57 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 4 vlanpcp: 0 parent interface: igb2
        carp: BACKUP vhid 1 advbase 1 advskew 100
        groups: vlan
root@dmzfwb:~ #

[greg124816]

To add to this, I saw the backup firewall to sending NSs to the Solicited-Multicast address, but it still uses the wrong src IP and the master firewall does not reply.... or it does so internally since the sender is the CARP IP it has assigned to itself.

This shows the frames with src mac of the backup firewall, and the src IP of the CARP IP.... it should be using it's permanent IP of 2001:db8:d:3

Code: [Select]

root@dmzfwa:~ # tcpdump -eni igb2_vlan4 ip6 and not proto 112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2_vlan4, link-type EN10MB (Ethernet), capture size 262144 bytes
08:14:19.238855 ac:1f:6b:67:01:fe > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: 2001:db8:d::1 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32
08:14:20.240621 ac:1f:6b:67:01:fe > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: 2001:db8:d::1 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32
08:14:21.242630 ac:1f:6b:67:01:fe > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: 2001:db8:d::1 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32

I tried setting the CARP IP to 2001:db8:d::12:1234  so that the last 24 bits were populated that get copied to solicited multicast address. It didn't make any difference in how things operated.

[franco]

Sounds like a kernel bug: a CARP address shall never be used as a source address.

https://bugs.freebsd.org/bugzilla/


Cheers,
Franco

[greg124816]

Thank for the pointer.

I was thinking it was not an opnsense issue but I figured I was doing something wrong and someone had seen it before.

before I try and submit a bug I found this promising info:

It seems freebsd 11.2 chooses an IPv6 src address with the longest match to dst address here:
https://github.com/freebsd/freebsd/blob/releng/11.2/sys/netinet6/in6.c#L1818

I found someone posted a patch long ago that prevents using the IP from a CARP interface here:
http://openbsd-archive.7691.n7.nabble.com/hack-for-carp-in-IPv6-source-address-selection-td256756.html

They said it can be worked around by choosing your IPv6 addresses to prevent the longest match selecting the carp interface. I had tried with 2001:db8:d::12:1234 and didn't see any different results. His specific issue was pinging from the master to the backup, I'm seeing the opposite, ping from backup to master.

Looks like i should look a little closer at the in6_matchlen() code and see what is going on and keep poking at it.
https://github.com/freebsd/freebsd/blob/releng/11.2/sys/netinet6/in6.c#L1709

[greg124816]

This is another old discussion I found but someone proposed a prefer_source flag to help prefer a "default" ipv6 address on an interface. There was some disagreement that you should just specify the src address in your application, but it looks like it was commited:
https://github.com/freebsd/freebsd/blob/releng/11.2/sys/netinet6/in6_src.c#L461

This is in a different source file. I'm not sure of how it all ties together but in the ping6.c source code i see a comment about not using raw sockets since they dropped root priv. Maybe in6_src.c is not involved.


But, man ifconfig shows:

Code: [Select]

     prefer_source
             Set a flag to prefer address as a candidate of the source address
             for outgoing packets.

     -prefer_source
             Clear a flag prefer_source.

 I set prefer_source on the non CARP ip and things still work the same so it seems it has no effect in this case.

I'm going to see if i can manipulate the address assignments to avoid longest match on the carp IP.

[greg124816]

After thinking this through and noting that the longest length match issue discussed for that old uncommitted patch was for master to backup.

On the master the CARP IP was getting use for src ip on outgoing by default when pinging the backup's non CARP IP. This caused confusion on the backup and it wouldn't reply since it had the IP in backup state.

That is apparently no longer an issue in 11.2 since I can force the CARP IP to be used outgoing from the master and ping6 works.  tcpdump looks good.... everything proper. The backup sends a reply with Dst=CARP Ip and Src=it's non-CARP IP.

In my current case, I can't manipulate the address assignments to affect the longest match because I'm trying to ping the CARP IP from the backup, so the destination is the CARP IP and it will always be the longest match to the CARP IP present in backup state.

What seems to be needed is some logic like whatever they have on IPv4 to ignore CARP interfaces/addresses etc., at least when in backup state. I haven't looked and don't really have time to go any deeper right now.

I'm going to run with things as they are since ping6 with -S option specifying the source works fine and pinging any hosts/ips on or off the opnsense boxes works fine.

As I turn up IPv6 stuff, if any issues crop up due to how it works I'll revisit the issue.