1
19.7 Legacy Series / Backup CARP member using CARP IPv6 address as source for ping6
« on: December 02, 2019, 03:29:28 pm »
I have a HA firewall setup that has been working in production with IPv4 for many years.
I'm adding IPv6 now and ran into an issue with source address selection on the Backup CARP interface.
ping6 to any IPv6 address the subnet works correctly from either firewall except for one case:
ping6 to the CARP IP from the Backup firewall fails
For some reason the Backup firewall uses the CARP IP as the source address, even though it is in BACKUP state. If I force ping6 to use the permanent IP assigned to the backup firewall it works fine.
I can see with tcpdump that the frames come with both src and dest IP as the CARP ip. Even the Neighbor Solicitation has incorrect src IP, and is also not sent to ff02::1:ff00:1 (Solicited-node Multicast address).
Here is tcpdump on the master firewall using a regular ping6 to CARP IP from the backup firewall ( ping6 2001:db8:d::1 )
Things work if I force the source IP selection of ping6 (ping6 -S 2001:db8:d::3 2001:db8:d::1)
Pinging the IPv4 CARP master IP works fine still. It's also not just ping6 having issues, ssh to the CARP master IPv6 ip has the same symptoms(tcpdump looks the same with src+dst as CARP IP).
Has any one seen anything like this? I've rebooted multiple times, built and rebuilt the IPv6 CARP as it's own CARP item in opnsense with different VHID and also as an IP alias on the same VHID. I get the same results both ways. Never any problems with IPv4.
With tcpdump -e option I did verify that the ping6 and NS frames had the proper SRC MAC of the backup firewall interface.
Here are ifconfig details for the interface on both firewalls:
Master firewall(oops, edited to change IPv6 first part to 2001:db8 like the rest):
I'm adding IPv6 now and ran into an issue with source address selection on the Backup CARP interface.
ping6 to any IPv6 address the subnet works correctly from either firewall except for one case:
ping6 to the CARP IP from the Backup firewall fails
For some reason the Backup firewall uses the CARP IP as the source address, even though it is in BACKUP state. If I force ping6 to use the permanent IP assigned to the backup firewall it works fine.
I can see with tcpdump that the frames come with both src and dest IP as the CARP ip. Even the Neighbor Solicitation has incorrect src IP, and is also not sent to ff02::1:ff00:1 (Solicited-node Multicast address).
Here is tcpdump on the master firewall using a regular ping6 to CARP IP from the backup firewall ( ping6 2001:db8:d::1 )
Code: [Select]
root@dmzfwa:~ # tcpdump -ni igb2_vlan4 ip6 and not proto 112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2_vlan4, link-type EN10MB (Ethernet), capture size 262144 bytes
05:54:09.722690 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 0, length 16
05:54:10.784383 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 1, length 16
05:54:11.821819 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 2, length 16
05:54:12.831525 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 3, length 16
05:54:13.845976 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 4, length 16
05:54:14.768000 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32
05:54:14.909059 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 5, length 16
05:54:15.768636 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32
05:54:15.963281 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 6, length 16
05:54:16.768648 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32
05:54:17.026216 IP6 2001:db8:d::1 > 2001:db8:d::1: ICMP6, echo request, seq 7, length 16
Things work if I force the source IP selection of ping6 (ping6 -S 2001:db8:d::3 2001:db8:d::1)
Code: [Select]
root@dmzfwa:~ # tcpdump -ni igb2_vlan4 ip6 and not proto 112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2_vlan4, link-type EN10MB (Ethernet), capture size 262144 bytes
06:03:25.427573 IP6 2001:db8:d::3 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2001:db8:d::1, length 32
06:03:25.427666 IP6 2001:db8:d::2 > 2001:db8:d::3: ICMP6, neighbor advertisement, tgt is 2001:db8:d::1, length 32
06:03:25.427740 IP6 2001:db8:d::3 > 2001:db8:d::1: ICMP6, echo request, seq 0, length 16
06:03:25.427776 IP6 2001:db8:d::1 > 2001:db8:d::3: ICMP6, echo reply, seq 0, length 16
Pinging the IPv4 CARP master IP works fine still. It's also not just ping6 having issues, ssh to the CARP master IPv6 ip has the same symptoms(tcpdump looks the same with src+dst as CARP IP).
Has any one seen anything like this? I've rebooted multiple times, built and rebuilt the IPv6 CARP as it's own CARP item in opnsense with different VHID and also as an IP alias on the same VHID. I get the same results both ways. Never any problems with IPv4.
With tcpdump -e option I did verify that the ping6 and NS frames had the proper SRC MAC of the backup firewall interface.
Here are ifconfig details for the interface on both firewalls:
Master firewall(oops, edited to change IPv6 first part to 2001:db8 like the rest):
Code: [Select]
root@dmzfwa:~ # ifconfig igb2_vlan4
igb2_vlan4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether ac:1f:6b:67:01:b0
inet6 fe80::ae1f:6bff:fe67:1b0%igb2_vlan4 prefixlen 64 scopeid 0xd
inet6 2001:db8:d::2 prefixlen 64
inet6 2001:db8:d::1 prefixlen 64 vhid 1
inet 10.10.144.2 netmask 0xffffffc0 broadcast 10.10.144.63
inet 10.10.144.1 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
inet 10.10.144.58 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
inet 10.10.144.54 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
inet 10.10.144.55 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
inet 10.10.144.56 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
inet 10.10.144.57 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 4 vlanpcp: 0 parent interface: igb2
carp: MASTER vhid 1 advbase 1 advskew 0
groups: vlan
root@dmzfwa:~ #
Backup firewall:Code: [Select]
root@dmzfwb:~ # ifconfig igb2_vlan4
igb2_vlan4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether ac:1f:6b:67:01:fe
inet6 fe80::ae1f:6bff:fe67:1fe%igb2_vlan4 prefixlen 64 scopeid 0xd
inet6 2001:db8:d::3 prefixlen 64
inet6 2001:db8:d::1 prefixlen 64 vhid 1
inet 10.10.144.3 netmask 0xffffffc0 broadcast 10.10.144.63
inet 10.10.144.1 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
inet 10.10.144.58 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
inet 10.10.144.54 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
inet 10.10.144.55 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
inet 10.10.144.56 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
inet 10.10.144.57 netmask 0xffffffc0 broadcast 10.10.144.63 vhid 1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 4 vlanpcp: 0 parent interface: igb2
carp: BACKUP vhid 1 advbase 1 advskew 100
groups: vlan
root@dmzfwb:~ #