[SOLVED] IPv6 neighbor solicitation unicasts to CARP MAC not working

[Monviech]

Hello,

I have a weird IPv6 Problem since I've updated the Opnsense Firewall. IPv6 Routing to the Internet stops working like every 3-5 minutes for around 1 minute on all hosts with a Global Unicast Address at the same time.

I am using CARP with link local IPv6 Adresses, for example fe80::172:16:0:1/128.

My troubleshooting suspects the Opnsense. When I Wireshark on the client (icmpv6.type == 135 || icmpv6.type == 136) I can see that at the same moment IPv6 stops working, there are Neighbor Solicitations Unicasts being sent to the virtual mac address of the CARP interface (IETF-VRRP-VRID_0a 00:00:5e:00:01:0a), asking for the Destination fe80::172:16:0:1.

These packets reach the firewall, I've checked with TCPdump, and they're not dropped by any firewall rules. But there is no answer from the Firewall. Thus the NDP Table puts the destination fe80::172:16:0:1 on unreachable, and probes it multiple times. During that time, there is no IPv6 Access to the Internet, since the Gateway is unreachable.

After multiple Unicast tries, there is a Neighbor Soliciation Multicast from the Client to the IPv6mcast (33:33:ff:00:9c:0f) to which the Firewall responds instantly with a Neighbor Advertisement. After this multicast, the Destination fe80::172:16:0:1 is "Reachable" in the ndp Table and the routing works again for around 3-5 Minutes.

Another quirk is that ICMPv6 Ping requests don't get any reply from the CARP IPv6 link local address of the opnsense. The packet is received by the firewall but it doesn't generate a reply.

If anybody has any idea, I'd like some help.

[franco]

I'd recommend trying a /64 as the /128 looks suspicious.


Cheers,
Franco

[Monviech]

I'd recommend trying a /64 as the /128 looks suspicious.


Cheers,
Franco

Thanks for your answer, I've tried it out.

Changing it from /128 to /64 broke all IPv6 connectivity. Changing it back to /128 restored functionality.

A link local CARP IPv6 Address has to be /128 considering these results.

[franco]

Not enough info about base setup and what "broke all connectivity" means.

All link-local is /64. You could just be looking at a wider configuration issue.


Cheers,
Franco

[Monviech]

With broke all connectivity I meant any connectivity to the internet. When I changed the CARP VIP from /128 to /64, I could still ping the GUA of the Firewall, but I couldn't ping the GUA of ipv6.google.com. When changing back from /64 to /128 I could ping the GUA of ipv6.google.com again.

EDIT: There was also a log entry at the time of this change from /128 to /64:

Code: [Select]

/firewall_virtual_ip.php: The command `/sbin/ifconfig 'vlan01.100' inet6 'fe80::172:16:100:1' -alias' failed to execute
There's a diagram of my base setup in the attachements, I hope it has enough information:

[franco]

You post in 23.1 forum but firewall_virtual_ip.php doesn't exist in 23.1? So what version are you on would be a good start

And thanks for diagram. I'll take a closer look.


Cheers,
Franco

[franco]

I've checked my setup and I use CARP IPv6 alias with /64.

The biggest question is... fe80::172:16:0:1 is a CARP VIP with a VHID configured on both ends the same way?

Mind you that IPv6 CARP only works with SLAAC and in Services: Router Advertisements you need to change "Source Address" to that CARP address on both ends as well.


Cheers,
Franco

[Monviech]

You are totally right, I'm in the wrong forum. I'm using OPNsense 22.10.2-amd64 which is based on 22.7.11. I should have posted in the other one.

The CARP VHIDs are the same on both Opnsense. (Screenshots provided)

Router Advertisment is SLAAC and Source Address is the CARP Address. (Screenshot provided) Router Advertisement works as expected.

The problem I saw in Wireshark is that the Neighbor Solicitation Unicast from Clients not getting answered by the Virtual Mac Address of the CARP Interface as Neighbor Discovery Unicast. The Gateway then turns "unreachable" in the NDP Table of the Client, until the Client starts a Neighbor Soliciaton Multicast to 33:33:, which gets answered by the Firewall as Neighbor Discovery. --- I really don't know if this is the expected behavior or not. I just found it weird, and it would explain the issues I have.

I have provided an edited Wireshark Capture as Screenshot.

[Monviech]

I don't understand, everything works again. IPv6 didnt work right for a week and the troubles survived multiple reboots. I didn't change anything... the NDP Troubles are just suddenly gone.

Wireshark doesn't show the behavior in the Screenshot above anymore. Solicitation and Discovery suddenly just work.

[Patrick M. Hausen]

Switch problem? Practically all of IPv6 relies heavily on multicast. Some switches do ... weird things.

[Monviech]

Switch problem? Practically all of IPv6 relies heavily on multicast. Some switches do ... weird things.

Hello,
thanks for the suggestion.

I've checked the switch configurations with a team member, and I haven't found any suspicious settings. The Firmware Versions I use also don't have known Multicast or NDP bugs (Netgear Fully Managed Switches series).

I have provided screenshots of the behavior, captured packets of a client (purple background) and the opnsense firewall (black background). There it's visible that the opnsense doesn't respond to the neighbor soliciation unicast, and only to the multicast.

I would like to investigate this further if possible, but I don't know how to continue.

[Monviech]

Not enough info about base setup and what "broke all connectivity" means.

All link-local is /64. You could just be looking at a wider configuration issue.


Cheers,
Franco

You were exactly right. I have deleted all link local CARP VIPs with /128, and recreated them as /64. Now ICMP and NDP works as expected. Thank you!

EDIT: I also had to configure the Router Advertisements to "Automatic", and then back to the "VIP". Guess that reset the configuration for it.

Code: [Select]

16:19:08.000946 IP6 fe80::ba36:c21f:c99f:5c10 > fe80::172:16:100:1: ICMP6, neighbor solicitation, who has fe80::172:16:100:1, length 32
16:19:08.000994 IP6 fe80::172:16:100:1 > fe80::ba36:c21f:c99f:5c10: ICMP6, neighbor advertisement, tgt is fe80::172:16:100:1, length 24