Messages

Just FYI: the 24.7.3 update that rolled out today fixes this.

Have been running it for several hours now, and IPv6 ND works just fine as expected.

Well... after running 24.7 kernel on an otherwise-24.7.2 OPNsense system for 4 hours, I can confirm that the IPv6 ND shenanigans are gone.

I have not observed any other weirdness or side-effects either with this unholy combo.
I am not running Unbound, I am using Pi-hole + Unbound on a separate box.

BR, -sjm

Well no problem, I am happy to help whenever I can, and it is also for my own benefit!
Is there anything else I could try digging up?

Anyway I will confirm after a few hours that the IPv6 ND shenanigans have really disappeared for good.
Now I can immediately see the correct behaviour with tcpdump because I know what to look after.

BR, -sjm

Well uh, OPNsense 24.7.2 just came out and I upgraded and rebooted my firewall.
Now the IPv6 ND shenanigans are back, as I expected. My opnsense does not bother answering to neighbor solicitation until after random-looking long delays like 10 seconds or so. Sometimes the delay is too much for the operator's device and I would see IPv6 outage until my opnsense would respond to ND again.

Proof:

Code Select Expand


15:50:43.362261 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:xxxx:yyyy:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:44.412213 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:xxxx:yyyy:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:45.442798 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:xxxx:yyyy:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:46.659362 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:47.682844 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:48.722961 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:50.021582 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:51.052573 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:52.082731 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:53.349779 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:54.402914 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:xxxx:yyyy:3a25, length 32
15:50:54.402945 IP6 fe80::2e2:xxxx:yyyy:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:xxxx:yyyy:3a25, length 32


Now I will again downgrade to 24.7 kernel and probably this particular problem will go away (again).

BR, -sjm

@wirehead well, my guess would be that your other IPv6 provider just has different ND settings aka their router is more patient with your neighbor solicitation answers.

Before downgrading to 24.7 kernel I could clearly see my opnsense box having long delays every time when asked for neighbor solicitation, and clearly sometimes the operator's box hit some timeout and I could see short IPv6 outages, until ND worked again.

Well, now I am examining my IPv6 ND traffic with tcpdump and I cannot see any more delays in answering to neighbor solicitation requests! My opnsense is now responding to every ND packet immediately.

So far, it clearly looks like 24.7.1 kernel did break IPv6 neighbor discovery somehow.
Downgrading to 24.7 helped me and the IPv6 shenanigans disappeared. YMMV.

BR, -sjm

Well... I tried opnsense-update -kr 24.7.2 and reboot, it did not help.
The occasional IPv6 outages are still there.

Now I ran opnsense-update -kr 24.7 to try with the previous kernel, aka did not downgrade any other packages.
I will report what happens with this setup. It will be clear in the next few hours...

BR, -sjm

Well uh, I was able to observe similar looking behaviour.

Code Select Expand



root@opns:~ # tcpdump -n -nn -i vlan0.18 'icmp6 and not ip6[40] == 128 and not ip6[40] == 129'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vlan0.18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:53:32.931734 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:34.002399 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:35.042986 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:38.957928 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:40.002543 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:41.052590 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:42.259048 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:43.282588 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:44.322688 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:46.333740 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:47.362818 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32
20:53:47.362850 IP6 fe80::xxxx:yyyy:zzzz:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is 2a0b:5c81:10:0:xxxx:yyyy:zzzz:3a25, length 32


BR, -sjm

The intermittent IPv6 connectivity breaks look like they happen at more or less random intervals. At least I cannot find any clear pattern. Usually there will be one or two breaks in a 10-15 minute period, but it can also run almost 30 minutes flawlessly.

I have a ping monitoring running on my raspi4 with Telegraf, monitoring 3 separate external IPv6 addresses.
Meanwhile, the IPv4 addresses in the same monitoring system are working just fine.

After OPNsense 24.7.1 upgrade, it looks like this.

https://imgur.com/a/ipv6-ping-loss-1WBVHcN

BR, -sjm

Well... I can confirm some weird IPv6 connectivity issues too.

Please note: the ping was ran on the OPNsense box itself, but the results look like exactly the same if I try it from the internal network.

My IPv6 WAN connectivity just breaks somehow occasionally.

Here I was running IPv6 ping to my cloud vm every 3 seconds, with ping -n6 -i 3 -c 200 xxx.yyy.zzz

and when checking the results, every now and then I can see this kind of weirdness:

Code Select Expand


16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=96 hlim=55 time=6.646 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=97 hlim=55 time=6.628 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=105 hlim=55 time=2126.610 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=106 hlim=55 time=6.617 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=107 hlim=55 time=6.510 ms
...
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=113 hlim=55 time=6.544 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=114 hlim=55 time=6.614 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=119 hlim=55 time=2084.833 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=120 hlim=55 time=6.610 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=121 hlim=55 time=6.538 ms
...
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=154 hlim=55 time=6.656 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=155 hlim=55 time=15.322 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=156 hlim=55 time=6.651 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=162 hlim=55 time=19.360 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=163 hlim=55 time=7.046 ms
16 bytes from 2a01:4f9:xxx:zzz::1, icmp_seq=164 hlim=55 time=6.548 ms


so... a bit randomly, total IPv6 blackout is observed for roughly 20 seconds.

Meanwhile, just for comparison, I have an OpenWrt box connected to the same WAN vlan and it does not exhibit this kind of IPv6 packet loss. So IMO problems in operator side are more or less ruled out and my OPNsense has something weird going on.

Futhermore, this problem began exactly after I upgraded to OPNsense 24.7.1.

If anyone has any good hints what to look after with tcpdump, I am all ears (and eyes).

BR, -sjm