24.7.2 IPv6 woes

[marjohn56]

I'm 99% sure the default route is stripped by a zero lifetime RA which the kernel reacts to by removing its default route and you can't even see it.

# ndp -r

should reveal that default route and if it's gone afterwards.

Funnily enough I just saw there is a "ndp -I xxx" which can set a default route via one particular interface if the route disappears for any reason but I've never tried it.


Cheers,
Franco

Interesting... I see this:
fe80::8ff:fe61:e32b%igb0 if=igb0, flags=MO, pref=medium, expire=2m59s
So that explains it.

[meyergru]

Interesting... I see this:
fe80::8ff:fe61:e32b%igb0 if=igb0, flags=MO, pref=medium, expire=2m59s
So that explains it.

Usually, the expiry is set to 30 minutes and the next update comes before that. Would be interesting if there is an invalidation or just a missing renewal.

[marjohn56]

Now I'm completely baffled. I rebooted again and sat there running the ndp -r command and watching the timer countdown and reset, and reset, and reset....
It's working fine now.

[franco]

I tried the ndp -I <wanif> but it doesn't work when removing the default route so I guess that's why nobody uses it...


Cheers,
Franco

[meyergru]

A watched pot never boils...  8)

[marjohn56]

24 hours later and still solid.... bloody gremlins.

[franco]

Should have stayed in their planes is what I always tell them.


Cheers,
Franco

[CruxtheNinth]

just upgraded to 24.7.5 and the old problems are back
IA_PD is visible, IA_NA is there but the default route is never showing up (which in effect kills almost all clients as the PD is available and addresses are getting assigned, where all ipv6 lookups will fail)

downgraded to old dhcp6c and everything works again.

EDIT: would it be possible to keep back the allocated Prefix from radvd and dhcpd processes unless there is a RA received? that would at least not kill the whole network due to ipv6 having a higher priority on clients

EDIT2: reverted fully to 24.7.4_1 as there was additional packetloss not only pinging the lan interface but also transit i.e pinging 8.8.8.8 or 1.1.1.1 - packetloss is gone with 24.7.4_1 (and ipv6 is also still stable now)

[CruxtheNinth]

should we start a new thread for 24.7.5 dhcp6c regression or move to github issues?
also not sure what to make out of the timeouts, i dont have time the next days to upgrade again to 24.7.5 trying to replicate it.

[franco]

To be honest I'm not sure what's going on now. It feels like every step good or bad now we have random issues after doing the correct verification process for each step.

Perhaps it makes sense to use the currently installed 24.7.4 with dhcp6c from 24.7.5 first:

# opnsense-revert -r 24.7.5 dhcp6c


Cheers,
Franco

[CruxtheNinth]

Thanks, Franco. I will try that as soon as i find some more time for it.
I wanted to spin up a fresh installation of 24.7.5 on another box for testing as well.

Will update asap.

[franco]

Thanks for all your help, BTW. Highly appreciated.  :)


Cheers,
Franco

[CruxtheNinth]

I upgraded my N100 Box to 24.7.5 this morning and as for right now everything seems stable.

The packet loss i believed from 24.7.5 is in my case seems to be caused by MacOS 15 / Sequoia. Regardless of which ICMP target, i have 0.3 to 0.9% packet loss. Sometimes this happens only after 100+ packets, so it was a coincidence that i saw it on 24.7.5 and not on 24.7.4. Kindly ignore my previous report about it (but maybe keep in mind if more people complain about sudden loss, i could replicate this across 2 updated M1 Macs but my work laptop with Sonoma does not have this issue and neither none of my Win11 systems)

As for IPv6, this seems fine right now but it took much longer for the default route to show up, when looking into the logs what happens directly after the reboot it puzzles me a bit:

Reboot was around 06:55:42.

At 6:56:22 i see

Code Select Expand

<29>1 2024-10-02T06:56:18+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="166"] send request to ff02::1:2%igc0
<29>1 2024-10-02T06:56:18+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="167"] reset a timer on igc0, state=REQUEST, timeo=0, retrans=932
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="168"] Sending Request
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="169"] set client ID (len 14)
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="170"] set server ID (len 14)
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="171"] set IA address
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="172"] set identity association
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="173"] set elapsed time (len 2)
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="174"] set option request (len 4)
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="175"] set IA_PD prefix
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="176"] set IA_PD
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="177"] send request to ff02::1:2%igc0
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="178"] reset a timer on igc0, state=REQUEST, timeo=1, retrans=1786
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="179"] receive reply from fe80::ff:fe01:101%igc0 on igc0
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="180"] get DHCP option client ID, len 14
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="181"]   DUID: 00:01:00:01:2d:7a:03:f4:7c:83:34:be:41:a3
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="182"] get DHCP option server ID, len 14
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="183"]   DUID: 00:01:00:01:26:2c:40:77:00:50:56:b1:8a:d7
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="184"] get DHCP option identity association, len 40
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="185"]   IA_NA: ID=0, T1=1800, T2=2880
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="186"] get DHCP option IA address, len 24
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="187"]   IA_NA address: 2a00:6020:1000:40::518d pltime=3600 vltime=3600
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="188"] get DHCP option DNS, len 32
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="189"] get DHCP option IA_PD, len 41
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="190"]   IA_PD: ID=0, T1=1800, T2=2880
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="191"] get DHCP option IA_PD prefix, len 25
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="192"]   IA_PD prefix: 2a00:6020:5051:7d00::/56 pltime=3600 vltime=3600
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="193"] Received REPLY for REQUEST
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="194"] nameserver[0] 2a00:6020:100::1
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="195"] nameserver[1] 2a00:6020:200::1
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="196"] make an IA: PD-0
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="197"] create a prefix 2a00:6020:5051:7d00::/56 pltime=3600, vltime=3600
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="198"] add an address 2a00:6020:5051:7d00:7e83:34ff:febe:41a4/64 on igc1
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="199"] make an IA: NA-0
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="200"] create an address 2a00:6020:1000:40::518d pltime=3600, vltime=3600
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="201"] add an address 2a00:6020:1000:40::518d/128 on igc0
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="202"] removing an event on igc0, state=REQUEST
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="203"] removing server (ID: 00:01:00:01:26:2c:40:77:00:50:56:b1:8a:d7)
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="204"] executes /var/etc/dhcp6c_wan_script.sh
<13>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 60365 - [meta sequenceId="205"] dhcp6c_script: REQUEST on igc0 executing
<13>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 64486 - [meta sequenceId="206"] dhcp6c_script: REQUEST on igc0 renewal
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="207"] script "/var/etc/dhcp6c_wan_script.sh" terminated
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="208"] got an expected reply, sleeping.

this should also contain the RA but the def route via fe80::ff:fe01:101 never makes into the routing table.

It takes till 07:02 till the route shows up:

Code Select Expand

29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="108"] set client ID (len 14)
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="109"] set server ID (len 14)
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="110"] set IA address
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="111"] set identity association
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="112"] set elapsed time (len 2)
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="113"] set option request (len 4)
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="114"] set IA_PD prefix
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="115"] set IA_PD
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="116"] send request to ff02::1:2%igc0
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="117"] reset a timer on igc0, state=REQUEST, timeo=2, retrans=4304
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="118"] receive reply from fe80::ff:fe01:101%igc0 on igc0
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="119"] get DHCP option client ID, len 14
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="120"]   DUID: 00:01:00:01:2d:7a:03:f4:7c:83:34:be:41:a3
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="121"] get DHCP option server ID, len 14
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="122"]   DUID: 00:01:00:01:26:2c:40:77:00:50:56:b1:8a:d7
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="123"] get DHCP option identity association, len 40
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="124"]   IA_NA: ID=0, T1=1800, T2=2880
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="125"] get DHCP option IA address, len 24
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="126"]   IA_NA address: 2a00:6020:1000:40::518d pltime=3600 vltime=3600
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="127"] get DHCP option DNS, len 32
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="128"] get DHCP option IA_PD, len 41
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="129"]   IA_PD: ID=0, T1=1800, T2=2880
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="130"] get DHCP option IA_PD prefix, len 25
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="131"]   IA_PD prefix: 2a00:6020:5051:7d00::/56 pltime=3600 vltime=3600
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="132"] Received REPLY for REQUEST
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="133"] nameserver[0] 2a00:6020:100::1
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="134"] nameserver[1] 2a00:6020:200::1
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="135"] make an IA: PD-0
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="136"] create a prefix 2a00:6020:5051:7d00::/56 pltime=3600, vltime=3600
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="137"] add an address 2a00:6020:5051:7d00:7e83:34ff:febe:41a4/64 on igc1
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="138"] make an IA: NA-0
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="139"] create an address 2a00:6020:1000:40::518d pltime=3600, vltime=3600
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="140"] add an address 2a00:6020:1000:40::518d/128 on igc0
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="141"] removing an event on igc0, state=REQUEST
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="142"] removing server (ID: 00:01:00:01:26:2c:40:77:00:50:56:b1:8a:d7)
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="143"] executes /var/etc/dhcp6c_wan_script.sh
<13>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 74159 - [meta sequenceId="144"] dhcp6c_script: REQUEST on igc0 executing
<13>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 80208 - [meta sequenceId="145"] dhcp6c_script: REQUEST on igc0 renewal
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="146"] script "/var/etc/dhcp6c_wan_script.sh" terminated
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="147"] got an expected reply, sleeping.
<13>1 2024-10-02T07:02:55+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="148"] /usr/local/etc/rc.newwanipv6: IP renewal starting (address: 2a00:6020:1000:40::518d, interface: wan, device: igc0)
<13>1 2024-10-02T07:02:55+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="149"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
<13>1 2024-10-02T07:02:55+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="150"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
<11>1 2024-10-02T07:02:56+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="151"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned
exit code '3', the output was 'daemon: process already running, pid: 78043'
<13>1 2024-10-02T07:02:56+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="152"] /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using wan, lan
<13>1 2024-10-02T07:02:56+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="153"] /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
<13>1 2024-10-02T07:02:56+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="154"] /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::ff:fe01:101%igc0
<13>1 2024-10-02T07:02:56+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId=



right now everything is fine, i dont know why the route is showing up today but did not during the last update.

full log in attachment