Text only | Text with Images

OPNsense Forum

English Forums => 24.7, 24.10 Legacy Series => Topic started by: CruxtheNinth on August 26, 2024, 08:28:06 AM

Title: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 26, 2024, 08:28:06 AM
Good Morning,

since the 24.7.2 update there is something odd with IPv6.
Besides that it takes incredibly long for DHCP6 on WAN to get an IP (for my ISP Deutsche Glasfaser  it can take normally 30 - 60 minutes but with 24.7.2 it takes multiple hours) something else is off.

Directly after the GUA is assigned IPv6 seems to work normally i.e ping6 google.de works, various ipv6 test websites show all is well.

After a few hours, seemingly random, IPv6 stops working. The GUA is still assigned, all Clients have an IP but traffic is no longer processed.

Looking at the Firewall log it seems all traffic originating from the GUA inside my network is dropped outbound on LAN with a state violation.


(https://i.imgur.com/m2jgzRB.png)

(https://i.imgur.com/4zP3G46.png)


Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 26, 2024, 08:31:22 AM
There is a test kernel for the ICMPv6 instabilities introduced in 24.7.1:

https://github.com/opnsense/src/issues/218#issuecomment-2308039278

# opnsense-update -zkr 24.7.2-nd


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 26, 2024, 08:36:22 AM
Quote from: franco on August 26, 2024, 08:31:22 AM
There is a test kernel for the ICMPv6 instabilities introduced in 24.7.1:

https://github.com/opnsense/src/issues/218#issuecomment-2308039278

# opnsense-update -zkr 24.7.2-nd


Cheers,
Franco

Thanks Franco, as a matter of fact i had this Kernel running already but it did not resolve the issue, so i reverted back to 24.7.2 default kernel to not mix up potential topics and issues before creating this post.

I can upgrade to 24.7.2-nd again and take some captures if needed
Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 26, 2024, 08:39:37 AM
The other thing people mentioned was:

# opnsense-revert -z dhcp6c

(and clean reboot)

However, both changes done to dhcp6c were done on edge cases and should improve recovery behaviour which for DG it may not? I'll need logs on the bad version to make sense of it.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 26, 2024, 08:47:53 AM
Quote from: franco on August 26, 2024, 08:39:37 AM
The other thing people mentioned was:

# opnsense-revert -z dhcp6c

(and clean reboot)

However, both changes done to dhcp6c were done on edge cases and should improve recovery behaviour which for DG it may not? I'll need logs on the bad version to make sense of it.


Cheers,
Franco


Thank you, i will test later this afternoon. If you need any debugs / pcaps / logs with the "bad" version please let me know what exactly you need.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 26, 2024, 08:56:26 AM
Interfaces: Settings: IPv6 DHCP Log level "Info". The problem appears on the renew more often than initially although reports have been unclear to me so far. Setting this option needs a reboot too. So switch to bad version before reboot (opnsense-revert -r 24.7.2 dhcp6c). Let me have the system log when this stopped renewing correctly to inspect what is going on. I'm assuming the ISP has a stricter policy on ordering DHCP and DHCPv6 requests which could be annoying because there is no way to tell from the system.
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 27, 2024, 07:49:42 AM
My issue as of now is fully resolved with running:

opnsense-revert -r 24.7.1 dhcp6c
opnsense-update -zkr 24.7.2-nd

observation:

at first i attempted, as you wrote, with opnsense-revert -z dhcp6c, which installed dhcp6c-20240820 which also had no effect on the problem whatsoever.

Trying to get some logs with the 24.7.2: opnsense-revert -r 24.7.2 dhcp6c also installed dhcp6c-20240820 so i figured the previous attempted revert failed for some unknown reason.

Which led me to try opnsense-revert -r 24.7.1 dhcp6c - which fixed all problems, all clients fine, no suddenly missing IPv6, all stable for over 10 hours now with dhcp6c-20240710

Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 27, 2024, 09:45:45 AM
> Which led me to try opnsense-revert -r 24.7.1 dhcp6c - which fixed all problems, all clients fine, no suddenly missing IPv6, all stable for over 10 hours now with dhcp6c-20240710

That's correct, yes. I fumbled that part.

I still need the bad logs. I suspect this is:

https://github.com/opnsense/dhcp6c/commit/14d87d18a71

And for som reason DG will not like that we let the RELEASE state go... but it's a clear recovery scenario...
Title: Re: 24.7.2 IPv6 woes
Post by: lp24db on August 27, 2024, 10:17:50 AM
Just want to tell that i have the exact same problem like @CruxtheNinth with DGF

Im first testing the reverted package and the new kernel.

If this works, i'm looking forward to test dhcp debug mode. But its not easy for me because four other people here are not happy about router reboots :)

Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 27, 2024, 12:38:47 PM
It's still odd that DG is the only ISP where the change seems to fail as far as reports go. But it's not that DG didn't have a special reputation already. Maybe this is even an opportunity to contact them and ask to fix their infrastructure? ;)


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 27, 2024, 03:02:47 PM
Quote from: franco on August 27, 2024, 12:38:47 PM
It's still odd that DG is the only ISP where the change seems to fail as far as reports go. But it's not that DG didn't have a special reputation already. Maybe this is even an opportunity to contact them and ask to fix their infrastructure? ;)


Cheers,
Franco

i attached the system.log of yesterday, maybe you can spot anything odd.
I noticed some  XID mismatch events that did not happen the days before or today.
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on August 27, 2024, 03:53:08 PM
If your specific problem is DG, I can tell you this from my own experience with 3 OpnSense installations on that ISP:

1. Do not - I repeat: do not - use traffic shaping with that provider. I never got to the bottom of it, but I know for sure that if you use traffic shaping and generate a bit more traffic, IPv6 suddenly drops for a few minutes and then comes on again. I can repeat that test over an over and always get the same result: ONLY IPv6 is down - IPv4 continues to work as usual.

2. For the same reason, I have one single "out" rule on WAN to never expose RFC1918 source or destination IPs to the WAN. This can happen if misconfigured clients try to access default IPs that are not present in my own network(s), because OpnSense is the default router which in turn has the ISP provider as default route.

What seems to be the case is that DG drops connections when they see something that they think to be misus of one kind or another (like spoofed source IPs).


Other than that, my 3 DG installations work fine with OpnSense 24.7.2.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 27, 2024, 03:57:40 PM
Uwe, wasn't there something about the dreaded IPv4 connectivity switch?
Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 27, 2024, 04:09:40 PM
It seems to be stuck in a RENEW loop, partially due to itself but I'm unsure why. We don't hit NoBind state so it could only be the patch that I mentioned earlier.

> pltime=3600 vltime=3600

This seems excessively unnecessary.

I'll keep digging.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 27, 2024, 04:35:06 PM
from what i understand DG does not answer to DHCPv6 IA-NA / IA-PD solicits but will somewhen (30 to 60 minutes) send a DHCPv6 Advertise containing IA-NA/IA-PD that can be requested afterwards. Not sure if that helps
Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 27, 2024, 04:36:32 PM
@CruxtheNinth Do you use "Use IPv4 connectivity" for DHCPv6 in your WAN setup?
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 27, 2024, 04:45:46 PM
Quote from: franco on August 27, 2024, 04:36:32 PM
@CruxtheNinth Do you use "Use IPv4 connectivity" for DHCPv6 in your WAN setup?

no, "use IPv4 connectivity" is disabled, i only ever enabled it once for testing after the 24.7.2 update when i misread the PPPoE/IPv6 Thread.

Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 27, 2024, 04:51:27 PM
Ok, thanks for the additional info. To me it looks like a timer stuck and triggering increasingly often without the need for it. It's not directly related to the patch. I triple checked the code. So what likely happened is we uncovered a real bug now that would fix itself in the previous code version, maybe related to a SIGHUP event.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on August 27, 2024, 05:15:49 PM
Quote from: franco on August 27, 2024, 03:57:40 PM
Uwe, wasn't there something about the dreaded IPv4 connectivity switch?

Yes, I had to switch that with 24.7.x. My working settings for the DG installations look like this. Note I use the new "IA-PD-only" scheme for DG and use one of the prefixes for the WAN interface, as I found early that DG does not answer at all if an IA-NA is requested. Maybe that is the problem?

Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 27, 2024, 06:03:26 PM
Ok, I built a test version that gives us visibility over timer create/remove and current number of events:

https://github.com/opnsense/dhcp6c/commit/1e84486429

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_1.pkg

(please reboot at your convenience)

# opnsense-log | grep 'dhcp6c.*event'

opnsense-log only keeps the last day so when this fails tomorrow I gladly take the full logs as before.

It lists things from here and doesn't crash so I'm fairly confident this can help with finding the actual problem.



Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 27, 2024, 08:33:14 PM
Ok, WTF, I can see that on my end as well already:

creating an event on igb1, state=INIT
removing an event on igb1, state=INIT
listing event on igb1, state=INIT
creating an event on igb1, state=INIT
removing an event on igb1, state=REQUEST
listing event on igb1, state=REQUEST
creating an event on igb1, state=RENEW
removing an event on igb1, state=RENEW
listing event on igb1, state=RENEW
creating an event on igb1, state=RENEW
removing an event on igb1, state=RENEW
listing event on igb1, state=RENEW
creating an event on igb1, state=RENEW
removing an event on igb1, state=RENEW
listing event on igb1, state=RENEW
creating an event on igb1, state=RENEW
creating an event on igb1, state=REBIND
listing event on igb1, state=RENEW
removing an event on igb1, state=REBIND
listing event on igb1, state=RENEW
listing event on igb1, state=REBIND
Title: Re: 24.7.2 IPv6 woes
Post by: itn3rd77 on August 27, 2024, 08:43:59 PM
Same thing happened to me. Suddently IPv6 was lost on my DG installation and not coming back to life since 6 hours. I'll see if I can catch the logs requested.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 27, 2024, 08:51:24 PM
I'm pretty sure the problem is somewhere here:

https://github.com/opnsense/dhcp6c/blob/abd51bd51ce0090a66dcec8aca0ef3c320490cbe/dhcp6c_ia.c#L577-L595

Maybe dhcp6_set_timer sets a zero time and the REBIND kicks in as well which isn't very helpful.

At least something to play with here.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 27, 2024, 10:40:00 PM
Doesn't always break it seems, but looks like this is an interesting lead.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 28, 2024, 07:20:10 AM
Quote from: meyergru on August 27, 2024, 05:15:49 PM
Quote from: franco on August 27, 2024, 03:57:40 PM
Uwe, wasn't there something about the dreaded IPv4 connectivity switch?

Yes, I had to switch that with 24.7.x. My working settings for the DG installations look like this. Note I use the new "IA-PD-only" scheme for DG and use one of the prefixes for the WAN interface, as I found early that DG does not answer at all if an IA-NA is requested. Maybe that is the problem?

I only use the Prefix-Hint option and i receive both PD and /128 IA-NA on WAN

(https://i.imgur.com/lt56c1s.png)
Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 28, 2024, 08:39:03 AM
Hmm, this gets weirder:

reset interval is sec=0, usec=663000
reset interval is sec=0, usec=516000
reset interval is sec=1, usec=84000
reset interval is sec=1, usec=80000
reset interval is sec=10, usec=511000
reset interval is sec=429497, usec=672000
reset interval is sec=0, usec=734000
reset interval is sec=0, usec=994000
reset interval is sec=0, usec=571000
reset interval is sec=0, usec=380000
reset interval is sec=0, usec=367000
reset interval is sec=0, usec=122000
reset interval is sec=0, usec=284000
reset interval is sec=1, usec=58000
reset interval is sec=1, usec=48000

I suspect https://github.com/opnsense/dhcp6c/commit/fe3ed661a5 is partly to blame here but the modifications look correct to me. There must be an edge case in there we've uncovered.

EDIT: Appears to be RELEASE specific issue, possibly not new and irrelevant.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: itn3rd77 on August 29, 2024, 10:13:25 AM
Hi,

I still don't get an IPv6 from DG. Don't know if this is an DG issue or related to the problem discussed here.

I did the following:

opnsense-revert -r 24.7.1 dhcp6c
opnsense-update -zkr 24.7.2-nd

But nothing changed for hours so I reverted to to 24.7.2 original state.

I installed

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_1.pkg

and rebooted. But I don't see any 'dhcp6c.*event' messages (I enabled DHCPv6 logging on INFO).

I only see the following messages for dhcpv6:

<13>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 727 - [meta sequenceId="224"] RTSOLD script - Sending SIGHUP to dhcp6c
<29>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="225"] restarting
<29>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="226"] duplicated interface: igb0
<29>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="240"] Sending Solicit
<29>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="251"] Sending Solicit
<27>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="252"] transmit failed: Network is down
<29>1 2024-08-29T09:57:00+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="256"] Sending Solicit
<29>1 2024-08-29T09:57:01+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="257"] Sending Solicit
<27>1 2024-08-29T09:57:01+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="258"] transmit failed: Network is down
<29>1 2024-08-29T09:57:03+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="270"] Sending Solicit
<29>1 2024-08-29T09:57:07+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="276"] Sending Solicit
<29>1 2024-08-29T10:01:06+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="4"] restarting
<29>1 2024-08-29T10:01:06+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="5"] duplicated interface: igb0
<29>1 2024-08-29T10:01:07+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="6"] Sending Solicit
<29>1 2024-08-29T10:01:07+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="7"] Sending Solicit
<27>1 2024-08-29T10:01:07+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="8"] transmit failed: Network is down
<29>1 2024-08-29T10:01:08+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="9"] Sending Solicit
<29>1 2024-08-29T10:01:08+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="10"] Sending Solicit
<27>1 2024-08-29T10:01:08+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="11"] transmit failed: Network is down
<29>1 2024-08-29T10:01:10+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="15"] Sending Solicit
Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 29, 2024, 10:29:36 AM
> Network is down

It's not very assuring.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: itn3rd77 on August 29, 2024, 12:43:51 PM
Ok Franco I disabled the PPP device. This was also requesting an IPv6 for no reason. Output looks ok now for my limited knowledge. But nothing happens ...

<13>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 65073 - [meta sequenceId="392"] RTSOLD script - Sending SIGHUP to dhcp6c
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="393"] restarting
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="394"] removing an event on igb0, state=SOLICIT
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="395"] listing event on igb0, state=SOLICIT
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="396"] <3>[interface] (9)
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="397"] <5>[igb0] (4)
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="398"] <3>begin of closure [{] (1)
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="399"] <3>[send] (4)
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="400"] <3>[ia-na] (5)
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="401"] <3>
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 29, 2024, 12:51:50 PM
why do you need PPP for DG? That should not longer be necessary according to their own published specs.
Both IPv4 and IPv6 should do DHCP (via IPOE)

Are you mistaking DG with Deutsche GigaNetz? DG in this thread is Deutsche Glasfaser, not Deutsche Giganetz.

Quote from: itn3rd77 on August 29, 2024, 12:43:51 PM
Ok Franco I disabled the PPP device. This was also requesting an IPv6 for no reason. Output looks ok now for my limited knowledge. But nothing happens ...

<13>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 65073 - [meta sequenceId="392"] RTSOLD script - Sending SIGHUP to dhcp6c
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="393"] restarting
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="394"] removing an event on igb0, state=SOLICIT
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="395"] listing event on igb0, state=SOLICIT
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="396"] <3>[interface] (9)
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="397"] <5>[igb0] (4)
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="398"] <3>begin of closure [{] (1)
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="399"] <3>[send] (4)
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="400"] <3>[ia-na] (5)
<29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="401"] <3>
  • (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="402"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="403"] <3>comment [# request stateful address] (26)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="404"] <3>[send] (4)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="405"] <3>[ia-pd] (5)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="406"] <3>
  • (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="407"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="408"] <3>comment [# request prefix delegation] (27)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="409"] <3>[request] (7)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="410"] <3>[domain-name-servers] (19)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="411"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="412"] <3>[request] (7)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="413"] <3>[domain-name] (11)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="414"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="415"] <3>[script] (6)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="416"] <3>["/var/etc/dhcp6c_wan_script.sh"] (31)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="417"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="418"] <3>comment [# we'd like some nameservers please] (35)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="419"] <3>end of closure [}] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="420"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="421"] <3>[id-assoc] (8)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="422"] <13>[na] (2)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="423"] <13>
  • (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="424"] <13>begin of closure [{] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="425"] <3>end of closure [}] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="426"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="427"] <3>[id-assoc] (8)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="428"] <13>[pd] (2)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="429"] <13>
  • (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="430"] <13>begin of closure [{] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="431"] <3>[prefix] (6)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="432"] <3>[::] (2)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="433"] <3>[/] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="434"] <3>[56] (2)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="435"] <3>[infinity] (8)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="436"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="437"] <3>[prefix-interface] (16)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="438"] <5>[vlan0.13] (8)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="439"] <3>begin of closure [{] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="440"] <3>[sla-id] (6)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="441"] <3>[1] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="442"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="443"] <3>[sla-len] (7)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="444"] <3>[8] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="445"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="446"] <3>end of closure [}] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="447"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="448"] <3>[prefix-interface] (16)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="449"] <5>[igb1] (4)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="450"] <3>begin of closure [{] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="451"] <3>[sla-id] (6)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="452"] <3>
  • (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="453"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="454"] <3>[sla-len] (7)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="455"] <3>[8] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="456"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="457"] <3>end of closure [}] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="458"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="459"] <3>[prefix-interface] (16)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="460"] <5>[vlan0.42] (8)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="461"] <3>begin of closure [{] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="462"] <3>[sla-id] (6)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="463"] <3>[2] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="464"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="465"] <3>[sla-len] (7)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="466"] <3>[8] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="467"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="468"] <3>end of closure [}] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="469"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="470"] <3>end of closure [}] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="471"] <3>end of sentence [;] (1)
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="472"] called
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="473"] duplicated interface: igb0
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="474"] called
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="475"] creating an event on igb0, state=INIT
    <29>1 2024-08-29T12:02:23+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="476"] reset a timer on igb0, state=INIT, timeo=0, retrans=540
    <29>1 2024-08-29T12:02:24+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="493"] Sending Solicit
    <29>1 2024-08-29T12:02:24+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="494"] a new XID (2e4d29) is generated
    <29>1 2024-08-29T12:02:24+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="495"] set client ID (len 14)
    <29>1 2024-08-29T12:02:24+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="496"] set identity association
    <29>1 2024-08-29T12:02:24+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="497"] set elapsed time (len 2)
    <29>1 2024-08-29T12:02:24+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="498"] set option request (len 4)
    <29>1 2024-08-29T12:02:24+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="499"] set IA_PD prefix
    <29>1 2024-08-29T12:02:24+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="500"] set IA_PD
    <29>1 2024-08-29T12:02:24+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="501"] send solicit to ff02::1:2%igb0
    <29>1 2024-08-29T12:02:24+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="502"] reset a timer on igb0, state=SOLICIT, timeo=0, retrans=1098
    <29>1 2024-08-29T12:02:25+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="503"] Sending Solicit
    <29>1 2024-08-29T12:02:25+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="504"] set client ID (len 14)
    <29>1 2024-08-29T12:02:25+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="505"] set identity association
    <29>1 2024-08-29T12:02:25+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="506"] set elapsed time (len 2)
    <29>1 2024-08-29T12:02:25+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="507"] set option request (len 4)
    <29>1 2024-08-29T12:02:25+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="508"] set IA_PD prefix
    <29>1 2024-08-29T12:02:25+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="509"] set IA_PD
    <29>1 2024-08-29T12:02:25+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="510"] send solicit to ff02::1:2%igb0
    <29>1 2024-08-29T12:02:25+02:00 fw.local.home dhcp6c 28307 - [meta sequenceId="511"] reset a timer on igb0, state=SOLICIT, timeo=1, retrans=117408

    No more output since last line. Can you spot anything relevant or is this just an DG issue and waiting should help  ;)
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 29, 2024, 12:59:30 PM
you reverted dhcp6c to 24.7.1 version (dhcp6c-20240710) and installed another newer one with pkg-add later.
Did you test in-between (and rebooted)?

Please test with dhcp6c-20240710 again as it seems you did some changes to the interfaces in between

Also please note that IPv6 DHCP on Deutsche Glasfaser takes 30-60 minutes depending on if you miss the Advertisement

Quote from: itn3rd77 on August 29, 2024, 10:13:25 AM
Hi,

I still don't get an IPv6 from DG. Don't know if this is an DG issue or related to the problem discussed here.

I did the following:

opnsense-revert -r 24.7.1 dhcp6c
opnsense-update -zkr 24.7.2-nd

But nothing changed for hours so I reverted to to 24.7.2 original state.

I installed

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_1.pkg

and rebooted. But I don't see any 'dhcp6c.*event' messages (I enabled DHCPv6 logging on INFO).

I only see the following messages for dhcpv6:

<13>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 727 - [meta sequenceId="224"] RTSOLD script - Sending SIGHUP to dhcp6c
<29>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="225"] restarting
<29>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="226"] duplicated interface: igb0
<29>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="240"] Sending Solicit
<29>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="251"] Sending Solicit
<27>1 2024-08-29T09:56:59+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="252"] transmit failed: Network is down
<29>1 2024-08-29T09:57:00+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="256"] Sending Solicit
<29>1 2024-08-29T09:57:01+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="257"] Sending Solicit
<27>1 2024-08-29T09:57:01+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="258"] transmit failed: Network is down
<29>1 2024-08-29T09:57:03+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="270"] Sending Solicit
<29>1 2024-08-29T09:57:07+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="276"] Sending Solicit
<29>1 2024-08-29T10:01:06+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="4"] restarting
<29>1 2024-08-29T10:01:06+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="5"] duplicated interface: igb0
<29>1 2024-08-29T10:01:07+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="6"] Sending Solicit
<29>1 2024-08-29T10:01:07+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="7"] Sending Solicit
<27>1 2024-08-29T10:01:07+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="8"] transmit failed: Network is down
<29>1 2024-08-29T10:01:08+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="9"] Sending Solicit
<29>1 2024-08-29T10:01:08+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="10"] Sending Solicit
<27>1 2024-08-29T10:01:08+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="11"] transmit failed: Network is down
<29>1 2024-08-29T10:01:10+02:00 fw.local.home dhcp6c 44898 - [meta sequenceId="15"] Sending Solicit
Title: Re: 24.7.2 IPv6 woes
Post by: itn3rd77 on August 29, 2024, 01:04:06 PM
Quote from: CruxtheNinth on August 29, 2024, 12:51:50 PM
why do you need PPP for DG? That should not longer be necessary according to their own published specs.
Both IPv4 and IPv6 should do DHCP (via IPOE)

Are you mistaking DG with Deutsche GigaNetz? DG in this thread is Deutsche Glasfaser, not Deutsche Giganetz.

No no :) I have a LTE modem (PPP) that was also requesting a dhcpv6 address. I have disabled that to not mix up the logs.  I am a poor Deutsche Glasfaser customer  :)
Title: Re: 24.7.2 IPv6 woes
Post by: itn3rd77 on August 29, 2024, 01:09:57 PM
Quote from: CruxtheNinth on August 29, 2024, 12:59:30 PM
you reverted dhcp6c to 24.7.1 version (dhcp6c-20240710) and installed another newer one with pkg-add later.
Did you test in-between (and rebooted)?

Please test with dhcp6c-20240710 again as it seems you did some changes to the interfaces in between

Also please note that IPv6 DHCP on Deutsche Glasfaser takes 30-60 minutes depending on if you miss the Advertisement

Thanks for jumping in CruxtheNinth. I testet in between with a reboot. After that I  switched back to bare 24.7.2 and than added the newer one with pkg-add. So currently I am on 24.7.2 with the debug package from Franco.
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 29, 2024, 01:36:47 PM
so with this constellation (and your disabled PPP interface, reboots etc)

opnsense-revert -r 24.7.1 dhcp6c
opnsense-update -zkr 24.7.2-nd

even after 1 hour+, you did not receive a Prefix? What about "Use IPv4 connectivity" - in my case that must be DISABLED otherwise it does not work at all for me.

curious how many different topologies Deutsche Glasfaser has and issues they seem to trigger

EDIT: the reason i am asking so specifically is that, while its fine and great that you want to provide logs with the debug version that Franco shared, if even the supposed to be working combination (24.7.1 dhcp6c + 24.7.2-nd kernel) fails in your case that it may be a totally different issue
Title: Re: 24.7.2 IPv6 woes
Post by: lp24db on August 29, 2024, 02:58:14 PM
For more confusing: i actually have "Use IPv4 connectivity" enabled (since start of DGF) . For me it works fine with the special-kernel and the older dhcp6c


Quote from: CruxtheNinth on August 29, 2024, 01:36:47 PM

opnsense-revert -r 24.7.1 dhcp6c
opnsense-update -zkr 24.7.2-nd

even after 1 hour+, you did not receive a Prefix? What about "Use IPv4 connectivity" - in my case that must be DISABLED otherwise it does not work at all for me.

Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on August 29, 2024, 03:26:42 PM
i am wondering why this should even be activated. Maybe this is from when DG was using PPPoE at one point? did they even ever? in case native DHCP is supported it should not be needed at all

@Ip24db: did you ever test without it, just wondering what would happen if you disable it (just curious)

Quote from: lp24db on August 29, 2024, 02:58:14 PM
For more confusing: i actually have "Use IPv4 connectivity" enabled (since start of DGF) . For me it works fine with the special-kernel and the older dhcp6c


Quote from: CruxtheNinth on August 29, 2024, 01:36:47 PM

opnsense-revert -r 24.7.1 dhcp6c
opnsense-update -zkr 24.7.2-nd

even after 1 hour+, you did not receive a Prefix? What about "Use IPv4 connectivity" - in my case that must be DISABLED otherwise it does not work at all for me.

Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 29, 2024, 04:57:53 PM
When not using PPP(oE) the IPv4 connectivity toggle serialises the IPv6 after IPv4. But what that means in practice is that it likely just fixes a glitch in the dhcp6c by starting it a tiny bit later.

Well, it does the same in PPP(oE) but it wasn't meant to be operable without PPP(oE) but it hasn't been put in the GUI this way unfortunately.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: lp24db on August 31, 2024, 02:33:54 PM
Just to complete the discussion:

Removing UseIPv4Connectivity causes a delay while retrieving the ipv6 addresses / prefixes on my setup. With this option set, the adresses appear in less time. Like <1min with and up to 3-4min without.

Quote from: CruxtheNinth on August 29, 2024, 03:26:42 PM
i am wondering why this should even be activated. Maybe this is from when DG was using PPPoE at one point? did they even ever? in case native DHCP is supported it should not be needed at all

@Ip24db: did you ever test without it, just wondering what would happen if you disable it (just curious)

Quote from: lp24db on August 29, 2024, 02:58:14 PM
For more confusing: i actually have "Use IPv4 connectivity" enabled (since start of DGF) . For me it works fine with the special-kernel and the older dhcp6c


Quote from: CruxtheNinth on August 29, 2024, 01:36:47 PM

opnsense-revert -r 24.7.1 dhcp6c
opnsense-update -zkr 24.7.2-nd

even after 1 hour+, you did not receive a Prefix? What about "Use IPv4 connectivity" - in my case that must be DISABLED otherwise it does not work at all for me.

Title: Re: 24.7.2 IPv6 woes
Post by: franco on August 31, 2024, 09:57:02 PM
Yes, it seems a bit like the ISP expects this to happen (IPv6 after IPv4). I remember at least one case where the ISP never sent another RA as soon as DHCPv6 was up, too. It's so hard to get all of this under a shared code base without an array of advanced toggles. oO


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: newsense on September 01, 2024, 12:07:04 AM
Either toggles or a drop down menu allowing the user to select their particular ISP that needs a certain set of tweaks to work properly when the default options are not enough - wouldn't be such a bad idea ?

Should also improve the clarity in the reports:

a) I'm using ISP ABC with Default IPv6 settings - doesn't work - but choosing problematic ISP2 from the list fixes things for me

b) I'm using ISP DEF - doesn't work with any of the options


Just an idea, usnsure if feasible... :)
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 01, 2024, 09:58:43 AM
on a global scale, for all possible ISP / Access combinations, not feasible.
A user/group maintained repo with config options, maybe.

The problem we see here already is that, e.g Deutsche Glasfaser, seems to have deviating configurations depending on the (*PON) Region you are in.

I would go so far and say that thankfully it is not the majority of ISPs that play against standard, like DG ignoring RA solicits or weird 6rd implementations. So its probably enough to keep track of the exotic variants and even for these its the minority of users that uses a 3rd party firewall / router instead of the ISP issued ones.


Quote from: newsense on September 01, 2024, 12:07:04 AM
Either toggles or a drop down menu allowing the user to select their particular ISP that needs a certain set of tweaks to work properly when the default options are not enough - wouldn't be such a bad idea ?

Should also improve the clarity in the reports:

a) I'm using ISP ABC with Default IPv6 settings - doesn't work - but choosing problematic ISP2 from the list fixes things for me

b) I'm using ISP DEF - doesn't work with any of the options


Just an idea, usnsure if feasible... :)
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 05, 2024, 07:40:37 AM
Quick reply for 24.7.3_1, all packages default, release kernel.

Updated System:

- Directly after reboot IA_NA almost instantly received; /128 shown on igc0 WAN
- few minutes later IA_PD received but no default route yet (as DG will not reply to RA Solicits) - /64 on igc1 LAN visible
- 10 minutes later, default route received but Prefix and Network Address are gone. tried restarting wan interface, no effect, same as above.
- > opnsense-revert -r 24.7.1 dhcp6c & reboot
- Directly after reboot  IA_NA & IA_PD visible, no gateway yet
- 10 minutes later, default route received, all green. Prefix and Network Address are still there.

i am staying with dhcp6c-20240710 for now



system latest.log attached. Upgrade reboot was approx 07:13 today, 2nd reboot was after downgrading
i took some pcaps during the problem, in case they are also needed.

Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 05, 2024, 08:51:11 AM
Given that all the other ICMPv6 shenanigans were ongoing for the last couple of weeks I couldn't look into it more yet.

I've had my IPv6 broken the last couple of days because the FritzBox insistent on NoPrefixAvail for a few times my main system tried to renew and it has been dead in the water since then as far as IPv6 was concerned. The whole protocol sucks IMO, because eventually attempts to recovery always seem to fail when something doesn't go right all the time, which is more often than not. Note this is completely unrelated to the problem at hand, but also a reason why have almost a week of missing log data to sift through.

Instead of guessing what made dhcp6c regress we should probably bisect to be sure. At this point to me it's a bit unclear what caused this change in behaviour and why it only manifests with DG.

I'll publish a new test version shortly.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 05, 2024, 08:59:52 AM
good: v20240710, bad v20240820

Bisecting: 7 revisions left to test after this (roughly 3 steps)
[ba9cddfcb6e568a5c83775ed0c917a316acb64db] dhcp6c: fix prototype

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_1.pkg
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 05, 2024, 10:41:02 AM
@CruxtheNinth: It now seems obvious that DG has different PON setups in different areas. My 3 installations are all in Lower-Saxony and are based on Nokia equipment. Just for the fun of it: I have read that DG also uses Genexis ONTs, is that what you have?
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 05, 2024, 10:50:42 AM
Quote from: franco on September 05, 2024, 08:59:52 AM
good: v20240710, bad v20240820

Bisecting: 7 revisions left to test after this (roughly 3 steps)
[ba9cddfcb6e568a5c83775ed0c917a316acb64db] dhcp6c: fix prototype

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_1.pkg

all green so far with this version
IA_NA directly after reboot, IA_PD a few moments later. RA also fine, route got few minutes after reboot
Will observe a bit longer

EDIT: logs attached; reboot was 10:46
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 05, 2024, 10:51:16 AM
Quote from: meyergru on September 05, 2024, 10:41:02 AM
@CruxtheNinth: It now seems obvious that DG has different PON setups in different areas. My 3 installations are all in Lower-Saxony and are based on Nokia equipment. Just for the fun of it: I have read that DG also uses Genexis ONTs, is that what you have?

South Hessen here, also on Nokia ONT
Title: Re: 24.7.2 IPv6 woes
Post by: tiermutter on September 05, 2024, 11:00:49 AM
Quote from: CruxtheNinth on September 05, 2024, 10:51:16 AM
South Hessen here, also on Nokia ONT

Just to complete this:
Lower Saxony, here we have both types in the same area, seems like it depends on when it what commissioned... Nokia seems to be the newer one.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 05, 2024, 11:14:03 AM
Quote from: CruxtheNinth on September 05, 2024, 10:50:42 AM
Quote from: franco on September 05, 2024, 08:59:52 AM
good: v20240710, bad v20240820

Bisecting: 7 revisions left to test after this (roughly 3 steps)
[ba9cddfcb6e568a5c83775ed0c917a316acb64db] dhcp6c: fix prototype

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_1.pkg

all green so far with this version
IA_NA directly after reboot, IA_PD a few moments later. RA also fine, route got few minutes after reboot
Will observe a bit longer

EDIT: logs attached; reboot was 10:46

Thanks a lot, here is the next one:

Bisecting: 3 revisions left to test after this (roughly 2 steps)
[6db1c1f5c07c0b53b3ab7671ab52598c4647a54d] common: simplify setloglevel()

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_2.pkg
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 05, 2024, 01:03:48 PM
Quote from: franco on September 05, 2024, 11:14:03 AM
Quote from: CruxtheNinth on September 05, 2024, 10:50:42 AM
Quote from: franco on September 05, 2024, 08:59:52 AM
good: v20240710, bad v20240820

Bisecting: 7 revisions left to test after this (roughly 3 steps)
[ba9cddfcb6e568a5c83775ed0c917a316acb64db] dhcp6c: fix prototype

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_1.pkg



all green so far with this version
IA_NA directly after reboot, IA_PD a few moments later. RA also fine, route got few minutes after reboot
Will observe a bit longer

EDIT: logs attached; reboot was 10:46

Thanks a lot, here is the next one:

Bisecting: 3 revisions left to test after this (roughly 2 steps)
[6db1c1f5c07c0b53b3ab7671ab52598c4647a54d] common: simplify setloglevel()

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_2.pkg


Reboot 12:50
Same as with _1;  but it took approx 10 minutes until the def route for ipv6 was available; probably just usual DG RA shenanigans. However all green since a minute.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 05, 2024, 09:52:37 PM
Bisecting: 1 revision left to test after this (roughly 1 step)
[4bd3f0c78be1683f0a1343af129d829e1a69f8f6] git: 21st century

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_3.pkg
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 05, 2024, 09:56:32 PM
PS: as far as spoilers go this seems to be circling around https://github.com/opnsense/dhcp6c/commit/fe3ed661a5fb9 so it would be neither fix that caused this

the good news in this case would be that it's something simple to fix but the bad news is I don't see what's wrong with these transformations, only that the old code was not even random...
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 06, 2024, 08:38:35 AM
Quote from: franco on September 05, 2024, 09:52:37 PM
Bisecting: 1 revision left to test after this (roughly 1 step)
[4bd3f0c78be1683f0a1343af129d829e1a69f8f6] git: 21st century

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_3.pkg

Now mayhem started (logs are attached)

Rebooted with _3 approx 08:05
- Directly after reboot IA_NA/IA_PD + ipv4 online but DNS was failing weirdly
- restarted unbound which fixed the issue for v4 resolution
- RA received approx 10 minutes later and it seemed ok

next i wanted to replicate the DNS problem and did another reboot with _3

reboot approx 08:23
- directly after reboot no IPv4, no IA_NA, no IA_PD
- ipv4 took a while but still no IA_NA or IA_PD
- few minutes late RA arrived (def route visible for v6) but still no IA_NA or IA_PD
(https://imgur.com/210CaAe)

still on _3 now and so far no working ipv6 in sight.

EDIT: 08:42 now, just did a manual restart of igc0/WAN (via reload button in the UI) and it instantly brought back IA_NA/IA_PD and its working now. (https://imgur.com/t1Hi43v)

EDIT: 08:49 its broken again. IA_NA/IA_PD are gone (added another latest.log zip (5) )

EDIT: 09:31 still broken


some weird race condition??

Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 06, 2024, 11:46:37 PM
Ok, kind of expected that, so last one:

Bisecting: 0 revisions left to test after this (roughly 0 steps)
[fe3ed661a5fb97f9eb9f14d51e7dadcf7a9364bb] dhcp6c: use arc4random_uniform()

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_4.pkg

And thanks a lot for your help so far!


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 07, 2024, 08:32:22 AM
Quote from: franco on September 06, 2024, 11:46:37 PM
Ok, kind of expected that, so last one:

Bisecting: 0 revisions left to test after this (roughly 0 steps)
[fe3ed661a5fb97f9eb9f14d51e7dadcf7a9364bb] dhcp6c: use arc4random_uniform()

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_4.pkg

And thanks a lot for your help so far!


Cheers,
Franco

you are welcome, happy to do some testing.

Behaviour is similar to _3

First reboot with _4 at 08:08:
- Everything fine directly after reboot, in fact it felt like it was never online this quickly. IA_NA/IA_PD and route there.

However, after a second reboot at 08:15, just for verification it started to act like _3.
- link-local fe80 only, no IA_NA or IA_PD BUT the RA was visible
- tried to recover with igc0 reload (log: <27>1 2024-09-07T08:20:26+02:00) which again did not help

its so odd to see the dhcp6c log to actually show that it received both IA_NA and PD but then just nothing happens.

will do another set of reboots. logs are attached

EDIT: 08:39 multiple reboots later, ipv6 staying down.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 07, 2024, 08:45:53 AM
Ok so that's it as unexpectedly expected...

fe3ed661a5fb97f9eb9f14d51e7dadcf7a9364bb is the first bad commit
commit fe3ed661a5fb97f9eb9f14d51e7dadcf7a9364bb
Author: Franco Fichtner <franco@opnsense.org>
Date:   Wed Aug 14 10:39:41 2024 +0200

    dhcp6c: use arc4random_uniform()
   
    No compatibility shim for now.

common.c | 17 +++--------------
common.h |  1 -
config.c |  2 +-
dhcp6c.c | 10 +---------
4 files changed, 5 insertions(+), 25 deletions(-)

https://github.com/opnsense/dhcp6c/commit/fe3ed661a5fb97f9eb9f14d51e7dadcf7a9364bb
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 07, 2024, 08:53:27 AM
So as a safe point for 24.7.4 we can revert this, but we're only about half-way there then

https://github.com/opnsense/dhcp6c/commit/6cb8c154d6

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240820_5.pkg

I honestly don't see the error in the arc4random-related transformations which just seem to provide correct randomness as per design now, not using a pseudo-random number from random() in these cases which bottom line wasn't really random to begin with. We'll have to dissect this commit a bit more but I'm sure we can fix this reliably and securely.

My theory is one of those timers is now giving "unexpected" values that maybe out of spec much more often now than before.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 07, 2024, 08:56:10 AM
This one is probably safe as is, not used anyway: https://github.com/opnsense/dhcp6c/commit/4711abcce51

I'll be back later, need to find an airport. ;)


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 07, 2024, 09:06:22 AM
Nothing really obvious sticks out, but some remote ideas:

1. The srandom() call was omitted. The arc4random documentation from OpenBSD says: "The subsystem is re-seeded from the kernel random(4) subsystem using getentropy(2) on a regular basis, and also upon fork(2)." Maybe one should still call srandom() to initialize arc4random alongside?

2. Also, I do not see how "r = (double)(arc4random_uniform(1000) + 1) / 10000;" could always lead to RAND being "strictly greater than 0 ([RFC3315 17.1.2])" - however, that was not guaranteed to be the case before, then.
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 07, 2024, 10:19:06 AM
_5 seems "stable" so far. In quotation marks because: Had the funny situation where i had working IPv6 but no DHCPv4 for like 3 minutes but it recovered.

Rebooted twice for testing and it still seems fine but weirdly it (feels) all takes a bit longer to converge compared to 24.7.1, had few visual glitches in the UI where interfaces showed the prefix gone and after reload of the browser tab it was there again.

i am trying to replicate that behaviour but i cant anymore. trying not to chase ghosts, will do a few more tests, may have been a local/pebcak glitch

EDIT: 10:30 another reboot and the problems are back. took manual reload of igc0 to get a IA_NA and IA_PD, no RA yet (attached logs for this). 

EDIT2: 10:35 RA received but IA_NA and IA_PD are gone :/ - seems its not stable

EDIT3: All green again. it feels weird.

Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 07, 2024, 11:58:17 AM
Quote from: meyergru on September 07, 2024, 09:06:22 AM
1. The srandom() call was omitted. The arc4random documentation from OpenBSD says: "The subsystem is re-seeded from the kernel random(4) subsystem using getentropy(2) on a regular basis, and also upon fork(2)." Maybe one should still call srandom() to initialize arc4random alongside?

The call is hidden under !HAVE_ARC4RANDOM so it wouldn't compile. But I actually need to put the shims back in order for that define to be created. -.-


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 07, 2024, 03:41:11 PM
@CruxtheNinth so I think you making clear that the last one "somewhat" worked I reverted the other commit that seems to play a role here and this should be the final thing for 24.7.4:

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240907.pkg

I'm actually glad that neither of the two fixes for operational issues we did caused this, but it's still a bit unfortunate there is something weird going on here with the random value generation.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 08, 2024, 09:41:44 AM
Quote from: franco on September 07, 2024, 03:41:11 PM
@CruxtheNinth so I think you making clear that the last one "somewhat" worked I reverted the other commit that seems to play a role here and this should be the final thing for 24.7.4:

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240907.pkg

I'm actually glad that neither of the two fixes for operational issues we did caused this, but it's still a bit unfortunate there is something weird going on here with the random value generation.


Cheers,
Franco

thank you. dhcp6c-20240907.pkg running without any issues so far. still feels like 24.7.1 reconciles a bit faster but i think thats me and just caused by DGs general weird non-standard behaviour.

How come not more people were affected by this?

Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 08, 2024, 04:14:14 PM
I think that DG is very sensitive, but the problem was probably in dhcp6c for decades, if not forever. I made it worse and DG suddenly stopped working. :)

I have the issue here sometimes, but not very reliable. It's probably the same for most.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: FatWightMan on September 08, 2024, 07:33:03 PM
I had the same problem with my ISP WightFibre (https://www.wightfibre.com).  The dhcp6c-20240907.pkg has fixed things for me too.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 08, 2024, 08:28:29 PM
@FatWightMan thanks, appreciate the feedback! 8)
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 08, 2024, 10:51:06 PM
FWIW, I sometimes had slow buildup of IPv6 on DG as well, but this was hit rather seldomly.

Franco, did you fix the "strictly greater than 0" case with that dhcpc version or switched back to random()? Will that new version be in 24.7.4 alongside the no-sa kernel?
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 09, 2024, 06:35:49 AM
Essentially I've only reverted the bulk of the two commits around the arc4random() rework. I'll be dissecting them a bit more and fix whatever seems to come up during that. It may take more time though.

And the no_sa kernel is just for reference. We're going with the same kernel side as 24.7.3 which is half the patches and disabling ND states. FreeBSD is in too deep on this, so eventually they will fix it. That's my hope at least.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: Server07 on September 09, 2024, 07:26:29 AM
Good morning,

since I updated to OPNsense 24.7.3_1-amd64 I have the same issue on not running IPV6DHCP.

Reading through did not see clear the solution so far... do I need to return to older version or did any tweak help? I went through all 5 pages but maybe missed the key sentence, maybe also due to lack of coffee this morning....
Title: Re: 24.7.2 IPv6 woes
Post by: Server07 on September 09, 2024, 07:37:36 AM
the IPv6 should not be local with start fe80... or?

Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 09, 2024, 07:53:16 AM
Quote from: Server07 on September 09, 2024, 07:26:29 AM
Good morning,

since I updated to OPNsense 24.7.3_1-amd64 I have the same issue on not running IPV6DHCP.

Reading through did not see clear the solution so far... do I need to return to older version or did any tweak help? I went through all 5 pages but maybe missed the key sentence, maybe also due to lack of coffee this morning....

to install the  new "fixed" version for testing:

pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240907.pkg

OR

revert to previously known working from 24.7.1

opnsense-revert -r 24.7.1 dhcp6c

reboot afterwards
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 09, 2024, 07:57:35 AM
Quote from: Server07 on September 09, 2024, 07:37:36 AM
the IPv6 should not be local with start fe80... or?

valid but not what you are looking for in this case.
FE80::/10 is link-local and not routed, you are missing a GUA out of 2000::/3

Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 09, 2024, 08:16:59 AM
Quote from: CruxtheNinth on September 09, 2024, 07:57:35 AM
Quote from: Server07 on September 09, 2024, 07:37:36 AM
the IPv6 should not be local with start fe80... or?

valid but not what you are looking for in this case.
FE80::/10 is link-local and not routed, you are missing a GUA out of 2000::/3

No, all of this question is completely out of context and irrelevant. There are literally dozens of threads about this topic.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 09, 2024, 10:42:00 AM
Quote from: franco on September 09, 2024, 06:35:49 AM
And the no_sa kernel is just for reference. We're going with the same kernel side as 24.7.3 which is half the patches and disabling ND states. FreeBSD is in too deep on this, so eventually they will fix it. That's my hope at least.

But there is the new problem with chopped IPv6 packets that doktornotor has reported (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c80) and to which - again - there was no reaction at all. I am not even talking about the one with the sloppy IPv4 TCP states (https://forum.opnsense.org/index.php?topic=42657.msg211988#msg211988) which seems unrelated as it is not fixed by no-sa.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 09, 2024, 10:44:47 AM
At this point I'm not surprised (mostly because I predicted so in the upstream report for the SA regressions), but there will be no wiggle room for 24.7.4 as we try to stabilise from this and dhcp6c issues being found at the same time. I can't have this go back and forth all the time for each release as it masquerades where actual IPv6 issues are coming from.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 09, 2024, 10:51:44 AM
Oh, so 24.7.4 seems close. I guess, there is always more work to be done... no objections (although I already regret my former advice: looking back I would now rather go with the full SA revert).
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 09, 2024, 10:52:45 AM
We can still do that. My plan is to test the amended SA as it comes out. If it's still whacky we go full revert. Ok?


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: Server07 on September 09, 2024, 06:11:42 PM
Quote
to install the  new "fixed" version for testing:

pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240907.pkg

OR

revert to previously known working from 24.7.1

opnsense-revert -r 24.7.1 dhcp6c

reboot afterwards

Many thanks - did the revert and WAN has now proper 2a00: .... address.
AdguardHome starting after upgrade - see other thread and crash due to service -yy ...



Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 09, 2024, 09:04:10 PM
Can we move the off-topic questions somewhere else please?

As far as the SA goes FreeBSD core responded making this our problem so 24.7.4 will scrub the whole SA effort and we can continue to build on good IPv6 connectivity...


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 13, 2024, 07:27:48 PM
Franco, I do not know if you followed the separate FreeBSD report.

You do not need to read all of that, just look at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=281395#c26 and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=281395#c27.

I wonder if some or all reports of this were on Proxmox VMs and were not related to the SA at all. After the Proxmox setting was changed, I could not reproduce the ND problems on FreeBSD 14.1-RELEASE or FreeBSD 14.1-STABLE any more.

But maybe it is just my inability to reproduce the correct environment to bring the problem to life under plain FreeBSD.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 13, 2024, 09:44:30 PM
I confirmed weird ND behaviour on the first patch of the SA set, namely pf not allowing the ND to be answered which ended up in two excess retransmissions. This was on plain hardware between OPNsense and a FritzBox.

As stated in the other patch disabling the state tracking brought the behaviour back to normal even with the patches applied. The impractical thing is that disabling state tracking for ICMP even by rule will resurface the SA.

Commit https://github.com/openbsd/src/commit/2633ae8c4c8a64 makes this very clear "Fixes a bug uncovered by one of the previous commits that virtually breaks IPv6 connectivity after few minutes of use.". It was eventually pulled in after claiming conspiracy but the SA was never amended so far. I feel this has never been our battle to begin with especially within FreeBSD release engineering practice that has never had such a negative impact on a supported release.

I understand the sentiment that this is our problem now. It made the decision to avoid the bad code completely much easier. If it was the right call on the FreeBSD end I'm in doubt given the technical evidence, but if they want to go by verifiable test case that is certainly fair. But not now, not at the will of a third party that apparently does not care if FreeBSD releases are reliable or not.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: Patrick M. Hausen on September 13, 2024, 09:52:07 PM
I am willing to buy you and kp@ any number of beers or other drinks provided that both of you sit down at a table with me at the same time  :)
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 13, 2024, 11:28:12 PM
No need, we certainly all made our own choices.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 14, 2024, 01:55:45 AM
I also know that I witnessed the ND problem on OpnSense bare metal, so I have no real doubt of what is going on.

I underestimated the difficulty to simulate the needed pf rules on a pure FreeBSD basis just to prove a point by "playing by their rules". I now can fully understand that you lack the enthusiasm to do that.

Do you expect that FreeBSD will at some point fix their code? If I had not filed those two bugs, they could simply have dropped all that.

BTW: When I saw this thread (https://forum.opnsense.org/index.php?topic=42770), I already thought that BSI had included CVE-2024-6640 in the list of 11 (!) CVEs contained in that advisory (as luck would have it, it was not).

It is scandalous how these guys bundle up a few CVEs and use the highest CVSS score for one combined advisory, emitting a false sense of urgency. Going by that rule, one would have to fix each and every small problem and creating huge usability concerns. Considerations like the one to revert the SA to restore operation were impossible like that.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 14, 2024, 08:50:28 AM
Quote from: meyergru on September 14, 2024, 01:55:45 AMI underestimated the difficulty to simulate the needed pf rules on a pure FreeBSD basis just to prove a point by "playing by their rules". I now can fully understand that you lack the enthusiasm to do that.

Do you expect that FreeBSD will at some point fix their code? If I had not filed those two bugs, they could simply have dropped all that.

My point is that I asked the researcher how much he was involved in the review and test of the SA. He told me he was not involved at all. He didn't even know it caused regressions until I told him.

The testing on this has been low effort, the patch size and age is high risk. It's basically a feature that was added into all supported FreeBSD release on SA/SO fast trck which to my knowledge never happened before. Please correct me if I'm wrong. I'm not overly fond of the strict errata policy (not fixing kernel panics) in supported FreeBSD releases, but it's a workable given that also nothing ever gets worse.

For now we wait and see what happens when this hits pfSense. When we have to deal with 14.2 inclusion we will add more patches from OpenBSD and reassess; and I'll try to chase relevant OpenBSD devs on the subject at EuroBSDCon.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 17, 2024, 10:58:17 AM
@CruxtheNinth back to the topic at hand :)

I've added

https://github.com/opnsense/dhcp6c/commit/9c42063ba5
https://github.com/opnsense/dhcp6c/commit/0a714091cb

Which should be safe changes:

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240907_1.pkg

Now what I think is a problem is the following...

https://github.com/opnsense/dhcp6c/blob/master/dhcp6c.c#L157-L159

You can see someone tried to be clever and discard random() use when arc4random() is available, but...

# git grep '[^s4]random('
common.c:               ev->retrans = (random() % (SOL_MAX_DELAY));
common.c:                       r = (double)((random() % 1000) + 1) / 10000;
common.c:                       r = (double)((random() % 2000) - 1000) / 10000;
missing/arc4random.c: * a stub function to make random() to return good random numbers.

There's all of the timer calculations in common.c that still use random() without srandom() ever seeding the RNG.

Why can this be a problem?

Because the values produced are likely not random and probably not uniformly distributed either. Changing it all to a consistent arc4random() use seems to cause the weird renewal issues seen before.

We will take a closer look, but for now I'd just like to know if the current build presented above doesn't make this worse.


Thanks,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 17, 2024, 11:34:50 AM
Okay I found the issue I think:

random() returns signed type long, arc4random*() returns unsigned type int.

This underflows during (random - 1000) as the calculation tries to return values between -0.1 and 0.1 causing values such as this instead:

RAND -0.1 <= 429926.6292 <= 0.1

Which will get the timer stuck or overflow itself as well.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 17, 2024, 12:23:00 PM
https://github.com/opnsense/dhcp6c/commit/99826a37fe

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/dhcp6c-20240907_2.pkg

That should be it... :)
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 20, 2024, 10:30:42 AM
Someone mind testing this second revision above? Ideally it's 24.7.5 material for next week but I would like to be sure.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: marjohn56 on September 20, 2024, 12:09:54 PM
I'm going to load it now. Seeing some weird IPv6 issues.


I'm running 25.1*. Yesterday I rebooted, everything came up OK, ten minutes later IPv6 stopped working, interfaces all have the correct addresses it just appears there's no route. dpinger is still saying all good. BTW, my ISP runs a five minute lease renewal, even though the address ranges are reserved!


I then killed and restarted dhcp6c and hey presto IPv6 started working and it's been working fine for 24 hours. I'll load the test dhcp6c and reboot, and see what happens.

++++


Same thing, five minutes after a reboot no route, restart dhcp6c all good again; may not be dhcp6c related but it's a pita. The only good thing is it appears I can cause the problem just by restarting opnsense.

Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 20, 2024, 12:23:24 PM
I tried it and at least for my two installations (one DHCPv6 on DG, one PPPoE on M-Net), there seem to be no regressions.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 22, 2024, 09:50:07 AM
@Martin can you try this one? https://forum.opnsense.org/index.php?topic=42556.msg213403#msg213403

@Uwe I may want to rewrite this to fixed point arithmetic later since we're only dealing with microsecond resolution in the timer and the fractions are actually much more coarse and calculated in millisecond resolution. But for now I think we should go ahead in 24.7.5 with this change.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: marjohn56 on September 22, 2024, 10:38:40 AM
I'd already tried that one. I'm leaning to it not being dhcp6c per say but some sort of race condition at boot. Killing dhcp6c and restarting it restores things. If it was dhcp6c I would expect the issue to happen again, it doesn't. I think because my ISP gives me a stupidly short lease time only 300 seconds that may be part of the issue. I'll play and see what I can find out.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 22, 2024, 12:01:50 PM
Let me have the dhcp6c debug log and I'll take a closer look.


Thanks,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 22, 2024, 12:10:29 PM
For me, the _3 version causes no regressions, either.
Title: Re: 24.7.2 IPv6 woes
Post by: marjohn56 on September 22, 2024, 11:09:58 PM
Quote from: franco on September 22, 2024, 12:01:50 PM
Let me have the dhcp6c debug log and I'll take a closer look.


Thanks,
Franco


The logs for dhcp6c look normal, all appears OK. As I said, restarting dhcp6c manually restores IPv6 functionality. However I've just done a netstat -r and there's no default route, restarting dhcp6c restores the default and it will stay that way until I reboot. Looking deeper now.




+++


So, the default route is set at boot and all is normal, the default route then disappears from the table at around 5 minutes after boot. Manually adding the route from the shell and leaving dhcp6c alone also restores IPv6 and then we are all good again.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 23, 2024, 09:54:08 AM
If you share the system log with me I can take a separate look. We've been talking about recent issues and to some degree it appears that ISPs have acquired new server behaviour clustering around 24.7.x release timing because the issues we see now are mid-protocol and hard to influence with networking/firewalling changes around it and dhcp6c hasn't exhibited such behaviour in many years before. It's all a bit of a mystery.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: marjohn56 on September 23, 2024, 02:11:54 PM
Here you go, dhcp6c seems OK to me. Something is removing the default route.


If I restart rtsold that will restore the route for a few minutes before it loses it again. If I restart dhcp6c then the route restores and holds, if I manually add the default route that also holds. Odd, very odd.
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 23, 2024, 05:25:08 PM
Probably RAs that change/delete the route? I have seen ISPs send out RAs despite managed IPv6.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 24, 2024, 02:36:49 PM
The dhcp6c looks good. The server always answers and gives you a PD. SLAAC/RA may be interfering as Uwe noted because DHCPv6 needs that to stay connected. The core does not appear to remove the route.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 24, 2024, 02:42:18 PM
But we don't force reloading the routes on RENEW either because ideally they would still be there.

I'm assuming this fixes the issue?

# /usr/local/etc/rc.routing_configure

So that would be:

https://github.com/opnsense/core/blob/8684443b6/src/etc/inc/interfaces.inc#L2957-L2963

But as noted the default route shouldn't be removed. DHCPv6 doesn't handle routes. We only do it "by accident".


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: marjohn56 on September 24, 2024, 04:42:50 PM
It's definitely NOT dhcp6c, I had already replaced the current version with one from quite a while back and the same thing happens. It's definitely a weirdo. I'll have a play this evening when I have the house to myself and see if I can narrow it down a bit more.



Title: Re: 24.7.2 IPv6 woes
Post by: marjohn56 on September 24, 2024, 04:45:56 PM
Quote from: franco on September 24, 2024, 02:42:18 PM
But we don't force reloading the routes on RENEW either because ideally they would still be there.

I'm assuming this fixes the issue?

# /usr/local/etc/rc.routing_configure

So that would be:

https://github.com/opnsense/core/blob/8684443b6/src/etc/inc/interfaces.inc#L2957-L2963 (https://github.com/opnsense/core/blob/8684443b6/src/etc/inc/interfaces.inc#L2957-L2963)

But as noted the default route shouldn't be removed. DHCPv6 doesn't handle routes. We only do it "by accident".


Cheers,
Franco


I'll play later and let you know on this. But why is the default being removed is the question..
Title: Re: 24.7.2 IPv6 woes
Post by: marjohn56 on September 24, 2024, 05:12:38 PM
Quote from: franco on September 24, 2024, 02:42:18 PM
But we don't force reloading the routes on RENEW either because ideally they would still be there.

I'm assuming this fixes the issue?

# /usr/local/etc/rc.routing_configure

So that would be:

https://github.com/opnsense/core/blob/8684443b6/src/etc/inc/interfaces.inc#L2957-L2963 (https://github.com/opnsense/core/blob/8684443b6/src/etc/inc/interfaces.inc#L2957-L2963)

But as noted the default route shouldn't be removed. DHCPv6 doesn't handle routes. We only do it "by accident".


Cheers,
Franco


Yep, that restores it.

Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 24, 2024, 09:03:46 PM
I'm 99% sure the default route is stripped by a zero lifetime RA which the kernel reacts to by removing its default route and you can't even see it.

# ndp -r

should reveal that default route and if it's gone afterwards.

Funnily enough I just saw there is a "ndp -I xxx" which can set a default route via one particular interface if the route disappears for any reason but I've never tried it.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: marjohn56 on September 24, 2024, 11:29:09 PM
Quote from: franco on September 24, 2024, 09:03:46 PM
I'm 99% sure the default route is stripped by a zero lifetime RA which the kernel reacts to by removing its default route and you can't even see it.

# ndp -r

should reveal that default route and if it's gone afterwards.

Funnily enough I just saw there is a "ndp -I xxx" which can set a default route via one particular interface if the route disappears for any reason but I've never tried it.


Cheers,
Franco


Interesting... I see this:
fe80::8ff:fe61:e32b%igb0 if=igb0, flags=MO, pref=medium, expire=2m59s
So that explains it.
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 24, 2024, 11:41:04 PM
Quote from: marjohn56 on September 24, 2024, 11:29:09 PM
Interesting... I see this:
fe80::8ff:fe61:e32b%igb0 if=igb0, flags=MO, pref=medium, expire=2m59s
So that explains it.

Usually, the expiry is set to 30 minutes and the next update comes before that. Would be interesting if there is an invalidation or just a missing renewal.
Title: Re: 24.7.2 IPv6 woes
Post by: marjohn56 on September 25, 2024, 01:40:46 AM
Now I'm completely baffled. I rebooted again and sat there running the ndp -r command and watching the timer countdown and reset, and reset, and reset....
It's working fine now.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 25, 2024, 08:13:33 AM
I tried the ndp -I <wanif> but it doesn't work when removing the default route so I guess that's why nobody uses it...


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: meyergru on September 25, 2024, 10:01:56 AM
A watched pot never boils...  8)
Title: Re: 24.7.2 IPv6 woes
Post by: marjohn56 on September 25, 2024, 11:37:34 PM
24 hours later and still solid.... bloody gremlins.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 26, 2024, 08:17:50 AM
Should have stayed in their planes is what I always tell them.


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 27, 2024, 05:00:16 PM
just upgraded to 24.7.5 and the old problems are back
IA_PD is visible, IA_NA is there but the default route is never showing up (which in effect kills almost all clients as the PD is available and addresses are getting assigned, where all ipv6 lookups will fail)

downgraded to old dhcp6c and everything works again.

EDIT: would it be possible to keep back the allocated Prefix from radvd and dhcpd processes unless there is a RA received? that would at least not kill the whole network due to ipv6 having a higher priority on clients

EDIT2: reverted fully to 24.7.4_1 as there was additional packetloss not only pinging the lan interface but also transit i.e pinging 8.8.8.8 or 1.1.1.1 - packetloss is gone with 24.7.4_1 (and ipv6 is also still stable now)
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 30, 2024, 07:57:04 AM
should we start a new thread for 24.7.5 dhcp6c regression or move to github issues?
also not sure what to make out of the timeouts, i dont have time the next days to upgrade again to 24.7.5 trying to replicate it.
Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 30, 2024, 08:01:06 AM
To be honest I'm not sure what's going on now. It feels like every step good or bad now we have random issues after doing the correct verification process for each step.

Perhaps it makes sense to use the currently installed 24.7.4 with dhcp6c from 24.7.5 first:

# opnsense-revert -r 24.7.5 dhcp6c


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on September 30, 2024, 08:07:45 AM
Thanks, Franco. I will try that as soon as i find some more time for it.
I wanted to spin up a fresh installation of 24.7.5 on another box for testing as well.

Will update asap.

Title: Re: 24.7.2 IPv6 woes
Post by: franco on September 30, 2024, 08:11:12 AM
Thanks for all your help, BTW. Highly appreciated.  :)


Cheers,
Franco
Title: Re: 24.7.2 IPv6 woes
Post by: CruxtheNinth on October 02, 2024, 07:37:32 AM
I upgraded my N100 Box to 24.7.5 this morning and as for right now everything seems stable.

The packet loss i believed from 24.7.5 is in my case seems to be caused by MacOS 15 / Sequoia. Regardless of which ICMP target, i have 0.3 to 0.9% packet loss. Sometimes this happens only after 100+ packets, so it was a coincidence that i saw it on 24.7.5 and not on 24.7.4. Kindly ignore my previous report about it (but maybe keep in mind if more people complain about sudden loss, i could replicate this across 2 updated M1 Macs but my work laptop with Sonoma does not have this issue and neither none of my Win11 systems)

As for IPv6, this seems fine right now but it took much longer for the default route to show up, when looking into the logs what happens directly after the reboot it puzzles me a bit:

Reboot was around 06:55:42.

At 6:56:22 i see

Code Select
<29>1 2024-10-02T06:56:18+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="166"] send request to ff02::1:2%igc0
<29>1 2024-10-02T06:56:18+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="167"] reset a timer on igc0, state=REQUEST, timeo=0, retrans=932
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="168"] Sending Request
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="169"] set client ID (len 14)
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="170"] set server ID (len 14)
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="171"] set IA address
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="172"] set identity association
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="173"] set elapsed time (len 2)
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="174"] set option request (len 4)
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="175"] set IA_PD prefix
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="176"] set IA_PD
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="177"] send request to ff02::1:2%igc0
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="178"] reset a timer on igc0, state=REQUEST, timeo=1, retrans=1786
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="179"] receive reply from fe80::ff:fe01:101%igc0 on igc0
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="180"] get DHCP option client ID, len 14
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="181"]   DUID: 00:01:00:01:2d:7a:03:f4:7c:83:34:be:41:a3
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="182"] get DHCP option server ID, len 14
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="183"]   DUID: 00:01:00:01:26:2c:40:77:00:50:56:b1:8a:d7
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="184"] get DHCP option identity association, len 40
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="185"]   IA_NA: ID=0, T1=1800, T2=2880
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="186"] get DHCP option IA address, len 24
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="187"]   IA_NA address: 2a00:6020:1000:40::518d pltime=3600 vltime=3600
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="188"] get DHCP option DNS, len 32
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="189"] get DHCP option IA_PD, len 41
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="190"]   IA_PD: ID=0, T1=1800, T2=2880
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="191"] get DHCP option IA_PD prefix, len 25
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="192"]   IA_PD prefix: 2a00:6020:5051:7d00::/56 pltime=3600 vltime=3600
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="193"] Received REPLY for REQUEST
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="194"] nameserver[0] 2a00:6020:100::1
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="195"] nameserver[1] 2a00:6020:200::1
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="196"] make an IA: PD-0
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="197"] create a prefix 2a00:6020:5051:7d00::/56 pltime=3600, vltime=3600
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="198"] add an address 2a00:6020:5051:7d00:7e83:34ff:febe:41a4/64 on igc1
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="199"] make an IA: NA-0
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="200"] create an address 2a00:6020:1000:40::518d pltime=3600, vltime=3600
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="201"] add an address 2a00:6020:1000:40::518d/128 on igc0
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="202"] removing an event on igc0, state=REQUEST
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="203"] removing server (ID: 00:01:00:01:26:2c:40:77:00:50:56:b1:8a:d7)
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="204"] executes /var/etc/dhcp6c_wan_script.sh
<13>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 60365 - [meta sequenceId="205"] dhcp6c_script: REQUEST on igc0 executing
<13>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 64486 - [meta sequenceId="206"] dhcp6c_script: REQUEST on igc0 renewal
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="207"] script "/var/etc/dhcp6c_wan_script.sh" terminated
<29>1 2024-10-02T06:56:19+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="208"] got an expected reply, sleeping.

this should also contain the RA but the def route via fe80::ff:fe01:101 never makes into the routing table.

It takes till 07:02 till the route shows up:
Code Select
29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="108"] set client ID (len 14)
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="109"] set server ID (len 14)
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="110"] set IA address
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="111"] set identity association
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="112"] set elapsed time (len 2)
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="113"] set option request (len 4)
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="114"] set IA_PD prefix
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="115"] set IA_PD
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="116"] send request to ff02::1:2%igc0
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="117"] reset a timer on igc0, state=REQUEST, timeo=2, retrans=4304
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="118"] receive reply from fe80::ff:fe01:101%igc0 on igc0
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="119"] get DHCP option client ID, len 14
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="120"]   DUID: 00:01:00:01:2d:7a:03:f4:7c:83:34:be:41:a3
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="121"] get DHCP option server ID, len 14
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="122"]   DUID: 00:01:00:01:26:2c:40:77:00:50:56:b1:8a:d7
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="123"] get DHCP option identity association, len 40
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="124"]   IA_NA: ID=0, T1=1800, T2=2880
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="125"] get DHCP option IA address, len 24
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="126"]   IA_NA address: 2a00:6020:1000:40::518d pltime=3600 vltime=3600
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="127"] get DHCP option DNS, len 32
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="128"] get DHCP option IA_PD, len 41
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="129"]   IA_PD: ID=0, T1=1800, T2=2880
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="130"] get DHCP option IA_PD prefix, len 25
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="131"]   IA_PD prefix: 2a00:6020:5051:7d00::/56 pltime=3600 vltime=3600
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="132"] Received REPLY for REQUEST
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="133"] nameserver[0] 2a00:6020:100::1
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="134"] nameserver[1] 2a00:6020:200::1
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="135"] make an IA: PD-0
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="136"] create a prefix 2a00:6020:5051:7d00::/56 pltime=3600, vltime=3600
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="137"] add an address 2a00:6020:5051:7d00:7e83:34ff:febe:41a4/64 on igc1
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="138"] make an IA: NA-0
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="139"] create an address 2a00:6020:1000:40::518d pltime=3600, vltime=3600
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="140"] add an address 2a00:6020:1000:40::518d/128 on igc0
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="141"] removing an event on igc0, state=REQUEST
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="142"] removing server (ID: 00:01:00:01:26:2c:40:77:00:50:56:b1:8a:d7)
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="143"] executes /var/etc/dhcp6c_wan_script.sh
<13>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 74159 - [meta sequenceId="144"] dhcp6c_script: REQUEST on igc0 executing
<13>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 80208 - [meta sequenceId="145"] dhcp6c_script: REQUEST on igc0 renewal
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="146"] script "/var/etc/dhcp6c_wan_script.sh" terminated
<29>1 2024-10-02T07:02:52+02:00 opnframe.epp.home.arpa dhcp6c 69302 - [meta sequenceId="147"] got an expected reply, sleeping.
<13>1 2024-10-02T07:02:55+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="148"] /usr/local/etc/rc.newwanipv6: IP renewal starting (address: 2a00:6020:1000:40::518d, interface: wan, device: igc0)
<13>1 2024-10-02T07:02:55+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="149"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
<13>1 2024-10-02T07:02:55+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="150"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
<11>1 2024-10-02T07:02:56+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="151"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned
exit code '3', the output was 'daemon: process already running, pid: 78043'
<13>1 2024-10-02T07:02:56+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="152"] /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using wan, lan
<13>1 2024-10-02T07:02:56+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="153"] /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
<13>1 2024-10-02T07:02:56+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId="154"] /usr/local/etc/rc.newwanipv6: ROUTING: keeping inet6 default route to fe80::ff:fe01:101%igc0
<13>1 2024-10-02T07:02:56+02:00 opnframe.epp.home.arpa opnsense 80564 - [meta sequenceId=



right now everything is fine, i dont know why the route is showing up today but did not during the last update.

full log in attachment



Text only | Text with Images