24.7.2 IPv6 woes

[meyergru]

I tried it and at least for my two installations (one DHCPv6 on DG, one PPPoE on M-Net), there seem to be no regressions.

[franco]

@Martin can you try this one? https://forum.opnsense.org/index.php?topic=42556.msg213403#msg213403

@Uwe I may want to rewrite this to fixed point arithmetic later since we're only dealing with microsecond resolution in the timer and the fractions are actually much more coarse and calculated in millisecond resolution. But for now I think we should go ahead in 24.7.5 with this change.


Cheers,
Franco

[marjohn56]

I'd already tried that one. I'm leaning to it not being dhcp6c per say but some sort of race condition at boot. Killing dhcp6c and restarting it restores things. If it was dhcp6c I would expect the issue to happen again, it doesn't. I think because my ISP gives me a stupidly short lease time only 300 seconds that may be part of the issue. I'll play and see what I can find out.

[franco]

Let me have the dhcp6c debug log and I'll take a closer look.


Thanks,
Franco

[meyergru]

For me, the _3 version causes no regressions, either.

[marjohn56]

Let me have the dhcp6c debug log and I'll take a closer look.


Thanks,
Franco

The logs for dhcp6c look normal, all appears OK. As I said, restarting dhcp6c manually restores IPv6 functionality. However I've just done a netstat -r and there's no default route, restarting dhcp6c restores the default and it will stay that way until I reboot. Looking deeper now.




+++


So, the default route is set at boot and all is normal, the default route then disappears from the table at around 5 minutes after boot. Manually adding the route from the shell and leaving dhcp6c alone also restores IPv6 and then we are all good again.

[franco]

If you share the system log with me I can take a separate look. We've been talking about recent issues and to some degree it appears that ISPs have acquired new server behaviour clustering around 24.7.x release timing because the issues we see now are mid-protocol and hard to influence with networking/firewalling changes around it and dhcp6c hasn't exhibited such behaviour in many years before. It's all a bit of a mystery.


Cheers,
Franco

[marjohn56]

Here you go, dhcp6c seems OK to me. Something is removing the default route.


If I restart rtsold that will restore the route for a few minutes before it loses it again. If I restart dhcp6c then the route restores and holds, if I manually add the default route that also holds. Odd, very odd.

[meyergru]

Probably RAs that change/delete the route? I have seen ISPs send out RAs despite managed IPv6.

[franco]

The dhcp6c looks good. The server always answers and gives you a PD. SLAAC/RA may be interfering as Uwe noted because DHCPv6 needs that to stay connected. The core does not appear to remove the route.


Cheers,
Franco

[franco]

But we don't force reloading the routes on RENEW either because ideally they would still be there.

I'm assuming this fixes the issue?

# /usr/local/etc/rc.routing_configure

So that would be:

https://github.com/opnsense/core/blob/8684443b6/src/etc/inc/interfaces.inc#L2957-L2963

But as noted the default route shouldn't be removed. DHCPv6 doesn't handle routes. We only do it "by accident".


Cheers,
Franco

[marjohn56]

It's definitely NOT dhcp6c, I had already replaced the current version with one from quite a while back and the same thing happens. It's definitely a weirdo. I'll have a play this evening when I have the house to myself and see if I can narrow it down a bit more.

[marjohn56]

But we don't force reloading the routes on RENEW either because ideally they would still be there.

I'm assuming this fixes the issue?

# /usr/local/etc/rc.routing_configure

So that would be:

https://github.com/opnsense/core/blob/8684443b6/src/etc/inc/interfaces.inc#L2957-L2963

But as noted the default route shouldn't be removed. DHCPv6 doesn't handle routes. We only do it "by accident".


Cheers,
Franco

I'll play later and let you know on this. But why is the default being removed is the question..

[marjohn56]

But we don't force reloading the routes on RENEW either because ideally they would still be there.

I'm assuming this fixes the issue?

# /usr/local/etc/rc.routing_configure

So that would be:

https://github.com/opnsense/core/blob/8684443b6/src/etc/inc/interfaces.inc#L2957-L2963

But as noted the default route shouldn't be removed. DHCPv6 doesn't handle routes. We only do it "by accident".


Cheers,
Franco

Yep, that restores it.

[franco]

I'm 99% sure the default route is stripped by a zero lifetime RA which the kernel reacts to by removing its default route and you can't even see it.

# ndp -r

should reveal that default route and if it's gone afterwards.

Funnily enough I just saw there is a "ndp -I xxx" which can set a default route via one particular interface if the route disappears for any reason but I've never tried it.


Cheers,
Franco