IPv6 stops routing a few minutes after boot

[gazd25]

Hi All,

Firstly, I'd like to thank everybody for their sterling work on OPNsense, people like me would be much worse off without it, so thank you very much to all contributors.

I've been refining my OPNsense config for some considerable time and while it's now relatively complex, I have reached a very positive place with pretty much everyting I want working correctly.

I have been having a minor problem for some time, think it actually started back in the times 23.1 release which was around the time I first deployed IPv6 on my network. It's more of a niggle than a serious issue but the ability to replicate the fault fairly consistently does suggest a potential timing issue in the code at boot that might be responsible.

My system publishes IPv6 /56 from my ISP using track interface on my LAN network to a /64 internally, DHCPv6 sends my prefix from the ISP and they dont publish me an IP so an autoassigned one is set, but this is relatively normal and all traffic routes and works as expected.

The problem comes in that after boot up the system will be working and routing IPv6 correctly, then an unknown number of minutes later, for some reason will stop routing. When this happens, I'll go to the dashboard interface and restart the routing service manually and it'll start routing IPv6 again and until I reboot next time, it'll continue working as expected.

I would say the above occurs maybe 9/10 boots and occassionally for a reason I also cant define it simply continues to work as expected.

I'm hoping one of the experts here can help me get to the bottom of the root cause and fix and happy to collect logs, and test as needed since i run OPNsense in a VM with easy snapshot and rollback capability.

Many thanks

Gareth

[gazd25]

I forgot to add, I'm currently running the latest stable release 24.1.6, though this issue has persisted for a long time throughout a fair number of updates.

Thanks

Gareth

[gazd25]

Just done a little further testing and important to note that when the IPv6 routing fails, I have also tried to ping from an IPv6 interface on the OPNsense firewall to an internet IPv6 address and the error I see is "No Route to Host"

Restarting routing service then allows the same ping set in the diagnostics from the firewall host to the Internet address to succeed with no losses so while I thought maybe it was just routing through from the LAN side to the internet that was failing, it appears the firewall host itself also cannot route to the IPv6 internet while in this state.

Thanks

Gareth

[opnfwb]

I've noticed something similar with my fiber provider. I think the issue is the provider's PD doesn't have a valid monitor address or has high packet loss. For instance, if I leave gateway monitoring enabled for my IPV6 WAN route I can see a high level of packet loss come and go just on the fe80% IP that gets discovered during the ISP handing out the PD.

This resolved my issue and resulted in stable IPv6. Again I'll caution that these settings might not be for everyone but this is what fixed my issue with some trial/error.

First you'll need to go to system/gateways/configuration and edit the WAN DHCP6 gateway. By default OPNsense has gateway monitoring disabled, enable it and you'll want to set a known good WAN IPv6 IP address. I like to use DNS servers like CloudFlare, Google, or Quad9 since they are anycast and always reachable if the WAN is up. I've attached a screenshot showing how I've configured my IPv6 WAN gateway.

I would also recommend doing something similar with your WAN_DHCP ipv4 gateway if you haven't already. It's okay to leave that IP to the ISP assigned WAN gateway (leave it blank and it will use the ISP gateway) as that usually always works. Enabling gateway monitoring for both of these will give you the "quality" graphs under Reporting/Health/Quality. Not only will you be able to see your average ping time across both gateways but you'll also be able to check if you're getting packet loss, which is quite handy.

Try these and see if your ipv6 stabilizes.

[gazd25]

Thanks Opnfwb, I'll give this a try when I get a minute free this weekend

[gazd25]

I've added the settings as per your advice Opnfwb, but using one of Cloudflares IPv6 addresses as my monitor: 2606:4700:4700::1111.

My IPv4 was already using gateway monitoring so no need to set anything there.

I can already see the upstream gateway monitoring of IPv6 changes dynamically in the dashboard at each reboot, similarly to the way IPv4 always has, I'm guessing based on what is dynamically handed out during the PD.

I'll monitor and let you know the outcome.

Many thanks again for the help.

[gazd25]

Initial indications are that the changes I've applied similar to yours Opnfwb don't seem to have resolved the issue, it still occurs intermittently.

That said I now have monitoring of it, so was thinking maybe as a workaround to use Monit to restart the routing service if it detects a drop of the IPv6 gateway for longer than a certain time.

Looking in to that now to see if possible.

[Wirehead]

Do you have shared forwarding enabled or disabled?

(Firewall -> settings -> advanced)

[opnfwb]

I do have shared forwarding enabled.

[gazd25]

I also have shared forwarding enabled on my config.

I've tried switching to using Quad 9 as my monitor IP like yours Opnfwb, cant see it'd make much difference, but no harm in trying and not able to reboot today for testing so will resume testing soon.

Thanks for the help guys.

[franco]

99.9% have shared forwarding on, because it's the default?

I would rather suspect some RA coming in and advertising a wrong default route or something.


Cheers,
Franco

[gazd25]

Hi Franco,

I had thought similar, but then had ruled the RA out because the issue occurs when testing pings from the firewall itself when IPv6 stops routing.

During earlier testing, when I noticed IPv6 had stopped routing I went on to the firewall and set up a ping from the interface diagnostics to Cloudflare DNS IPv6 address and the error I saw is "No route to host" after this I restarted the routing service manually from the dashboard and ran the same ping from interface diagnostics from the firewall and it worked correctly with no loss as well as all IPv6 LAN traffic now forwarding too.

Or do you mean an RA from the ISP side? Not sure I can do much about that other than what I'm already doing to workaround

Is there a way i could tell in the logs if another RA is being recieved from the ISP?

When you were helping me resolve a previous issue with IPv6 after being forced to change from setting a static IPv6 address to dynamic, as I recall it became obvious my ISP weren't even sure what servers were giving the DHCPv6 info/prefix out to me because they hadn't set any up, only to find that it was a set of upstream Entanet/Cityfibre servers that were doing it after investigation.

Thanks

Gareth

[barold]

Hi all,

I've been following this thread since its beginning since I have IPv6 intermittent issues. My ISP is Xfinity. I have Monit email me when its ping6 test fails. For the most part it's stable. That is, for a given boot of my router, IPv6 will either work till the next boot or not work at all.

Whether it works or not, I have the same details when I run "ifctl -O -i igc1". Even when it's not working, I can still ping6 the Xfinity gateway (a link-local address). I cannot ping6 to any host I've tried on the internet; I can't even ping the Xfinity name server.

My conclusion has been that it's an Xfinity issue. Maybe when they receive my DHCP request they do some setup and there's some variability in that. My single call to Xfinity support was not helpful; the support person didn't know anything about the topic and refused to escalate my issue.

I'm just adding those details in case it sheds any light, even though my problem seems different.

What happens when one restarts routing from the dashboard? Is it a combination of commands from action_interface.conf? Does that make a new DHCP request to my ISP? I'm trying to figure out the most light-weight action that could trigger Xfinity to perform its setup again.

[andreaslink]

Hej, just throwing in your virtualization. Are you running your OPNsense VM on Proxmox? There is a known issue (I am trying to find a solution for as well) where Proxmox looses IPv6 routing/connectivity after ~20 minutes from and to all guests. I have tried several parameters, but nothing fixes the issue - testing is also cumbersome, as this requires a lot of waiting after each reboot to see it fail again. Every reboot fixes the issue, for the next ~20 minutes.
So, if you are not running on bare metal, you might search for your issue on Proxmox and not on OPNsense.