IPv6 Selective Routing Failure

[alias_neo]

Hi folks.

I've been using OPNSense for a few years now and have been using IPv6 on and off with my ISP, YouFibre in the UK. I didn't have issues in the past with the same router and ISP.

When I reorganised my network I disabled IPv6 and have just gotten around to trying to set it up again because I needed IPv6 to test some connectivity for a service I'm running on the web.

I haven't really had setup issues in the past and it has just configured fine, but now I'm having selective routing failures. I've been through a whole range of debugging, but at this stage I don't know what I don't know, so it's hard to say if what appears to me to be a routing issue somewhere inside my ISP is actually a misconfiguration on my end.

Here's a quick summary of what's going on:

If I ping -6 cloudflare DNS, it works as expected, if I ping -6 Quad9 DNS address, it also works, if I ping -6 my blog, which sits behind cloudflare it also works. If I connect to them in my browser, they all work. If I go to one of those "ipv6 only site" lists, I can connect to roughly half and the other half fail.

When I try to run ipv6 test on the web at test-ipv6.com it says I have no ipv6 address. Other ipv6 test sites fail the same way. My OPNSense _is_ giving my devices proper addresses, they _do_ have IPv6, and some sites do work.

If I try to ping the ipv6 addresses of Google DNS they fail, as does attempting to connect to ipv6.google.com. I have a VPS in Digital Ocean and I also can't connect to it on IPv6 (the VPS is about a decade old and I have been able to connect to it with IPv6 in the past).

Having ruled out DNS, my searches suggested it could be an MTU/PMTU issues. This is where my knowledge starts to fail;

I've set the WAN MTU to 1400, my Linux deskop that I'm testing from I have tried configuring MTU of 1300 and 1400 to no effect. Nothing MTU related appears to have an effect, so I've ruled that out, albeit not confidently. I also tried clamping MSS on the WAN to small values such as 1300.

I've run some traceroutes that my ISP asked for, they seem to fail and show asymmetry after about 4-6 hops when trying to hit any of the Google addresses. Is there some way I can use these to suggest/prove whether the issue is local or within the ISP?

DHCPv6 on WAN, LAN is tracking WAN, prefix id 1 on LAN, router advertisement I've switched around from managed, unmanaged and assisted, I messed with this mostly because android phones wouldn't get addresses, but typically I use managed so I can reserve addresses for devices. Ive tried requesting an ipv6 address for the WAN both on and off, I do get an address, it's not in the same range as my prefix, not sure how useful it is, but with or without it tuentests don't change.

I've tried testing from OPNSense itself to rule out issues within my network with no effect, pings fail the same way, can't connect to the same networks.

I don't want to throw too much information in the post so please let me know if there's any additional info that'll help, but here's the basic swtup:

[Monviech (Cedrik)]

If you suspect weird routing try using the tool mtr which will show traceroute in real time.

https://man.freebsd.org/cgi/man.cgi?query=mtr

Also exists for linux, really helpful.

[alias_neo]

Thanks, I've given that a go, it shows major packet loss, any suggestions if I can draw any conclusions from this?

It was run from a machine in my LAN, the first hop address ending b519 is my LAN interface ipv6 address in OPNSense.

I think I've attached a screenshot, I'm on mobile so it's not very clear, apologies.

[Monviech (Cedrik)]

That pretty much looks like a routing loop between :112 and :113 I suppose.

Theyre just forwarding the packet between ech other until the TTL runs out I guess. Could be a provider thing, maybe misconfigured BGP.

[alias_neo]

Here's a successful one for comparison, route to one.one.one.one.

Would you say that routing loop proves the issue is beyond my network? I need to know how much to insist to my ISP that the issue isn't with my configuration, but I'll probably need to get past their first tier support.

[Monviech (Cedrik)]

I cannot say for sure but the evidence makes the ISP a suspect since the next hops ping pong the packets between them.

The working one doesnt have this issue.

[alias_neo]

Thanks, that makes me feel better. I've spent the last few evenings (I have to wait until everyone has gone to bed before I take down the internet) tearing my hair out that I'd misconfigured something but all of the evidence suggested otherwise.

I'll follow up again with my ISP and see if I can get to the right people.

[Maurice]

Looks like an ISP issue, agreed. Have you tried reverse trace routes, too (e. g. from public looking glass services to your own system at home)?
RIPE Atlas is also a good tool to identify ISP-wide issues.

Getting past level 1 support can be challenging. When an ISP I once was with had routing issues, I might have had success with looking up an email address of their NIC an emailing them directly. They didn't respond, but the issue was fixed soon after.

Don't make it sound like a support request. Just a brief heads-up "hey, you have this exact routing issue, this is how to reproduce it". The correct people might read it and silently fix the issue.

Cheers
Maurice