Have ipv6 prefix delegation but no ipv6 internet

[emzy]

Hi everyone, I'm new to OPNsense and trying to use it to replace my Unifi UDM router. With the UDM I have ipv6 connectivity and everything works great. With OPNsense I receive a /56 prefix from my ISP and lan clients can configure GUAs using SLAAC. However, I have no ipv6 Internet connectivity from the LAN or the OPNsense machine.

I currently have ipv6 disabled on LAN since LAN devices are experiencing high initial connection latency due to trying ipv6 addresses and timing out.

OPNsense has its ipv6 gateway set to the link local address of my ISP’s machine. I ran pcap and saw neighbor solicitations from OPNsense to the ISP link local address. I also noticed a RA from a GUA address, and when I looked at the Ethernet frame I noticed that the GUA corresponds to the link local address OPNsense is using as the gateway.

I tried pinging that GUA from OPNsense and it works, but pinging any other GUA that's not part of my lan fails with 100% packet loss.

I also see that icmp-v6 echo requests are sent using the OPNsense link local address. I assume this is part of the issue since the responses won't be routeable, even if the requests aren't dropped by the gateway for using a non GUA address.

Based on all this, I think the issue is that OPNsense itself isn't getting a GUA ipv6 address, but I'm not 100% sure if that's the issue or how to fix it.

Any advice would be really appreciated since I've been spinning my wheels for over a day at this point.

[bartjsmit]

You need to freely allow ICMPv6 everywhere so your clients can join multicast groups and do neighbour discovery. The firewall doesn't need an IPv6 address because you are not using NAT (and nor should you).

If you worry about an attacker using ping to scout out your LAN hosts, calculate the number of IP addresses they need to try 

Bart...

[Maurice]

OPNsense has its ipv6 gateway set to the link local address of my ISP’s machine.

This happened automatically, correct? You didn't manually configure this gateway?

I ran pcap and saw neighbor solicitations from OPNsense to the ISP link local address.

Do you also see the corresponding Neighbor Advertisements? Without them, OPNsense won't know the gateway's MAC address, which would make it unreachable.

I also noticed a RA from a GUA address

That's highly unusual. Unless I've missed a significant RFC update, the source address of RAs MUST be link-local.

and when I looked at the Ethernet frame I noticed that the GUA corresponds to the link local address OPNsense is using as the gateway.

So you see some RAs with a link-local source address and some with a GUA source address, both from the same MAC address?

I tried pinging that GUA from OPNsense and it works

Which source address did you use for this ping?

I also see that icmp-v6 echo requests are sent using the OPNsense link local address. I assume this is part of the issue since the responses won't be routeable, even if the requests aren't dropped by the gateway for using a non GUA address.

Are these echo requests sent to the link-local gateway address? Then these are most likely the gateway monitoring pings (unless you've disabled this feature). They don't have to be routable since the gateway is supposed to be on-link.

Based on all this, I think the issue is that OPNsense itself isn't getting a GUA ipv6 address

IPv6 WAN interface numbering is handled differently depending on the ISP. Some use SLAAC, some use DHCPv6, some don't number the WAN interface at all. This should have no impact on IPv6 Internet access for hosts in the LANs though.

Cheers
Maurice

[emzy]

Thank you both for your replies. I'm replying in line to Maurice's questions.

This happened automatically, correct? You didn't manually configure this gateway?

Yes, I didn't manually specify the gateway or the prefix delegation. It all happened automatically via DHCPv6.

Do you also see the corresponding Neighbor Advertisements? Without them, OPNsense won't know the gateway's MAC address, which would make it unreachable.

Yes, I see the NA as well. Here's an excerpt from the pcap.

ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::11:32ff:fe2b:115a > fe80::46f4:77ff:fe93:ebbe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::46f4:77ff:fe93:ebbe
     source link-address option (1), length 8 (1): 02:11:32:2b:11:5a
       0x0000:  0211 322b 115a

ethertype IPv6 (0x86dd), length 92: (class 0xc0, hlim 255, next-header ICMPv6 (58) payload length: 24) 2600:4041:41c0::1 > fe80::11:32ff:fe2b:115a: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is fe80::46f4:77ff:fe93:ebbe, Flags [router, solicited]

This 2600:4041:41c0::1 address looks like a GUA address for the gateway. If I ping this from OPNsense, I get echo replies back. If I ping the link local address fe80::46f4:77ff:fe93:ebbe, which is what OPNsense is using as the gateway, then I don't get a reply.

fe80::11:32ff:fe2b:115a is the link-local address for my OPNsense router.

That's highly unusual. Unless I've missed a significant RFC update, the source address of RAs MUST be link-local.

I might have misspoken. I probably saw the neighbor advertisement that I pasted above, not a router advertisement.

Which source address did you use for this ping?

I'm using the GUI and not setting a source address. I also tried ping6 google.com from the command line, which doesn't work either. In the pcap I see ICMP6 echo requests that look like this with the source address set as the link local address for the OPNsense router.

ethertype IPv6 (0x86dd), length 70: (hlim 64, next-header ICMPv6 (58) payload length: 16) fe80::11:32ff:fe2b:115a > 2607:f8b0:4006:817::200e: [icmp6 sum ok] ICMP6, echo request, seq 91

Are these echo requests sent to the link-local gateway address? Then these are most likely the gateway monitoring pings (unless you've disabled this feature). They don't have to be routable since the gateway is supposed to be on-link.

No I think I was referring to the echo requests sent to Google's 2607:f8b0:4006:817::200e address (see above). I have gateway monitoring turned off, but pings to the gateway's link local fe80::46f4:77ff:fe93:ebbe address go unanswered. It only responds to pings at its GUA 2600:4041:41c0::1 address.

IPv6 WAN interface numbering is handled differently depending on the ISP. Some use SLAAC, some use DHCPv6, some don't number the WAN interface at all. This should have no impact on IPv6 Internet access for hosts in the LANs though.

That sounds right... but does that mean that I can't reach the internet over ipv6 unless I have ipv6 configured for the LAN? I have it disabled for my LAN currently because of high latency for clients when they tried and failed to load webpages using ipv6 and failed before falling back to ipv4. If none of OPNsense's interfaces have an ipv6 GUA, I assume ipv6 internet connectivity wouldn't work, right?


Regarding Bart's comments, I don't think the firewall is blocking icmpv6. Just to be sure I set allow all in rules for WAN and LAN, but it didn't make any difference. I'm pretty sure OPNsense already has default allow icmpv6 rules anyways.

[Maurice]

I also tried ping6 google.com from the command line, which doesn't work either. In the pcap I see ICMP6 echo requests that look like this with the source address set as the link local address for the OPNsense router.

Yeah, this won't work. As you said correctly, link-local addresses aren't routable.

If none of OPNsense's interfaces have an ipv6 GUA, I assume ipv6 internet connectivity wouldn't work, right?

Correct, OPNsense needs a GUA. Typically on the WAN interface, but can be on a LAN interface, too.
Do the RAs sent by your ISP include a prefix with the A flag set? This would allow SLAAC on the WAN interface. If not, make sure that 'Request only an IPv6 prefix' is not checked (in the WAN interface's DHCPv6 client configuration).
If you can't get a WAN GUA either way, you'll have to enable IPv6 (track interface) on a LAN interface in order to get IPv6 connectivity for OPNsense itself. The WAN interface unfortunately can't assign itself an address from the delegated prefix.

[emzy]

I think the router is trying to get an ipv6 address, but my ISP isn't giving it one. I see this DHCPv6 response after OPNsense sends a solicit:

DHCPv6
    Message type: Advertise (2)
    Transaction ID: 0xa9cbe7
    Client Identifier
        Option: Client Identifier (1)
        Length: 14
        DUID: 000100012d492931021132229627
        DUID Type: link-layer address plus time (1)
        Hardware type: Ethernet (1)
        DUID Time: Jan 28, 2024 10:03:13.000000000 EST
        Link-layer address: 02:11:32:22:96:27
        Link-layer address (Ethernet): MS-NLB-PhysServer-17_32:22:96:27 (02:11:32:22:96:27)
    Server Identifier
        Option: Server Identifier (2)
        Length: 26
        DUID: 00020000058334343a66343a37373a39333a66323a3030000000
        DUID Type: assigned by vendor based on Enterprise number (2)
        Enterprise ID: Juniper Networks/Funk Software (1411)
        Identifier: 34343a66343a37373a39333a66323a3030000000
    Identity Association for Non-temporary Address
        Option: Identity Association for Non-temporary Address (3)
        Length: 59
        IAID: 00000000
        T1: 0
        T2: 0
        Status code
            Option: Status code (13)
            Length: 43
            Status Code: NoAddrAvail (2)
            Status Message: No addresses have been assigned for IA_NA


So I'll turn on ipv6 on the LAN and see if anything changes. But it wasn't working before when I had it enabled, so I think there's still more to this story. Thanks for your help so far and I'll report back once I have ipv6 re-enabled on LAN.

[emzy]

Well, I'm not sure what changed, but I have ipv6 connectivity now after enabling it on the LAN. Both the router and clients are able to get GUA from the prefix delegated by my ISP.

However I did notice that radvd doesn't seem to send router advertisements after rebooting opnsense unless I manually restart it. I have the LAN configured to track interface with manual adjustments and router advertisements set to "unmanaged" since I only want to use SLAAC.

I see that there were some issues with this in the past, but it looks like those have been resolved. I even tried upgrading to the new 24.x release, but radvd is still not sending RAs.

- https://github.com/opnsense/core/issues/6637

I'm guessing there's a race between when radvd starts up and when the router gets a prefix delegation from the ISP.

From the logs:

2024-02-02T04:13:08   Warning   opnsense   /usr/local/etc/rc.bootup: dhcpd_radvd_configure(manual) found no suitable IPv6 address on lan(vtnet0)   
2024-02-02T03:50:41   Warning   opnsense   /usr/local/etc/rc.bootup: dhcpd_radvd_configure(manual) found no suitable IPv6 address on lan(vtnet0)

Any chance you know how to solve this issue? Rebooting radvd by hand works, but doesn't seem like a real solution.

[Maurice]

radvd should get reconfigured automatically when the DHCPv6 client gets a prefix delegation. For some reason this doesn't seem to work for you. My gut feeling says it might be related to the fact that your ISP doesn't give you a WAN GUA. I'll try to reproduce this in my test setup.

Did you get a chance to take a closer look at the RAs sent by your ISP? Do they advertise a /64 prefix at all?

[emzy]

I haven't been able to capture any RA's from the ISP. Do you know how I can trigger them while the router is still running? Maybe turning ipv6 on and off again while running pcap?

Edit: My ISP is Verizon Fios in case you or anyone else is familiar with their config.

[emzy]

Ok, I'm pretty sure I grabbed the RA from my ISP. I'm not 100% how to read this, but I think it indicates that my ISP isn't going to assign an address to the router itself.

DHCPv6
    Message type: Advertise (2)
    Transaction ID: 0x7c6989
    Client Identifier
        Option: Client Identifier (1)
        Length: 14
        DUID: 000100012d492931021132229627
        DUID Type: link-layer address plus time (1)
        Hardware type: Ethernet (1)
        DUID Time: Jan 28, 2024 10:03:13.000000000 EST
        Link-layer address: 02:11:32:22:96:27
        Link-layer address (Ethernet): MS-NLB-PhysServer-17_32:22:96:27 (02:11:32:22:96:27)
    Server Identifier
        Option: Server Identifier (2)
        Length: 26
        DUID: 00020000058334343a66343a37373a39333a66323a3030000000
        DUID Type: assigned by vendor based on Enterprise number (2)
        Enterprise ID: Juniper Networks/Funk Software (1411)
        Identifier: 34343a66343a37373a39333a66323a3030000000
    Identity Association for Non-temporary Address
        Option: Identity Association for Non-temporary Address (3)
        Length: 59
        IAID: 00000000
        T1: 0
        T2: 0
        Status code
            Option: Status code (13)
            Length: 43
            Status Code: NoAddrAvail (2)
            Status Message: No addresses have been assigned for IA_NA
    Identity Association for Prefix Delegation
        Option: Identity Association for Prefix Delegation (25)
        Length: 41
        IAID: 00000000
        T1: 3600
        T2: 5760
        IA Prefix
            Option: IA Prefix (26)
            Length: 25
            Preferred lifetime: 7200
            Valid lifetime: 7200
            Prefix length: 56
            Prefix address: 2600:1234:1234:1234::


I redacted the prefix address to 2600:1234:1234:1234::, but it's a valid prefix and I'm able to use it to get GUAs for my LAN.

[emzy]

In the process of grabbing that pcap I turned ipv6 on and off both for the wan and lan interfaces. I'm not sure what's changed, and I'm pretty sure that I have everything configured the same way as it was before at this point. But, I'm not able to reproduce the issue where opnsense doesn't send router announcements on lan anymore. I've rebooted several times and my lan devices have consistently gotten ipv6 addresses without me needing to manually restart radvd.

Go figure... I'll keep an eye on things and see if anything changes.

[emzy]

Shoot... I spoke too soon. The issue with radvd not advertising properly has returned. Manually restarting it resolves the issue.

[Maurice]

What you grabbed there is a DHCPv6 Advertise, not a Router Advertisement. If you don't see these, your ISP might only send solicited RAs. You should be able to trigger a Router Solicitation by initiating a DHCP reload in the interfaces overview.

I wasn't able to reproduce the radvd issue, worked fine every time. Might be hard to diagnose if this is an intermittent issue on your system.

[emzy]

Sorry, I think I grabbed an RA as well. I don't think I see anything about a /64 there.

Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    Checksum: 0xe011 [correct]
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0x80, Managed address configuration, Prf (Default Router Preference): Medium
        1... .... = Managed address configuration: Set
        .0.. .... = Other configuration: Not set
        ..0. .... = Home Agent: Not set
        ...0 0... = Prf (Default Router Preference): Medium (0)
        .... .0.. = ND Proxy: Not set
        .... ..00 = Reserved: 0
    Router lifetime (s): 1800
    Reachable time (ms): 0
    Retrans timer (ms): 0
    ICMPv6 Option (Source link-layer address : 46:f4:77:93:eb:be)
        Type: Source link-layer address (1)
        Length: 1 (8 bytes)
        Link-layer address: 46:f4:77:93:eb:be (46:f4:77:93:eb:be)


I'll do some more testing to see if I can reproduce the LAN RA issue and figure out what causes it. It seemed like leaving DHCPv6 enabled on LAN helped, but maybe that was just a coincidence.

[emzy]

Steps to reproduce:

1. Set LAN ipv6 to "track interface"

2. Click "Reload' on WAN in the "Commands" column of Interfaces > Overview on 24.1

Looking at the logs, it looks like opnsense sends a RELEASE for the ipv6 prefix after reloading the interface, and radvd tries to configure itself before dhcp6c sends a request for a new prefix. Perhaps, like you said, since no ipv6 address is assigned to the WAN interface radvd doesn't get reset again, and fails to advertise.

This is with "prevent release" checked in the interface settings, by the way. Maybe "prevent release" doesn't do anything if you manually reset the interface. I am definitely getting a new prefix every time I hit the button.

I will test this again later with "prevent release" checked, but instead of using the reload command I'll just reboot opnsense completely.


2024-02-04T02:27:48   Notice   opnsense   /usr/local/etc/rc.newwanipv6: Failed to detect IP for interface wan   
2024-02-04T02:27:45   Notice   opnsense   /usr/local/etc/rc.newwanipv6: Failed to detect IP for interface wan   
2024-02-04T02:27:45   Notice   dhcp6c   dhcp6c_script: REQUEST on vtnet1 renewal   
2024-02-04T02:27:45   Notice   dhcp6c   dhcp6c_script: REQUEST on vtnet1 executing

2024-02-04T02:27:43   Warning   opnsense   /usr/local/etc/rc.configure_interface: dhcpd_radvd_configure(auto) found no suitable IPv6 address on lan(vtnet0)

2024-02-04T02:27:41   Notice   dhcp6c   RTSOLD script - Sending SIGHUP to dhcp6c   
2024-02-04T02:27:41   Notice   dhcp6c   dhcp6c_script: RELEASE on vtnet1 executing   
2024-02-04T02:27:41   Notice   opnsense   /usr/local/etc/rc.configure_interface: ROUTING: entering configure using 'wan'   
2024-02-04T02:27:41   Notice   dhcp6c   RTSOLD script - Sending SIGHUP to dhcp6c