Hi everyone, I'm new to OPNsense and trying to use it to replace my Unifi UDM router. With the UDM I have ipv6 connectivity and everything works great. With OPNsense I receive a /56 prefix from my ISP and lan clients can configure GUAs using SLAAC. However, I have no ipv6 Internet connectivity from the LAN or the OPNsense machine.
I currently have ipv6 disabled on LAN since LAN devices are experiencing high initial connection latency due to trying ipv6 addresses and timing out.
OPNsense has its ipv6 gateway set to the link local address of my ISP's machine. I ran pcap and saw neighbor solicitations from OPNsense to the ISP link local address. I also noticed a RA from a GUA address, and when I looked at the Ethernet frame I noticed that the GUA corresponds to the link local address OPNsense is using as the gateway.
I tried pinging that GUA from OPNsense and it works, but pinging any other GUA that's not part of my lan fails with 100% packet loss.
I also see that icmp-v6 echo requests are sent using the OPNsense link local address. I assume this is part of the issue since the responses won't be routeable, even if the requests aren't dropped by the gateway for using a non GUA address.
Based on all this, I think the issue is that OPNsense itself isn't getting a GUA ipv6 address, but I'm not 100% sure if that's the issue or how to fix it.
Any advice would be really appreciated since I've been spinning my wheels for over a day at this point.
You need to freely allow ICMPv6 everywhere so your clients can join multicast groups and do neighbour discovery. The firewall doesn't need an IPv6 address because you are not using NAT (and nor should you).
If you worry about an attacker using ping to scout out your LAN hosts, calculate the number of IP addresses they need to try ;)
Bart...
Quote from: emzy on February 01, 2024, 05:02:55 PM
OPNsense has its ipv6 gateway set to the link local address of my ISP's machine.
This happened automatically, correct? You didn't manually configure this gateway?
Quote from: emzy on February 01, 2024, 05:02:55 PM
I ran pcap and saw neighbor solicitations from OPNsense to the ISP link local address.
Do you also see the corresponding Neighbor Advertisements? Without them, OPNsense won't know the gateway's MAC address, which would make it unreachable.
Quote from: emzy on February 01, 2024, 05:02:55 PM
I also noticed a RA from a GUA address
That's highly unusual. Unless I've missed a significant RFC update, the source address of RAs MUST be link-local.
Quote from: emzy on February 01, 2024, 05:02:55 PM
and when I looked at the Ethernet frame I noticed that the GUA corresponds to the link local address OPNsense is using as the gateway.
So you see some RAs with a link-local source address and some with a GUA source address, both from the same MAC address?
Quote from: emzy on February 01, 2024, 05:02:55 PM
I tried pinging that GUA from OPNsense and it works
Which source address did you use for this ping?
Quote from: emzy on February 01, 2024, 05:02:55 PM
I also see that icmp-v6 echo requests are sent using the OPNsense link local address. I assume this is part of the issue since the responses won't be routeable, even if the requests aren't dropped by the gateway for using a non GUA address.
Are these echo requests sent to the link-local gateway address? Then these are most likely the gateway monitoring pings (unless you've disabled this feature). They don't have to be routable since the gateway is supposed to be on-link.
Quote from: emzy on February 01, 2024, 05:02:55 PM
Based on all this, I think the issue is that OPNsense itself isn't getting a GUA ipv6 address
IPv6 WAN interface numbering is handled differently depending on the ISP. Some use SLAAC, some use DHCPv6, some don't number the WAN interface at all. This should have no impact on IPv6 Internet access for hosts in the LANs though.
Cheers
Maurice
Thank you both for your replies. I'm replying in line to Maurice's questions.
QuoteThis happened automatically, correct? You didn't manually configure this gateway?
Yes, I didn't manually specify the gateway or the prefix delegation. It all happened automatically via DHCPv6.
QuoteDo you also see the corresponding Neighbor Advertisements? Without them, OPNsense won't know the gateway's MAC address, which would make it unreachable.
Yes, I see the NA as well. Here's an excerpt from the pcap.
ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::11:32ff:fe2b:115a > fe80::46f4:77ff:fe93:ebbe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::46f4:77ff:fe93:ebbe
source link-address option (1), length 8 (1): 02:11:32:2b:11:5a
0x0000: 0211 322b 115a
ethertype IPv6 (0x86dd), length 92: (class 0xc0, hlim 255, next-header ICMPv6 (58) payload length: 24) 2600:4041:41c0::1 > fe80::11:32ff:fe2b:115a: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is fe80::46f4:77ff:fe93:ebbe, Flags [router, solicited]
This 2600:4041:41c0::1 address looks like a GUA address for the gateway. If I ping this from OPNsense, I get echo replies back. If I ping the link local address fe80::46f4:77ff:fe93:ebbe, which is what OPNsense is using as the gateway, then I don't get a reply.
fe80::11:32ff:fe2b:115a is the link-local address for my OPNsense router.
QuoteThat's highly unusual. Unless I've missed a significant RFC update, the source address of RAs MUST be link-local.
I might have misspoken. I probably saw the neighbor advertisement that I pasted above, not a router advertisement.
QuoteWhich source address did you use for this ping?
I'm using the GUI and not setting a source address. I also tried ping6 google.com from the command line, which doesn't work either. In the pcap I see ICMP6 echo requests that look like this with the source address set as the link local address for the OPNsense router.
ethertype IPv6 (0x86dd), length 70: (hlim 64, next-header ICMPv6 (58) payload length: 16) fe80::11:32ff:fe2b:115a > 2607:f8b0:4006:817::200e: [icmp6 sum ok] ICMP6, echo request, seq 91
QuoteAre these echo requests sent to the link-local gateway address? Then these are most likely the gateway monitoring pings (unless you've disabled this feature). They don't have to be routable since the gateway is supposed to be on-link.
No I think I was referring to the echo requests sent to Google's 2607:f8b0:4006:817::200e address (see above). I have gateway monitoring turned off, but pings to the gateway's link local fe80::46f4:77ff:fe93:ebbe address go unanswered. It only responds to pings at its GUA 2600:4041:41c0::1 address.
QuoteIPv6 WAN interface numbering is handled differently depending on the ISP. Some use SLAAC, some use DHCPv6, some don't number the WAN interface at all. This should have no impact on IPv6 Internet access for hosts in the LANs though.
That sounds right... but does that mean that I can't reach the internet over ipv6 unless I have ipv6 configured for the LAN? I have it disabled for my LAN currently because of high latency for clients when they tried and failed to load webpages using ipv6 and failed before falling back to ipv4. If none of OPNsense's interfaces have an ipv6 GUA, I assume ipv6 internet connectivity wouldn't work, right?
Regarding Bart's comments, I don't think the firewall is blocking icmpv6. Just to be sure I set allow all in rules for WAN and LAN, but it didn't make any difference. I'm pretty sure OPNsense already has default allow icmpv6 rules anyways.
Quote from: emzy on February 01, 2024, 07:35:41 PM
I also tried ping6 google.com from the command line, which doesn't work either. In the pcap I see ICMP6 echo requests that look like this with the source address set as the link local address for the OPNsense router.
Yeah, this won't work. As you said correctly, link-local addresses aren't routable.
Quote from: emzy on February 01, 2024, 07:35:41 PM
If none of OPNsense's interfaces have an ipv6 GUA, I assume ipv6 internet connectivity wouldn't work, right?
Correct, OPNsense needs a GUA. Typically on the WAN interface, but can be on a LAN interface, too.
Do the RAs sent by your ISP include a prefix with the A flag set? This would allow SLAAC on the WAN interface. If not, make sure that 'Request only an IPv6 prefix' is
not checked (in the WAN interface's DHCPv6 client configuration).
If you can't get a WAN GUA either way, you'll have to enable IPv6 (track interface) on a LAN interface in order to get IPv6 connectivity for OPNsense itself. The WAN interface unfortunately can't assign itself an address from the delegated prefix.
I think the router is trying to get an ipv6 address, but my ISP isn't giving it one. I see this DHCPv6 response after OPNsense sends a solicit:
DHCPv6
Message type: Advertise (2)
Transaction ID: 0xa9cbe7
Client Identifier
Option: Client Identifier (1)
Length: 14
DUID: 000100012d492931021132229627
DUID Type: link-layer address plus time (1)
Hardware type: Ethernet (1)
DUID Time: Jan 28, 2024 10:03:13.000000000 EST
Link-layer address: 02:11:32:22:96:27
Link-layer address (Ethernet): MS-NLB-PhysServer-17_32:22:96:27 (02:11:32:22:96:27)
Server Identifier
Option: Server Identifier (2)
Length: 26
DUID: 00020000058334343a66343a37373a39333a66323a3030000000
DUID Type: assigned by vendor based on Enterprise number (2)
Enterprise ID: Juniper Networks/Funk Software (1411)
Identifier: 34343a66343a37373a39333a66323a3030000000
Identity Association for Non-temporary Address
Option: Identity Association for Non-temporary Address (3)
Length: 59
IAID: 00000000
T1: 0
T2: 0
Status code
Option: Status code (13)
Length: 43
Status Code: NoAddrAvail (2)
Status Message: No addresses have been assigned for IA_NA
So I'll turn on ipv6 on the LAN and see if anything changes. But it wasn't working before when I had it enabled, so I think there's still more to this story. Thanks for your help so far and I'll report back once I have ipv6 re-enabled on LAN.
Well, I'm not sure what changed, but I have ipv6 connectivity now after enabling it on the LAN. Both the router and clients are able to get GUA from the prefix delegated by my ISP.
However I did notice that radvd doesn't seem to send router advertisements after rebooting opnsense unless I manually restart it. I have the LAN configured to track interface with manual adjustments and router advertisements set to "unmanaged" since I only want to use SLAAC.
I see that there were some issues with this in the past, but it looks like those have been resolved. I even tried upgrading to the new 24.x release, but radvd is still not sending RAs.
- https://github.com/opnsense/core/issues/6637
I'm guessing there's a race between when radvd starts up and when the router gets a prefix delegation from the ISP.
From the logs:
2024-02-02T04:13:08 Warning opnsense /usr/local/etc/rc.bootup: dhcpd_radvd_configure(manual) found no suitable IPv6 address on lan(vtnet0)
2024-02-02T03:50:41 Warning opnsense /usr/local/etc/rc.bootup: dhcpd_radvd_configure(manual) found no suitable IPv6 address on lan(vtnet0)
Any chance you know how to solve this issue? Rebooting radvd by hand works, but doesn't seem like a real solution.
radvd should get reconfigured automatically when the DHCPv6 client gets a prefix delegation. For some reason this doesn't seem to work for you. My gut feeling says it might be related to the fact that your ISP doesn't give you a WAN GUA. I'll try to reproduce this in my test setup.
Did you get a chance to take a closer look at the RAs sent by your ISP? Do they advertise a /64 prefix at all?
I haven't been able to capture any RA's from the ISP. Do you know how I can trigger them while the router is still running? Maybe turning ipv6 on and off again while running pcap?
Edit: My ISP is Verizon Fios in case you or anyone else is familiar with their config.
Ok, I'm pretty sure I grabbed the RA from my ISP. I'm not 100% how to read this, but I think it indicates that my ISP isn't going to assign an address to the router itself.
DHCPv6
Message type: Advertise (2)
Transaction ID: 0x7c6989
Client Identifier
Option: Client Identifier (1)
Length: 14
DUID: 000100012d492931021132229627
DUID Type: link-layer address plus time (1)
Hardware type: Ethernet (1)
DUID Time: Jan 28, 2024 10:03:13.000000000 EST
Link-layer address: 02:11:32:22:96:27
Link-layer address (Ethernet): MS-NLB-PhysServer-17_32:22:96:27 (02:11:32:22:96:27)
Server Identifier
Option: Server Identifier (2)
Length: 26
DUID: 00020000058334343a66343a37373a39333a66323a3030000000
DUID Type: assigned by vendor based on Enterprise number (2)
Enterprise ID: Juniper Networks/Funk Software (1411)
Identifier: 34343a66343a37373a39333a66323a3030000000
Identity Association for Non-temporary Address
Option: Identity Association for Non-temporary Address (3)
Length: 59
IAID: 00000000
T1: 0
T2: 0
Status code
Option: Status code (13)
Length: 43
Status Code: NoAddrAvail (2)
Status Message: No addresses have been assigned for IA_NA
Identity Association for Prefix Delegation
Option: Identity Association for Prefix Delegation (25)
Length: 41
IAID: 00000000
T1: 3600
T2: 5760
IA Prefix
Option: IA Prefix (26)
Length: 25
Preferred lifetime: 7200
Valid lifetime: 7200
Prefix length: 56
Prefix address: 2600:1234:1234:1234::
I redacted the prefix address to 2600:1234:1234:1234::, but it's a valid prefix and I'm able to use it to get GUAs for my LAN.
In the process of grabbing that pcap I turned ipv6 on and off both for the wan and lan interfaces. I'm not sure what's changed, and I'm pretty sure that I have everything configured the same way as it was before at this point. But, I'm not able to reproduce the issue where opnsense doesn't send router announcements on lan anymore. I've rebooted several times and my lan devices have consistently gotten ipv6 addresses without me needing to manually restart radvd.
Go figure... I'll keep an eye on things and see if anything changes.
Shoot... I spoke too soon. The issue with radvd not advertising properly has returned. Manually restarting it resolves the issue.
What you grabbed there is a DHCPv6 Advertise, not a Router Advertisement. If you don't see these, your ISP might only send solicited RAs. You should be able to trigger a Router Solicitation by initiating a DHCP reload in the interfaces overview.
I wasn't able to reproduce the radvd issue, worked fine every time. Might be hard to diagnose if this is an intermittent issue on your system.
Sorry, I think I grabbed an RA as well. I don't think I see anything about a /64 there.
Internet Control Message Protocol v6
Type: Router Advertisement (134)
Code: 0
Checksum: 0xe011 [correct]
[Checksum Status: Good]
Cur hop limit: 64
Flags: 0x80, Managed address configuration, Prf (Default Router Preference): Medium
1... .... = Managed address configuration: Set
.0.. .... = Other configuration: Not set
..0. .... = Home Agent: Not set
...0 0... = Prf (Default Router Preference): Medium (0)
.... .0.. = ND Proxy: Not set
.... ..00 = Reserved: 0
Router lifetime (s): 1800
Reachable time (ms): 0
Retrans timer (ms): 0
ICMPv6 Option (Source link-layer address : 46:f4:77:93:eb:be)
Type: Source link-layer address (1)
Length: 1 (8 bytes)
Link-layer address: 46:f4:77:93:eb:be (46:f4:77:93:eb:be)
I'll do some more testing to see if I can reproduce the LAN RA issue and figure out what causes it. It seemed like leaving DHCPv6 enabled on LAN helped, but maybe that was just a coincidence.
Steps to reproduce:
1. Set LAN ipv6 to "track interface"
2. Click "Reload' on WAN in the "Commands" column of Interfaces > Overview on 24.1
Looking at the logs, it looks like opnsense sends a RELEASE for the ipv6 prefix after reloading the interface, and radvd tries to configure itself before dhcp6c sends a request for a new prefix. Perhaps, like you said, since no ipv6 address is assigned to the WAN interface radvd doesn't get reset again, and fails to advertise.
This is with "prevent release" checked in the interface settings, by the way. Maybe "prevent release" doesn't do anything if you manually reset the interface. I am definitely getting a new prefix every time I hit the button.
I will test this again later with "prevent release" checked, but instead of using the reload command I'll just reboot opnsense completely.
2024-02-04T02:27:48 Notice opnsense /usr/local/etc/rc.newwanipv6: Failed to detect IP for interface wan
2024-02-04T02:27:45 Notice opnsense /usr/local/etc/rc.newwanipv6: Failed to detect IP for interface wan
2024-02-04T02:27:45 Notice dhcp6c dhcp6c_script: REQUEST on vtnet1 renewal
2024-02-04T02:27:45 Notice dhcp6c dhcp6c_script: REQUEST on vtnet1 executing
2024-02-04T02:27:43 Warning opnsense /usr/local/etc/rc.configure_interface: dhcpd_radvd_configure(auto) found no suitable IPv6 address on lan(vtnet0)
2024-02-04T02:27:41 Notice dhcp6c RTSOLD script - Sending SIGHUP to dhcp6c
2024-02-04T02:27:41 Notice dhcp6c dhcp6c_script: RELEASE on vtnet1 executing
2024-02-04T02:27:41 Notice opnsense /usr/local/etc/rc.configure_interface: ROUTING: entering configure using 'wan'
2024-02-04T02:27:41 Notice dhcp6c RTSOLD script - Sending SIGHUP to dhcp6c
Here's logs from when I restarted opnsense instead of reloading the WAN interface. vtnet1 is the WAN interface here.
2024-02-04T12:46:40 Notice opnsense /usr/local/etc/rc.newwanipv6: Failed to detect IP for interface wan
2024-02-04T12:46:37 Notice kernel <118>>>> Invoking start script 'freebsd'
2024-02-04T12:46:37 Notice kernel <118>Reconfiguring IPv6 on vtnet1
2024-02-04T12:46:37 Notice kernel <118>Reconfiguring IPv4 on vtnet1
2024-02-04T12:46:37 Notice kernel <118>>>> Invoking start script 'newwanip'
2024-02-04T12:46:23 Notice opnsense /usr/local/etc/rc.newwanipv6: IP renewal deferred during boot on 'vtnet1'
2024-02-04T12:46:23 Notice dhcp6c dhcp6c_script: REQUEST on vtnet1 renewal
2024-02-04T12:46:23 Notice dhcp6c dhcp6c_script: REQUEST on vtnet1 executing
2024-02-04T12:46:21 Notice kernel <118>Starting router advertisement service...done.
2024-02-04T12:46:21 Notice opnsense /usr/local/etc/rc.bootup: plugins_configure dns (execute task : unbound_configure_do(1))
2024-02-04T12:46:21 Notice opnsense /usr/local/etc/rc.bootup: plugins_configure dns (execute task : dnsmasq_configure_do(1))
2024-02-04T12:46:21 Notice opnsense /usr/local/etc/rc.bootup: plugins_configure dns (1)
2024-02-04T12:46:21 Notice opnsense /usr/local/etc/rc.bootup: plugins_configure dhcrelay (execute task : dhcpd_dhcrelay_configure(1))
2024-02-04T12:46:21 Notice opnsense /usr/local/etc/rc.bootup: plugins_configure dhcrelay (1)
2024-02-04T12:46:21 Notice dhcp6c RTSOLD script - Sending SIGHUP to dhcp6c
2024-02-04T12:46:21 Warning opnsense /usr/local/etc/rc.bootup: dhcpd_radvd_configure(manual) found no suitable IPv6 address on lan(vtnet0)
I'm not certain, but it seems like what's happening is that opnsense starts radvd before it's gotten ipv6 configured on the WAN. After the ipv6 prefix is established, the newwanipv6 procedure runs and doesn't detect a WAN ipv6 address, so it doesn't trigger any of the service restarts that normally happen when the WAN ip changes. If the new ip procedure was aware or listening for prefix changes this probably wouldn't be an issue.
I'm still not sure if there's a race happening or not between the radvd setup and the wan/lan ipv6 address assignment. I thought that everything was working for me at some point, but I haven't been able to get to that state again.
Ok Maurice, I dug through the code this morning and I think I've figured out exactly what's happening. I think you might have suspected something like this could be the problem.
I filed a bug report with a full description and pointers to the code.
- https://github.com/opnsense/core/issues/7202
Here is the summary:
If the ISP only delegates an ipv6 prefix but no GUA address for the WAN interface, the rc.newwanipv6 script exits early and does not configure or restart radvd. This means it never starts to advertise a prefix on the LAN, and clients don't get ipv6 addresses.
The script can't tell if the ISP has only delegated a prefix or not, so when it fails to find the WAN GUA it exits early. If you check the "Request only an IPv6 prefix" option on the WAN interface then the script does not exit early, and radvd is properly configured.
I would have thought that the "Request only an IPv6 prefix" option only affects the solicitation opnsense sends to the ISP, not that it is necessary to check this box if the ISP is only delegating a prefix. I'm not sure how difficult it would be to actually check if the ISP only delegated a prefix, but if it's possible I think that would be less surprising to the user.
Anyways, I think we can say case closed for now and follow up on the bug to see if there's a way to make opnsense smarter about the prefix delegation. Thanks for all the help!
The cheapest solution would be to rename the UI option to "ISP delegates only a prefix" or similar to clarify that this is about what OPNsense should expect, not what it should ask for.
Thanks for the detailed report. I posted an experimental patch over here https://github.com/opnsense/core/issues/7202#issuecomment-1925840590
Been meaning to change this myself, but did not want to go overboard the last years with changes in that area so after verifying the "force" reload flag works fine now we can do this step actually.
All feedback and complaints welcome. I'll be testing this next week in a production setup as well.
Cheers,
Franco