OPNsense Forum

Archive => 20.7 Legacy Series => Topic started by: staticznld on August 11, 2020, 08:16:10 pm

Title: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: staticznld on August 11, 2020, 08:16:10 pm
Hi,

After installer 20.7 i see some issues regarding IPV6.
After a complete restart of a Windows 10 clients it takes up to 5 a 10 minutes to acquire an IPV6 address.
I am not using DHCPv6 only stateless address configuration.
When restarting the Radvd service in the opnsense interface the windows 10 client immediately picks up an address.



Anyone an idea?
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: franco on August 12, 2020, 09:26:06 am
Hi,

Probably... https://github.com/opnsense/core/commit/20efa4f46c2

At least that settles "MinRtrAdvInterval / MaxRtrAdvInterval were set to very low values (3 / 10) for no apparent reason" from the commit message. ;)

The default values (200 s / 600 s --- 3,3 minutes / 10 minutes) are sane in general as they do not stress mobile devices so much constantly waking them up to say nothing has changed.

It's possible to set these manually in manual mode (interface settings), but it may require some more tinkering on your part.


Cheers,
Franco
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: staticznld on August 12, 2020, 06:02:05 pm
Thanks for the info!
Tried setting it up as before and indeed immediately an ipv6 address!

Switched back to default though.
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: franco on August 12, 2020, 06:27:02 pm
Ok, so case closed? :)


Cheers,
Franco
Title: Re: [SOLVED] Acquiring ipv6 address can take 5 to 10 minutes.
Post by: gpb on August 13, 2020, 08:41:43 pm
@franco, I ran into this issue (or at least similar) this morning after a power outage (clean shutdown), and upon restart I had no ipv6.  This went on for a few hours as I tried to figure out what was going on.  From the firewall (ssh) I could ping example.com, but from clients I could not.  Using the opnsense gui under interface diagnostics ping, I could not ping using the LAN interface, but could from the wan.  Updated to 20.7.1, no difference. 

I saw your notes posted above and went to manual RA settings, enabled for LAN, and found the setting to be disabled.  Set it to Stateless and was able to connect.  This may have been a coincidence, I can't say.  I'd have to try to recreate this with another reboot followed by this exact change.  It seems to make sense clients maybe didn't have a valid ipv6 prefix/route to the LAN interface and running a tracert timed out (which works fine now).

Maybe this provides a clue to others reports.  Is "disabled" what should be seen on first opening the RA for LAN interface once customization is enabled?  Is that the default?  Thx.
Title: Re: [SOLVED] Acquiring ipv6 address can take 5 to 10 minutes.
Post by: Maurice on August 14, 2020, 12:34:19 am
MinRtrAdvInterval / MaxRtrAdvInterval values only affect unsolicited RAs. When new clients connect to a network, they send Router Solicitations to which radvd should always respond immediately.

Maybe the Router Solicitations never reach OPNsense. Windows misconfiguration / rogue firewall on the Windows machine / switch or access point which doesn't always properly forward IPv6 multicasts?

Setting MinRtrAdvInterval / MaxRtrAdvInterval to very low values might mask such issues by constantly flooding the network with RAs. But I wouldn't consider that a solution.

Cheers

Maurice
Title: Re: [SOLVED] Acquiring ipv6 address can take 5 to 10 minutes.
Post by: gpb on August 14, 2020, 12:42:31 am
Thanks for the clarification, so would the multicast RA be disabled normally?  I left defaults in place, only set it from disabled to stateless.  I don't have an explanation, could have been coincidence.  The problem affected several Rpi's and Windows PC...didn't check other devices.  Was strange I couldn't ping the LAN interface address which would normally work fine (obviously).  Checking the windows PC after an ipconfig release/renew I didn't have a unicast address, just link local.  Again thanks.
Title: Re: [SOLVED] Acquiring ipv6 address can take 5 to 10 minutes.
Post by: Maurice on August 14, 2020, 12:51:19 am
@gpb, this was meant as a response to @staticznld's issue.

I don't think yours is related because this should work even with RAs disabled:

Using the opnsense gui under interface diagnostics ping, I could not ping using the LAN interface, but could from the wan.

Cheers

Maurice
Title: Re: [SOLVED] Acquiring ipv6 address can take 5 to 10 minutes.
Post by: gpb on August 14, 2020, 01:04:06 am
@Maurice...thanks...still good clarification.  :) 
Title: Re: [SOLVED] Acquiring ipv6 address can take 5 to 10 minutes.
Post by: marjohn56 on August 14, 2020, 10:38:37 am
Seeing similar issue this morning, cannot say if it was after I had just updated to 20.7.1 or not, but no devices are getting a route.. very odd. Will investigate further tomorrow as no time today.


*** update ***


Turns out it was my primary switch...
Title: Re: [SOLVED] Acquiring ipv6 address can take 5 to 10 minutes.
Post by: gpb on August 14, 2020, 04:18:07 pm
Doing a packet capture, it looks like the router solicitation for the host is never answered and ipv6 address determination is done only when a router advertisement to all hosts is received.  That means there's a variable time delay spanning seconds to minutes depending on the intervals of those RAs.

Testing is easy enough, using an ipad, disable wifi, start packet capture on opnsense, enable wifi, watch the ipad screen until the screen updates to show the ipv6 addresses appear, then stop capture.

I'll assume I'm the only one having this problem, what can I do to further debug this issue?  I'm also assuming this is why I couldn't get ANY ipv6 addresses yesterday until I enabled RAs in general...using @franco's instruction about setting manual override for RAs on the LAN interface.  What component is responsible for answering a router solicitation?  I'm assuming it's RADVD...and not dhcpv6 (which is disabled on that interface and was never an option until enabling RA customization). 

Any guidance is appreciated...thanks!
Title: Re: [SOLVED] Acquiring ipv6 address can take 5 to 10 minutes.
Post by: Maurice on August 15, 2020, 03:42:47 am
Actually, I'm now observing this issues with 20.7.1, too: Like @gpb says, a packet capture shows OPNsense receiving Router Solicitations, but often not responding with Router Advertisements. This is intermittent. It sometimes works, but mostly doesn't. Unsolicited RAs don't seem to be affected and are still being sent.

I didn't observe this on older versions. @franco, any idea if there recently have been changes which might be related?

Cheers

Maurice
Title: Re: [SOLVED] Acquiring ipv6 address can take 5 to 10 minutes.
Post by: marjohn56 on August 15, 2020, 09:00:57 am
Actually, I'm now observing this issues with 20.7.1, too: Like @gpb says, a packet capture shows OPNsense receiving Router Solicitations, but often not responding with Router Advertisements. This is intermittent. It sometimes works, but mostly doesn't. Unsolicited RAs don't seem to be affected and are still being sent.

I didn't observe this on older versions. @franco, any idea if there recently have been changes which might be related?



Yup, same again this morning, so not my switch. Seeing the same issue as you on wireshark; the radvd.conf appears the same, at least there were no changes to the creation routines that I can see between 20.1 and 20.7.
I've also tried the radvd deamon from a 20.1 dev version and that makes no difference either.


*** UPDATE***


3 minutes later it's working again !

Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: staticznld on August 15, 2020, 10:27:18 am
While digging further i also see the same issue.

Router solicitations are sent from a client (VU+ linux stb) but not answered.
When OPNsense sents an Router advertisement the stb is picking up an address and is connected through IPV6.

The packet capture future with Wireshark is great to analyse the problem!

Edit:

When creating a firewall rule to allow icmp6 to any it looks like its working.
Nope not working
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: Maurice on August 15, 2020, 02:10:24 pm
When looking at the 20.7.1 changelog, the only thing that gets my attention is "src: assorted multicast group join/leave corrections". But I've not looked at the code yet.

Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: marjohn56 on August 15, 2020, 05:21:56 pm
I've installed 20.1 on my test router ( damn it, just remembered I forgot to copy over some of the dev work! ), it was showing the same issue.


 I've found it quicker to test just use my Android mobile, a disable/enable wifi and using the Net Analyzer App shows me an immediate v6 address if RADVD is working. Again it appears to be my primary switch causing the issue. I disabled MLD on the switch and immediately IPv6 was back up on my mobile.


Still testing, but at the moment it's all good again, slightly baffled, as I'm sure I did the same thing yesterday too.  :-\
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: gpb on August 16, 2020, 12:28:54 am
@marjohn56, Did you do a packet capture?  The ipad I tested with performed three identical solicitations before the RA to ALL hosts was received, then it was able to establish a unicast global address.  That took about 90 seconds.  I hate to put this on github for support if this is a config issue on my side...though I didn't have the problem until the power outage we had made it noticeable.  I think in the past the hosts would all keep their addresses during a router reboot...but shutting everything down caused a full reset.  Speculating...but that's how it seems as I didn't have the problem with 20.7RC/20.7 initially then network reset and problem became visible.

I wonder if radvd is seeing the request and not responding or just never seeing it.  And if it's not seeing it, why.  Not sure if it can be monitored with some kind of verbose output...couldn't find much.

Thanks.
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: marjohn56 on August 17, 2020, 04:23:21 pm
Sorry it took a while to get back, been busy. I did use Wireshark but only on my PC, which is a couple of switches away from Opnsense. Over the weekend I saved off the config, did a fresh install of 20.7, updated to 20.7.1 and re-imported the config - it's my usual routine if I have strangeness. I reset all of my switches too, now 48 hours plus later and no further issues on any of my devices, mobile or fixed, everything appears happy.
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: gpb on August 17, 2020, 05:05:45 pm
Thanks @marjohn56, so just to clarify, when you say "no further issues" you mean you verified that RADVD is now replying to solicitations?  Cheers!
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: marjohn56 on August 17, 2020, 06:04:36 pm
Correct, no issues here with radvd,
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: staticznld on August 18, 2020, 06:47:32 pm
Still isseus here.

I see in the firewall log an message from a link local address to the FF02::2 and it passed.
So I assume the Router solicitation will come to Radvd.

When the link local address of the OPNsense box sends the router advertisements to FF02::2 then the client can configure its ip address.
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: gpb on August 18, 2020, 07:19:16 pm
I'm also still looking at this trying to understand the radvd advertisement process.  As I understand it, it sends unsolicited and (should send) solicited adv's...but I never see the solicited responses, any response goes to ff02::1 (all-nodes).  And I'm not sure if @marjohn56 ever did either, he indicated only that he's not having an issue (and I don't have an issue once I get an address).  Was more interested from a learning and debugging aspect.  Can't find a lot of info unfortunately...still looking tho.

Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: dave on August 19, 2020, 10:32:37 am
I'm experiencing this with an Android device.  Not noticeable on my Win10 box because as soon as ipv4 connectivity's established Windows is happy; but Android reports my WLAN has no internet connectivity until IPv6 kicks in which can take ~20 seconds.  For the time being I've just disabled IPv6 on my WLAN.
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: marjohn56 on August 19, 2020, 04:10:57 pm
I still see this randomly, so I have enabled debug on RADVD to see what's happening, of course, since I've enabled the debug it's behaving itself!


If you want to enable full debug and see if you can see something odd in your logs do this. Firstly you need to kill the existing RADVD daemon, either from the GUI or shell. Next you need to re-launch it from the shell. Normally what little logging it gives in default mode goes to syslog, we'll send it to a new log of its own. Issue the following command in the shell.


# /usr/local/sbin/radvd -d 5 -p /var/run/radvd.pid -C /var/etc/radvd.conf -m logfile -l /var/log/radvd.log


You'll now get verbose output in the /var/log/radvd.log file. In the log you SHOULD see something like, this, it's an RS from my PC which I had disabled then re-enabled IPv6.

[Aug 19 15:07:16] radvd (21158): igb1_vlan101 received RS from: fe80::2887:a02e:93cd:6d96

See if gives any clues.
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: gpb on August 19, 2020, 07:43:28 pm
Thanks for that.  It looks like I am having an issue with these three messages repeating frequently (every 20 secs or so). 

EDIT: This appears normal, it's apparently just a router advertisement from WAN (igb1).

Code: [Select]
[Aug 19 13:07:30] radvd (85849): igb1 recvmsg len=16
[Aug 19 13:07:30] radvd (85849): igb1 received a packet
[Aug 19 13:07:30] radvd (85849): igb1 received icmpv6 RS/RA packet on an unknown interface with index 2

My network is simplistic, WAN (igb1), LAN (igb0) and VLAN (no ipv6; igb0).

I re-ran the ipad test and there is nothing in the log that radvd received a request.  It actually never shows a request received from ibg0 ever.  The above snippet continues repeatedly until about 90 seconds past when I initiated the test and then there's finally a mention of igb0:

Code: [Select]
[Aug 19 13:15:22] radvd (85849): igb1 received icmpv6 RS/RA packet on an unknown interface with index 2
[Aug 19 13:15:22] radvd (85849): polling for 3.394 second(s), next iface is igb0
[Aug 19 13:15:26] radvd (85849): timer_handler called for igb0
[Aug 19 13:15:26] radvd (85849): ioctl(SIOCGIFFLAGS) succeeded on igb0
[Aug 19 13:15:26] radvd (85849): igb0 is up
[Aug 19 13:15:26] radvd (85849): igb0 is running
[Aug 19 13:15:26] radvd (85849): igb0 supports multicast or is point-to-point
[Aug 19 13:15:26] radvd (85849): mtu for igb0 is 1500
[Aug 19 13:15:26] radvd (85849): link layer token length for igb0 is 48
[Aug 19 13:15:26] radvd (85849): prefix length for igb0 is 64
[Aug 19 13:15:26] radvd (85849): checking ipv6 forwarding of interface not supported
[Aug 19 13:15:26] radvd (85849): igb0 linklocal address: fe80::f6ce:46ff:fea9:7ad0
[Aug 19 13:15:26] radvd (85849): igb0 address: 2607:fcc8:f142:xxxx:f6ce:46ff:fea9:7ad0
[Aug 19 13:15:26] radvd (85849): igb0 address: fe80::f6ce:46ff:fea9:7ad0
[Aug 19 13:15:26] radvd (85849): igb0 is ready
[Aug 19 13:15:26] radvd (85849): checking ipv6 forwarding not supported
[Aug 19 13:15:26] radvd (85849): sending RA to ff02::1 on igb0 (fe80::f6ce:46ff:fea9:7ad0), 5 options (using 104/1210 bytes)
[Aug 19 13:15:26] radvd (85849): igb0 next scheduled RA in 124.646 second(s)
[Aug 19 13:15:26] radvd (85849): polling for 124.645 second(s), next iface is igb0
[Aug 19 13:15:26] radvd (85849): igb0 recvmsg len=104
[Aug 19 13:15:26] radvd (85849): igb0 received a packet
[Aug 19 13:15:26] radvd (85849): igb0 received RA from: fe80::f6ce:46ff:fea9:7ad0 (myself)
[Aug 19 13:15:26] radvd (85849): processed RA on igb0
[Aug 19 13:15:26] radvd (85849): polling for 124.645 second(s), next iface is igb0
[Aug 19 13:15:29] radvd (85849): igb1 recvmsg len=16

Here is the radvd initialization section:

Code: [Select]
[Aug 19 12:56:18] radvd (84538): version 2.18 started
[Aug 19 12:56:18] radvd (84538): igb0 interface definition ok
[Aug 19 12:56:18] radvd (84538): config file, /var/etc/radvd.conf, syntax ok
[Aug 19 12:56:18] radvd (84538): checking ipv6 forwarding not supported
[Aug 19 12:56:18] radvd (84538): radvd startup PID is 84538
[Aug 19 12:56:18] radvd (84538): opened pid file /var/run/radvd.pid
[Aug 19 12:56:18] radvd (84538): locked pid file /var/run/radvd.pid
[Aug 19 12:56:18] radvd (85849): opened pid file /var/run/radvd.pid
[Aug 19 12:56:18] radvd (85849): radvd PID is 85849
[Aug 19 12:56:18] radvd (85849): wrote pid 85849 to pid file: /var/run/radvd.pid
[Aug 19 12:56:18] radvd (84538): child signaled pid file written: 85849
[Aug 19 12:56:18] radvd (84538): Freeing Interfaces
[Aug 19 12:56:18] radvd (84538): freeing interface igb0
[Aug 19 12:56:18] radvd (85849): validated pid file, /var/run/radvd.pid: 85849
[Aug 19 12:56:18] radvd (85849): igb0 if_index changed from 0 to 1
[Aug 19 12:56:18] radvd (85849): ioctl(SIOCGIFFLAGS) succeeded on igb0
[Aug 19 12:56:18] radvd (85849): igb0 is up
[Aug 19 12:56:18] radvd (85849): igb0 is running
[Aug 19 12:56:18] radvd (85849): igb0 supports multicast or is point-to-point
[Aug 19 12:56:18] radvd (85849): mtu for igb0 is 1500
[Aug 19 12:56:18] radvd (85849): link layer token length for igb0 is 48
[Aug 19 12:56:18] radvd (85849): prefix length for igb0 is 64
[Aug 19 12:56:18] radvd (85849): checking ipv6 forwarding of interface not supported
[Aug 19 12:56:18] radvd (85849): igb0 linklocal address: fe80::f6ce:46ff:fea9:7ad0
[Aug 19 12:56:18] radvd (85849): igb0 address: 2607:fcc8:f142:xxxx:f6ce:46ff:fea9:7ad0
[Aug 19 12:56:18] radvd (85849): igb0 address: fe80::f6ce:46ff:fea9:7ad0
[Aug 19 12:56:18] radvd (85849): igb0 is ready
[Aug 19 12:56:18] radvd (85849): setting LinkMTU (1500) for igb0 is not supported
[Aug 19 12:56:18] radvd (85849): setting CurHopLimit (64) for igb0 is not supported
[Aug 19 12:56:18] radvd (85849): ioctl(SIOCGIFFLAGS) succeeded on igb0
[Aug 19 12:56:18] radvd (85849): igb0 is up
[Aug 19 12:56:18] radvd (85849): igb0 is running
[Aug 19 12:56:18] radvd (85849): igb0 supports multicast or is point-to-point
[Aug 19 12:56:18] radvd (85849): mtu for igb0 is 1500
[Aug 19 12:56:18] radvd (85849): link layer token length for igb0 is 48
[Aug 19 12:56:18] radvd (85849): prefix length for igb0 is 64
[Aug 19 12:56:18] radvd (85849): checking ipv6 forwarding of interface not supported
[Aug 19 12:56:18] radvd (85849): igb0 linklocal address: fe80::f6ce:46ff:fea9:7ad0
[Aug 19 12:56:18] radvd (85849): igb0 address: 2607:fcc8:f142:xxxx:f6ce:46ff:fea9:7ad0
[Aug 19 12:56:18] radvd (85849): igb0 address: fe80::f6ce:46ff:fea9:7ad0
[Aug 19 12:56:18] radvd (85849): igb0 is ready
[Aug 19 12:56:18] radvd (85849): checking ipv6 forwarding not supported
[Aug 19 12:56:18] radvd (85849): sending RA to ff02::1 on igb0 (fe80::f6ce:46ff:fea9:7ad0), 5 options (using 104/1210 bytes)
[Aug 19 12:56:18] radvd (85849): igb0 next scheduled RA in 16 second(s)
[Aug 19 12:56:18] radvd (85849): polling for 16 second(s), next iface is igb0
[Aug 19 12:56:18] radvd (85849): igb0 recvmsg len=104
[Aug 19 12:56:18] radvd (85849): igb0 received a packet
[Aug 19 12:56:18] radvd (85849): igb0 received RA from: fe80::f6ce:46ff:fea9:7ad0 (myself)

I'm not sure what to make of this.  Any ideas...?  Thanks!
Title: Re: Acquiring ipv6 address can take 5 to 10 minutes.
Post by: Maurice on January 06, 2021, 01:24:56 pm
This has finally improved with the recent 21.1.a_296 update (development branch). Not sure yet whether it's fixed for good, but radvd's responsiveness seems to be more or less back to what it was pre-20.7.

[Update]
Issue occurred again, so either it's only partially fixed or the apparent improvement was just a coincidence.
[/Update]