OPNsense Forum
Archive => 21.1 Legacy Series => Topic started by: glasi on February 13, 2021, 10:02:23 pm
-
Hi all,
my OPNsense is not assigned an IPv6 address on the WAN interface after starting, but is only assigned after 10 minutes.
As configuration type I use PPPoE for IPv4 and SLAAC for IPv6.
A packet dump is showing that the OPNsense and the provider successfully exchange and acknowledge their interfaces identifiers via PPP IPV6CP. After about 5 seconds, the provider sends a Router Advertisement with information on DNS servers and IPv6 prefix via ICMPv6 . However, OPNsense does not respond to this Router Advertisement.
Another Router Advertisement from my provider 15 seconds later is once again ignored by OPNsense. Hence, I won't get assigned an IPv6 address on the WAN interface.
There seems to be a race condition as OPNsense is not reacting on Router Advertisements during startup process.
To get an IPv6 address assigned on my WAN interfance, I'll have to wait 10 minutes till my provider sends a new Router Advertisement. This Router Advertisement is now being answered by OPNsense with a Neighbor Solicitation message.
Furthermore, there is an issue with DynDNS. I've noticed that the update script is called during the startup process, however at this point in time there is no IPv6 address on the WAN interface. If the IPv6 address is then assigned later (after 10 minutes), the script does not appear to be called any more.
Does anyone have an idea how to solve these issues.
Regards
glasi
-
My Opnsense also suffers from this problem. Assigning of an IPv6 prefix takes very long time, though a packet dump shows that the provider immediately sends router advertisement messages after establishing the pppoe connection. But the Opnsense does never send any router solicit messages.
So, it seems that first advertisement messages get lost, because after receiving a third one after 8 to 10 minutes the WAN interface itself assigns an IPv6 address.
The waiting time can be reduced by manually triggering a router solicit message by executing "rtsol" command in the terminal.
Regarding dyndns my Opnsense shows the same problem not updating the the dynamic dns for IPv6 after an IPv6 address has been assigned to the WAN interface.
-
Thank you for the tip with rtsol command.
I can confirm that a Router Solicitation message is being send when executing rtsol. The provider is then sending DNS server and IPv6 prefix information via Router Advertisment message. OPNsense is responding via Neighbor Solicitation message.
This raises two questions:
1) Why doesn't OPNsense automatically send any Router Solicitation messages?
2) Why is the DynDNS update script not being triggered?
-
Things seem to get a bit messy.
Further analyses are showing that OPNsense does not send any Router Solicitation messages when "Use IPv4 connectivity" option is enabled for IPv4 Configuration Types SLAAC or DPHCv6.
In contrast, Router Solicitation messages can be seen in the packet dump file when "Use IPv4 connectivity" option is disabled. However, OPNsense is then sending the Router Solicitation messages via the wrong interface. Instead of using my PPPoE interface, OPNsense sends these messages via the parent interface ix0_vlan7 without PPPoE encapsulation. That's like sending to /dev/null.
As the documentation is not 100% clear on this, I can't say for sure whether or not it's intended that the parent interace will be used when the option "Use IPv4 connectivity" is disabled (can we be sure that interfaces.inc is doing the right things?).
Nevertheless, Router Solicitation messages should definitely work when "Use IPv4 connectivity" option is enabled.
IMHO, it's a bug.
-
Then raise it on Github.
-
I did some further analysis.
The problem with the missing router solicitation messages first appeared with version 20.7.
With older versions (e.g. 19.7, 20.1) router solicitation messages are sent without any problems.
BTW I've tested the latest version of pfsense (v. 2.5.0) with identical settings. And pfsense sends router solicitation messages without any problems.
So this is definitely broken in OPNsense > 20.1.
-
You'll be fine in the self-proposed competency of pfSense 2.5, I'm sure.
To me this sounds like beating a dead horse with lengthy discussions and solutions for other use cases already in place.
Yes, we know there is an issue with FreeBSD 12.1-RELEASE and, yes, pfSense is using a 12.2-STABLE release that has no official FreeBSD release engineering process and, yes, FreeBSD errata almost never happens on RELEASE.
So is the issue fixed on FreeBSD 12.2 or 12.3? I don't know until someone can pinpoint the issue and upstream patch.
There is quite a lot of time sunk in chasing shadows with upstream so use what works and pray that there are no other issues that appear after 12.1. I guess we only see what we want to see. ;)
Cheers,
Franco
-
There is no need to be sarcastic. The hint that it works with pfsense 2.5 should be understood as constructive information.
Passing problems on to the upstream channel, i. e. FreeBSD 12.1, is just a bit cheap, isn't it?
And no, the problem for sure is not related to FreeBSD 12.1. As mentioned, router solicitation messages do work when the option "Use IPv4 connectivity" is deactivated. And they can also be successfully sent via rtsol command even when the option "Use IPv4 connectivity" is activated.
The problem can be pinpointed to code changes in interfaces.inc in 20.7.r.
Introducing autoprobing parameter (-a) for $rtsoldcommand in function interface_dhcpv6_configure is causing problems as rtsold fails detecting the correct outgoing interface.
rtsold: received RA from fe80::xxxx:xxxx:xxxx:xxxx on an unexpected IF(pppoe0)
Former versions were explicitly specifying the outgoing interface rather than using autoprobing.
And I don't see any reason why we should rely on autoprobing at all.
-
I think the difference here is from my memory of using pfSense and the work I did around dhcp6c on that system it uses rtsol with the name of the interface that's being configured.
I have just completed some testing using a VM of Microtik RouterOS acting as a PPPoE server and sure enough there is no router solicitation sent from Opnsense over the PPPoE interface, however sending a one shot command using rtsol provides the RS and away we go. Looking further into RTSOLD we are using the -a flag which probes all non point to point interfaces, forgive me if I'm wrong, but pppoe is a point to point interface.
Anyway, I replaced the -a with pppoe0 and igb0 ( my test wans ) and sure enough a solicit is sent on the pppoe & igb0 interfaces, I'll raise this as an issue and post my patch later.
https://github.com/opnsense/core/issues/4733
-
Marjohn56,
I am relieved that you could reproduce the problem.
Thanks a lot for creating a Github ticket. Because I have to admit that Github is a black box for me.
As a non-professional programmer I am already more than happy that I could identify rtsold autoprobing as a potential cause for this issue. I invested quite some time to get there.
Nevertheless, I'll spend some time on Github Guides and Tutorials to gain some Github knowledge.
Regards,
glasi
-
There is a patch there now, if you want to try it the command is:
opnsense-patch c0b7165
I've updated it in the last few minutes as I've added a check to make sure the interfaces is enabled, don't want to be sending out packets on an interface that's not!
-
https://github.com/opnsense/src/commit/2485dc0351e
I'm outta here and wholeheartedly recommend any other solution if a one liner for a 9 month old patch that already made it to two production releases is causing such a belated fuzz. It's really not worth any more time.
Cheers,
Franco
-
I for one was not aware that rtsold had been patched, hence I was going by the man pages.
I don't think it's belated, I think it may be related.... to other issues.
-
There is a patch there now, if you want to try it the command is:
opnsense-patch c0b7165
I've updated it in the last few minutes as I've added a check to make sure the interfaces is enabled, don't want to be sending out packets on an interface that's not!
The proposed patch works like a charm. Router solicitation messages are now send via correctly identified interface pppoe0.
-
> As configuration type I use PPPoE for IPv4 and SLAAC for IPv6.
So the PPPoE is too slow to come up as we try to immediately start dhcp6c anyway. We cannot expect to keep all quirks for ISPs alive if we change code to accommodate other setups.
> The proposed patch works like a charm. Router solicitation messages are now send via correctly identified interface pppoe0.
It's not happening.
Cheers,
Franco
-
But the fix to rtsold, which you have just issued a commit for and I've tested also works and does exactly the same thing, whatever as long as it works I don't really care how it's done.
-
> As configuration type I use PPPoE for IPv4 and SLAAC for IPv6.
So the PPPoE is too slow to come up as we try to immediately start dhcp6c anyway. We cannot expect to keep all quirks for ISPs alive if we change code to accommodate other setups.
> The proposed patch works like a charm. Router solicitation messages are now send via correctly identified interface pppoe0.
It's not happening.
Cheers,
Franco
This is absolutely ridiculous.
Neither is PPPoE too slow to come up nor is any quirk for my ISP necessary. PPPoE is ready. The only part not being ready is OPNsense. Hence, OPNsense misses out on ISP's RA messages during startup.
And to be honest, I don't understand why you get in a huff now. We've a good reason raising an issue with codes changes made in 20.7.r. Because it's simply not working as intended. Code changes have been made without paying attention to rtsold man pages. That's sloppy work!
And it's IMHO completely irrelevant that these changes already made it to two production releases.
I'm really concerned how things develop here. The problem was clearly defined and named. Rtsold autoprobing does not work on point-to-point interfaces. Have you ask yourself why this limitation is in place? Rather than fixing the problem at the point where it arises, we are now tinkering with rtsold's source code.
That's insane!
-
Time for you to pack your bags and take your energy elsewhere. Bye bye.
-
So the PPPoE is too slow to come up as we try to immediately start dhcp6c anyway. We cannot expect to keep all quirks for ISPs alive if we change code to accommodate other setups.
Can you please clarify, what kind of quirks do you mean. Looking at the packet dump of pppoe dial-in looks like my provider acts correctly. But as already mentioned, I noticed similar problems with my opnsense box. So, I cannot see any quirks my ISP is doing.
Please consider, the whole problem needs to be divided into two parts.
- Opnsense ignores router advertisements sent by the ISP for period of time after boot-up and pppoe dial-in
- Opnsense does not send router solicit messages after pppoe dial-in
The first one occurs only directly after booting the opnsense box and performing pppoe dial-in. When insctructing the mpd daemon to re-establish the pppoe connection router advertisement messages are immediately accepted and the pppoe interface generates a new SLACC address.
The second should trigger IPSs to send router advertisement messages in case the ISP does not send gracious ones after dial-in. But, this trigger (sending router solicit messages) does not happen as already discussed above.
-
Time for you to pack your bags and take your energy elsewhere. Bye bye.
I am a little bit confused regarding the whole discussion. Emotions are hijacking this thread which helps nobody of us to solve the problem. This project can only survive if work will be honored. And systems analyst's work is equally valuable like the programmer's one.
So, what is the way you like to go?
-
@schnipp - it's dealt with. The patch of interfaces.inc has been deleted as we are going with the patch to rtsold itself. That will make through it in the next release, If you cannot wait then you can either compile the updated version yourself or I've attached the compiled version of rtsold here. Replace the existing /usr/sbin/rtsold with the attached version; make sure the flags are correctly set to 0555. Kill the running rtsold instance and then restart a Wan interface, that should kick the new one into life.
Now lets just relax and enjoy life.
-
Time for you to pack your bags and take your energy elsewhere. Bye bye.
Why should I? I' ll just ignore your bullshit from now on.
-
@schnipp - it's dealt with. The patch of interfaces.inc has been deleted as we are going with the patch to rtsold itself. That will make through it in the next release, If you cannot wait then you can either compile the updated version yourself or I've attached the compiled version of rtsold here. Replace the existing /usr/sbin/rtsold with the attached version; make sure the flags are correctly set to 0555. Kill the running rtsold instance and then restart a Wan interface, that should kick the new one into life.
Thank's a lot.
Now lets just relax and enjoy life.
You're right :).
-
Time for you to pack your bags and take your energy elsewhere. Bye bye.
Why should I? I' ll just ignore your bullshit from now on.
Take it down @glasi. @Franco's new addition is probably teething.. :) He's not a bad bloke really.
-
Wow... simply wow...
-
Ah.. a little flame now and then is good for the soul, just so long as we can move on and forget about it.
-
Exactly.
Nevertheless, I want to add another point.
It seems that with a more extensive setup dhcp6c is being started too late and misses out on my ISP's RA messages. In my case PPPoE link is already up for more than two minutes before dhcp6c starts.
A lot of other tasks are running in between. Time consumimg seem to be e. g.: unbound_configure_do, ipsec_starter.
An empty monitor address for IX0_WAN_SLAAC is also causing delays when I read the logs correctly.
-
dhcp6c doesn't use RA messages. dhcp6c sends out a multicast solicit and waits for a dhcpv6 unicast advertise response from the server, it then sends a request to the server. RS/RA is SLAAC, dhcp6 is a different beast altogether. If sometimes fails if the ISP does not have the 'other configuration' flag set in the original chit chat. dhcp6c is actually started at roughly the same time as rtsold, I'm talking milliseconds. Unlike pfsense the dhcp6c daemon used by opnsense is effectively resident and we rarely shut it down completely, having modified it heavily to allow us to use sighup to add/remove interfaces or config parameters.
-
For me it looks like there is a sequence of actions without proper synchronization during the dial-in procedure. But, currently it's ony a guess. I can try to investigate a little bit more and in case there is an issue raise an appropriate github ticket.
Can everybody live with it? :)
Edit:
let's make the project a success story together…
-
All I can say is that I have run a PPPoE server on a VM serving to my test router and that logs in and gets prefixes without issues. Of course every ISP has a different way of doing things. Some require that an RS is sent before solicit and get upset if that doesn't happen; Sky UK used to be that way, they have now relaxed that but that could be an issue. Difficult to know without seeing all of the logs and a wireshark capture.
-
dhcp6c doesn't use RA messages. dhcp6c sends out a multicast solicit and waits for a dhcpv6 unicast advertise response from the server, it then sends a request to the server. RS/RA is SLAAC, dhcp6 is a different beast altogether. If sometimes fails if the ISP does not have the 'other configuration' flag set in the original chit chat. dhcp6c is actually started at roughly the same time as rtsold, I'm talking milliseconds. Unlike pfsense the dhcp6c daemon used by opnsense is effectively resident and we rarely shut it down completely, having modified it heavily to allow us to use sighup to add/remove interfaces or config parameters.
Thank you for the detailed explanations.
-
For me it looks like there is a sequence of actions without proper synchronization during the dial-in procedure. But, currently it's ony a guess. I can try to investigate a little bit more and in case there is an issue raise an appropriate github ticket.
As mentioned OPNsense is doing a lot stuff during startup. According to my oberservations it seems that the accept_rtadv flag will be set quite late for the pppoe interface. Maybe a reason why OPNsense misses out on ISP's RA messages?
I look forward to your analysis.
-
the acceptrtadv flag is set before rtsold sends out it's first solicit. It's in the same function in interfaces.inc, so that's not what's causing your issue. Some ISPs send out RAs before they get an RS, so don't get fooled by that.
-
had the same problem, can confirm the patch works. thanks a lot!
-
As already mentioned the following two scerarios don't have any causal dependency, thus we need to handle them separately.
- unsolicited router advertisements (immediately) sent by the ISP after the CPE link comes up (pppoe dial-in)
- solicited router advertisements sent by the ISP after requested by router solicit messages
Regarding the first scenario router advertisement messages received by the opnsense are ignored for a couple of minutes after booting the machine. This is caused by the opnsense itself because pppoe dial-in is performed before the network is fully set-up. Internal tests showed up that the "ACCEPT_RTADV" flag is added to the pppoe interface around 2 minutes after the dial-in to the ISP has been performed. The better way is to finish network setup before the link to the ISP will be established. This prevents loosing unsolicited router advertisement messages.
The second sceanrio has something to do with the rtsol daemon (rtsold) and its behavior in sending router solicit messages to the correct interface. So far I haven't looked into that. But again, both scenarios are independent to each other, but have the same goal. Therefore, both of them contribute to obtain an IPv6 address prefix.
I think, improving this situation is needed, because at least in Germany a lot of DSL connections still use pppoe encapsulation and some of them also suffer from 24h disconnects initiated by their ISP. Finally, we need to create appropriate github tickets. But before I'd like to discuss the basics in this forum.
Release: OPNsense 21.1.1-amd64
-
Scenario 1 is likely caused by deferring the IPv6 setup until the IPv4 connectivity is ready because "Use IPv4 connectivity" is checked.
https://github.com/opnsense/core/blob/70f856bf2fa9d2b1e5fc11e0b7e5bbfae04be92e/src/etc/inc/interfaces.inc#L2455
I don't recall that this ever was different. WAN comes up during boot but is not allowed to go through because it messes with the boot sequence. The WAN events are handled after boot is complete, but for PPPoE that is "too late", at least the ISP sees it that way.
https://github.com/opnsense/core/blob/70f856bf2fa9d2b1e5fc11e0b7e5bbfae04be92e/src/etc/rc.newwanip#L44
https://github.com/opnsense/core/blob/70f856bf2fa9d2b1e5fc11e0b7e5bbfae04be92e/src/etc/rc.newwanipv6#L43
https://github.com/opnsense/core/blob/master/src/etc/rc.syshook.d/start/10-newwanip
We can't assume the ISP will send an RA as soon as the IPv4 connectivity comes up or do we? How should we and/or the ISP know we are ready for it?
Cheers,
Franco
-
Maybe this works, I don't know. But it is worth a try. https://github.com/opnsense/core/commit/943db279
Cheers,
Franco
-
Scenario 1 is likely caused by deferring the IPv6 setup until the IPv4 connectivity is ready because "Use IPv4 connectivity" is checked.
Thanks for the links to the code snippets. In my eyes we should at first discuss this topic how it should look like from functional perspective.
I don't recall that this ever was different. WAN comes up during boot but is not allowed to go through because it messes with the boot sequence. The WAN events are handled after boot is complete, but for PPPoE that is "too late", at least the ISP sees it that way.
The description of "Use IPv4 connectivity" and its explanation in the docs is one thing which confuses me. IPv6 requests are not sent over the IPv4 connectivity layer but encapsulated in ppp. It may sound a little bit pedantic but the description can lead to misunderstanding. As far as I understood by activating this function IPv6 packets are sent over the ppp connectivity link and otherwise directly over the parent interface (e.g. the corresponding ethernet interface). Did I understand correctly?
I don't understand what do you mean with "messes the boot sequence". The WAN link should only come up after the interface itself is successfully prepared. Otherwise it's clear that we'll loose control messages for establishing the ip link.
We can't assume the ISP will send an RA as soon as the IPv4 connectivity comes up or do we? How should we and/or the ISP know we are ready for it?
I believe this question cannot be answered with either 'yes' or 'no' for all situations. I know some fiber modems are capable of doing ethernet link monitoring on customer's side perfomed by the ISP. Whether it is used to control the network connection is unknown. I have observed that ISPs like Deutsche Telekom and 1&1 are doing something similar on the pppoe layer. They immediately send RA messages after ppp dial-in succeeds.
So, we should assume they (ISPs) can. In case an ISP sends unsolicited RA messages we are happy, otherwise it doesn't matter and we do a fallback to router solicit messages. We can only win :).
-
Maybe this works, I don't know. But it is worth a try. https://github.com/opnsense/core/commit/943db279
Thanks a lot, I'll give it a try.
-
The description of "Use IPv4 connectivity" and its explanation in the docs is one thing which confuses me. IPv6 requests are not sent over the IPv4 connectivity layer but encapsulated in ppp. It may sound a little bit pedantic but the description can lead to misunderstanding. As far as I understood by activating this function IPv6 packets are sent over the ppp connectivity link and otherwise directly over the parent interface (e.g. the corresponding ethernet interface). Did I understand correctly?
Correct. It's one of those things that is neither wrong or right and thus may lead to false interpretation.
I don't understand what do you mean with "messes the boot sequence". The WAN link should only come up after the interface itself is successfully prepared. Otherwise it's clear that we'll loose control messages for establishing the ip link.
When we talk about DHCP and the boot sequence obtaining an address and configuring the services (routing, firewall) for it takes an synchronous event which messes with the boot sequence that cannot tolerate undefined concurrent behaviour. For this reason pfSense a long time ago prohibited many async paths during boot sequence using a file designating that the boot sequence is in progress.
I believe this question cannot be answered with either 'yes' or 'no' for all situations. I know some fiber modems are capable of doing ethernet link monitoring on customer's side perfomed by the ISP. Whether it is used to control the network connection is unknown. I have observed that ISPs like Deutsche Telekom and 1&1 are doing something similar on the pppoe layer. They immediately send RA messages after ppp dial-in succeeds.
So, we should assume they (ISPs) can. In case an ISP sends unsolicited RA messages we are happy, otherwise it doesn't matter and we do a fallback to router solicit messages. We can only win :).
I revised the patch a bit so that basically we get ready for dealing with SLAAC as soon as PPPoE is "dialling" and created its interface. If that is not soon enough we are still screwed since we cannot push the accept_rtadv flag to a nonexistent interface, but I expect some sort of grace period for the ISP RA so testing the theory is the best approach.
https://github.com/opnsense/core/commit/a0248c7e
Cheers,
Franco
-
FYI - From a reboot to complete configuration on dual wan system, one dhcp/dhcp6 one PPPoE/dhcp6, One LAN and three VLANs all tracking respective WANs ( 3 on dhcp WAN, one on PPPoE WAN ) takes my test system 21 seconds.
-
I revised the patch a bit so that basically we get ready for dealing with SLAAC as soon as PPPoE is "dialling" and created its interface. If that is not soon enough we are still screwed since we cannot push the accept_rtadv flag to a nonexistent interface, but I expect some sort of grace period for the ISP RA so testing the theory is the best approach.
https://github.com/opnsense/core/commit/a0248c7e
I have updated to the latest version of opnsense (21.1.2-amd64) and applied both patches (943db279 and a0248c7e). The first test looks fine. Now, the IPv6 address is immediately assigned to the pppoe interface after dial-up has succeeded. Also the respective IPv6 Dyndns are updated in time. I do some further tests and I'll report report here in case there is something new.
@Franco: Thanks for the patch :).
-
FYI - From a reboot to complete configuration on dual wan system, one dhcp/dhcp6 one PPPoE/dhcp6, One LAN and three VLANs all tracking respective WANs ( 3 on dhcp WAN, one on PPPoE WAN ) takes my test system 21 seconds.
Thank you for the information. The startup time mainly depends on the specific system configuration and used modules. So, it's not really comparable. Startup time of my opnsense takes several minutes. I habe seen that much time is consumed by unbound (updating its blacklists?) and squid (updating its blacklists?). This need some further investigation the next time and can be discussed in separate threads if needed.
-
@schnipp, thanks a lot! landed as an official commit via https://github.com/opnsense/core/commit/f1afe998ad
We will do a bit of regression testing but it can go into a later stable update, maybe 21.1.4 or 21.1.5.
Cheers,
Franco
-
Hi Franco,
I wanna test the new behavior, but not sure which patch(es) to apply on my OPNsense 21.1.2? All of them (943db279 and a0248c7e)?
Regards
Torsten
-
I have applied both patches (943db279 and a0248c7e) on my system and can confirm that the IPv6 address is immediately assigned to the pppoe interface and that DynDNS for IPv6 is being updated accordingly.
A first test is also showing an improvement on my system's boot time by approx. 20%. For me it seems the system was "wasting" time on dyn_dns_status.widget.php beforehand as there was no IPv6 address available ("/widgets/widgets/dyn_dns_status.widget.php: Aborted IPv6 detection: no address for pppoe0").
-
I wanna test the new behavior, but not sure which patch(es) to apply on my OPNsense 21.1.2? All of them (943db279 and a0248c7e)?
Both of them. See also post #41
-
I‘m new to OPNsense and GitHub. Can someone tell me how to patch my OPNsense 21.1.2 with the mentioned patches to solve this problem?
Kind regards
-
Go to the shell and issue the commands
# opnsense-patch 943db279
# opnsense-patch a0248c7e
-
I did some further tests (SLAAC configuration only) and did not see any negative impacts by these two patches :). Can anybody give a positive result for DHCP configuration?
-
Sure, it's ok for me on DHCPv6 in automatic tracking mode.
Cheers,
Franco