OPNsense Forum

Archive => 20.7 Legacy Series => Topic started by: bimmerdriver on May 18, 2020, 07:12:07 pm

Title: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 18, 2020, 07:12:07 pm
I updated my test system and after it restarted, dhcpd6 was not running and would not start. It was a while since I last updated it, so I bootstrapped to a previous version which does not have the problem, then updated to see if the problem returned.

After bootstrapping, here is the version:

OPNsense 20.1.6-amd64
FreeBSD 11.2-RELEASE-p19-HBSD
OpenSSL 1.1.1g 21 Apr 2020

This version works properly.

After updating from the GUI, here is the version:

OPNsense 20.7.b_97-amd64
FreeBSD 11.2-RELEASE-p19-HBSD
OpenSSL 1.1.1g 21 Apr 2020

This version works properly.

After updating from the command line (opnsense-update, opnsense-code core, make upgrade), here is the version:

OPNsense 20.7.b_156-amd64
FreeBSD 11.2-RELEASE-p19-HBSD
OpenSSL 1.1.1g 21 Apr 2020

With this version, dhcpd6 will not start and the Windows 10 client cannot get an IPv6 address.

The only related message in the log is this:

Code: [Select]
opnsense-devel: /usr/local/etc/rc.bootup: Warning! dhcpd_dhcp6_configure() found no suitable IPv6 address on lan
My ISP requires the use of the "Directly send SOLICIT" setting, which appears to be missing.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 18, 2020, 08:51:42 pm
I started up another VM running the latest development version:

OPNsense 20.7.b_97-amd64
FreeBSD 11.2-RELEASE-p19-HBSD
OpenSSL 1.1.1g 21 Apr 2020

It's working properly, but it also doesn't have the Directly send SOLICIT setting.

I started up another VM running the latest release version:

OPNsense 20.1.6-amd64
FreeBSD 11.2-RELEASE-p19-HBSD
OpenSSL 1.1.1g 21 Apr 2020

It's also working properly, but it also doesn't have the Directly send SOLICIT setting.

This is strange. Perhaps the setting disappeared from the GUI, but it's still in the configuration?
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 18, 2020, 10:44:15 pm
Can you grab your dhcpd6.conf and post the contents, obfuscate any GUAs, same goes for dhcp6c.conf, post the contents.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 18, 2020, 10:45:48 pm
You might need the new dhcp6c, in fact I'm pretty sure you will, so do a ps -auxw | grep dhcp6c and see if it's running.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 18, 2020, 10:46:55 pm
Directly send solicit does not exist anymore, and hasn't since 20.1.1 or even earlier.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 18, 2020, 10:52:17 pm
Do this, from the shell.


# cd /usr
# git clone https://github.com/opnsense/dhcp6c.git (https://github.com/opnsense/dhcp6c.git)
# cd dhcp6c
# ./configure
# make
# killall -TERM dhcp6c
# make install


Now reboot.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 18, 2020, 10:59:30 pm
You'll find the dhcp6c debug and no-release options are now in interfaces->settings.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 19, 2020, 12:25:37 am
Can you grab your dhcpd6.conf and post the contents, obfuscate any GUAs, same goes for dhcp6c.conf, post the contents.

Thank you for your reply.

dhcpd6.conf

Code: [Select]
# Server configuration file example for DHCPv6
# From the file used for TAHI tests - addresses chosen
# to match TAHI rather than example block.

# IPv6 address valid lifetime
#  (at the end the address is no longer usable by the client)
#  (set to 30 days, the usual IPv6 default)
default-lease-time 2592000;

# IPv6 address preferred lifetime
#  (at the end the address is deprecated, i.e., the client should use
#   other addresses for new connections)
#  (set to 7 days, the usual IPv6 default)
preferred-lifetime 604800;

# T1, the delay before Renew
#  (default is 1/2 preferred lifetime)
#  (set to 1 hour)
option dhcp-renewal-time 3600;

# T2, the delay before Rebind (if Renews failed)
#  (default is 3/4 preferred lifetime)
#  (set to 2 hours)
option dhcp-rebinding-time 7200;

# Enable RFC 5007 support (same than for DHCPv4)
allow leasequery;

# Global definitions for name server address(es) and domain search list
option dhcp6.name-servers 3ffe:501:ffff:100:200:ff:fe00:3f3e;
option dhcp6.domain-search "test.example.com","example.com";

# Set preference to 255 (maximum) in order to avoid waiting for
# additional servers when there is only one
##option dhcp6.preference 255;

# Server side command to enable rapid-commit (2 packet exchange)
##option dhcp6.rapid-commit;

# The delay before information-request refresh
#  (minimum is 10 minutes, maximum one day, default is to not refresh)
#  (set to 6 hours)
option dhcp6.info-refresh-time 21600;

# The path of the lease file
dhcpv6-lease-file-name "/var/db/dhcpd6.leases";

# Static definition (must be global)
host myclient {
# The entry is looked up by this
host-identifier option
dhcp6.client-id 00:01:00:01:00:04:93:e0:00:00:00:00:a2:a2;

# A fixed address
fixed-address6 3ffe:501:ffff:100::1234;

# A fixed prefix
fixed-prefix6 3ffe:501:ffff:101::/64;

# Override of the global definitions,
# works only when a resource (address or prefix) is assigned
option dhcp6.name-servers 3ffe:501:ffff:100:200:ff:fe00:4f4e;

# For debug (to see when the entry statements are executed)
#  (log "sol" when a matching Solicitation is received)
##if packet(0,1) = 1 { log(debug,"sol"); }
}

host otherclient {
        # This host entry is hopefully matched if the client supplies a DUID-LL
        # or DUID-LLT containing this MAC address.
        hardware ethernet 01:00:80:a2:55:67;

        fixed-address6 3ffe:501:ffff:100::4321;
}

# The subnet where the server is attached
#  (i.e., the server has an address in this subnet)
subnet6 3ffe:501:ffff:100::/64 {
# Two addresses available to clients
#  (the third client should get NoAddrsAvail)
range6 3ffe:501:ffff:100::10 3ffe:501:ffff:100::11;

# Use the whole /64 prefix for temporary addresses
#  (i.e., direct application of RFC 4941)
range6 3ffe:501:ffff:100:: temporary;

# Some /64 prefixes available for Prefix Delegation (RFC 3633)
prefix6 3ffe:501:ffff:100:: 3ffe:501:ffff:111:: /64;
}

# A second subnet behind a relay agent
subnet6 3ffe:501:ffff:101::/64 {
range6 3ffe:501:ffff:101::10 3ffe:501:ffff:101::11;

# Override of the global definitions,
# works only when a resource (address or prefix) is assigned
option dhcp6.name-servers 3ffe:501:ffff:101:200:ff:fe00:3f3e;

}

# A third subnet behind a relay agent chain
subnet6 3ffe:501:ffff:102::/64 {
range6 3ffe:501:ffff:102::10 3ffe:501:ffff:102::11;
}

dhcp6c.conf

Code: [Select]
interface hn0 {
  send ia-pd 0; # request prefix delegation
  request domain-name-servers;
  request domain-name;
  script "/var/etc/dhcp6c_wan_script.sh"; # we'd like some nameservers please
};
id-assoc pd 0 {
  prefix-interface hn1 {
    sla-id 0;
    sla-len 8;
  };
};
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 19, 2020, 12:27:58 am
You might need the new dhcp6c, in fact I'm pretty sure you will, so do a ps -auxw | grep dhcp6c and see if it's running.
dhcp6c isn't running, as you suspected.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 19, 2020, 12:35:48 am
Directly send solicit does not exist anymore, and hasn't since 20.1.1 or even earlier.
I didn't notice that. I guess that's the result of upgrading in place and not paying attention until something doesn't work.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 19, 2020, 12:36:41 am
Do this, from the shell.


# cd /usr
# git clone https://github.com/opnsense/dhcp6c.git (https://github.com/opnsense/dhcp6c.git)
# cd dhcp6c
# ./configure
# make
# killall -TERM dhcp6c
# make install


Now reboot.
Done and it's working.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 19, 2020, 12:44:51 am
You'll find the dhcp6c debug and no-release options are now in interfaces->settings.
Would it be helpful for me to enable debug logging and post the log?
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 19, 2020, 12:52:34 am
No, I think you are fine on that unless you see something is wrong. We did some pretty heavy testing of it during the changes we made to dhcp6c, and I'm pretty sure that dhcp6c and the debug option - as you see now has multiple levels, and no-release should all be fine. The major testing will come as we move forward with multiwan dhcp6. It's halfway there but it needs further work in other areas now.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 19, 2020, 12:59:02 am
There's something else you should also notice, if you take a look at the interfaces->Overview for your WAN, you'll see the delegated prefix and size that's been assigned by your ISP, this is another new addition to dhcp6c and simplifies a few things around dhcpdv6 as we now don't have to rely on the user getting the prefix size correct.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 19, 2020, 02:53:16 am
A few comments / questions.

In Lobby / Dashboard / Gateways, WAN_DHCP6 shows statistics and status, but the address is ~, rather than the (link-local) address of the gateway.

In Interfaces / Overview / WAN, it shows the link-local address of the IPv6 gateway.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 19, 2020, 08:41:03 am
Not on my test unit. It's showing the link-local addres.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: franco on May 19, 2020, 11:06:20 am
If there are no router advertisements that rtsold can pick up (the main cause for the former send SOLICIT option) you do not get a router address notification.

We removed send SOLICIT because we use a hybrid mode now that has no averse affects on users of the former option.


Cheers,
Franco
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 23, 2020, 08:13:49 pm
I used wireshark to capture DHCPv6 and ICMPv6 messages.

When OPNsense starts, the following sequence of messages are exchanged:

send ICMPv6 router solicitation
send DHCPv6 solicit
receive DHCPv6 advertise
receive ICMPv6 router advertisement
send DHCPv6 request
receive DHCPv6 reply

After that, the link is up and the gateway periodically sends router advertisement messages, at random intervals usually not longer than 30 minutes. The router advertisements come from the gateway always using the same link-local address which is used in the default route.

Later on,

Send DHCPv6 renew
Receive DHCPv6 reply

The router advertisements continued after the renew / reply.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 23, 2020, 11:25:50 pm
That's perfectly normal.


Under the original pfSense, and this is where Opnsense was forked from so it suffered from the same problem, with certain ISPs you needed to send a dhcp6 solicit before router advertisements or should I says you had to send both independently. Now, the problem was that pfSense would wait for a advertisement response before it launched dhcp6c, thus no advertisement response, no dhcp6c. Now, my original fix on pfSense was to to give the option to launch dhcpc6 regardless of the state of the advertisements, but the option was also added to turn that on and off.


Franco and co did some research and found that it causes no issues with ISPs that don't require it and works fine with those that do - so now the option to disable it has been removed and you could say it's permanently on, in fact they both run pretty much simultaneously.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 24, 2020, 01:11:53 am
That's perfectly normal.


Under the original pfSense, and this is where Opnsense was forked from so it suffered from the same problem, with certain ISPs you needed to send a dhcp6 solicit before router advertisements or should I says you had to send both independently. Now, the problem was that pfSense would wait for a advertisement response before it launched dhcp6c, thus no advertisement response, no dhcp6c. Now, my original fix on pfSense was to to give the option to launch dhcpc6 regardless of the state of the advertisements, but the option was also added to turn that on and off.


Franco and co did some research and found that it causes no issues with ISPs that don't require it and works fine with those that do - so now the option to disable it has been removed and you could say it's permanently on, in fact they both run pretty much simultaneously.
Understood. OPNsense is receiving RA messages, so I'm not clear why the gateway status on the dashboard reports the address as ~ rather than the actual address.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 24, 2020, 05:05:34 am
Hmm.. I think I might know why.  If you look at system->gateways->single, is that showing an entry?
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 24, 2020, 06:56:37 am
Can you send me the output of netstat -6rW please.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 24, 2020, 05:59:01 pm
Hmm.. I think I might know why.  If you look at system->gateways->single, is that showing an entry?
System / Gateways / Single shows both the IPv4 and IPv6 gateways.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 24, 2020, 06:03:49 pm
Can you send me the output of netstat -6rW please.
Here you go:
Code: [Select]
root@OPNsense:~ # netstat -6rW
Routing tables

Internet6:
Destination        Gateway            Flags       Use    Mtu    Netif Expire
default            fe80::ea4:2ff:fe29:5001%hn0 UG     9025   1500      hn0
localhost          link#1             UH            0  16384      lo0
node-1w7jr9ql8gh2x5hajvvh1dr7k.ipv6.telus.net link#6 U          2894   1500      hn1
OPNsense           link#6             UHS           0  16384      lo0
fe80::%lo0/64      link#1             U             0  16384      lo0
fe80::1%lo0        link#1             UHS           0  16384      lo0
fe80::%hn0/64      link#5             U         83596   1500      hn0
fe80::215:5dff:fe67:5b19%hn0 link#5   UHS           0  16384      lo0
fe80::%hn1/64      link#6             U        128314   1500      hn1
fe80::215:5dff:fe67:5b1a%hn1 link#6   UHS           0  16384      lo0
root@OPNsense:~ #
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 24, 2020, 06:12:11 pm
Hmm.. I think I might know why.  If you look at system->gateways->single, is that showing an entry?
System / Gateways / Single shows both the IPv4 and IPv6 gateways.
So it it's only the lobby widget that's missing the IPv6 gateway then?
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 24, 2020, 06:33:25 pm
Hmm.. I think I might know why.  If you look at system->gateways->single, is that showing an entry?
System / Gateways / Single shows both the IPv4 and IPv6 gateways.
So it it's only the lobby widget that's missing the IPv6 gateway then?
The lobby widget is reporting the IPv6 gateway address as ~ and there is no reported IPv6 gateway address in Interfaces / Overview / Wan. It's not blank or ~. There isn't a line for the address.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 24, 2020, 06:45:04 pm
In the main gateways page the address is under 'Gateway'. Are you saying there's no entry for your WAN dhcp6 in gateways at all?
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 24, 2020, 07:11:53 pm
In the main gateways page the address is under 'Gateway'. Are you saying there's no entry for your WAN dhcp6 in gateways at all?
No, that's not what I meant. System: Gateways: Single shows both gateways. Interfaces: Overview WAN shows information about both gateways, but it does not report an address for the IPv6 gateway. The address isn't blank or ~, there is not even a line for it.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 24, 2020, 08:03:18 pm
now I'm confused. don't worry about interfaces, is the gateways page showing a gateway address for v4 and v6.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 24, 2020, 08:12:44 pm
ok, re-Read the message.  that's pretty odd... can you email me an image of the gateways single page.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 24, 2020, 08:15:34 pm
ok, re-Read the message.  that's pretty odd... can you email me an image of the gateways single page.
Here are some screen captures.

Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 24, 2020, 10:04:22 pm
can you look in /tmp/ and see if you have a file called ***_defaultgw6 ?
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 24, 2020, 11:17:21 pm
can you look in /tmp/ and see if you have a file called ***_defaultgw6 ?
There is no such file.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 24, 2020, 11:39:16 pm
Sorry wrong file. look for ***_routerv6
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 25, 2020, 12:08:42 am
pretty sure we've been down this road before with your system. I think we have to manually add a gateway for dpinger to use.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 25, 2020, 12:25:54 am
pretty sure we've been down this road before with your system. I think we have to manually add a gateway for dpinger to use.
There is hn0_router and hn0_pdinfo, but no hn0_routerv6. We've done a lot of hacking over the past few years, but I don't recall ever having to manually add a gateway. The default route points to the gateway so the system already knows about it. Why would it have to be manually created?
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 25, 2020, 02:51:47 am
because your not getting a response from rtsold probes to your ISP, which in turn means there's no *_routerv6  file which in turn means that the gateway address is not found, so it's not displayed. I guess the option might be to use netstat to find the gateway when all else fails.
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: bimmerdriver on May 25, 2020, 04:53:12 am
Every time I executed /sbin/rtsol, there was an RS and an RA, observed using wireshark.

Here is the output from the command line:

Code: [Select]
root@OPNsense:~ # /sbin/rtsol -a
root@OPNsense:~ # /sbin/rtsol -d -a
probing hn0
root@OPNsense:~ # /sbin/rtsol -D -a
probing hn0
checking if hn0 is ready...
hn0 is ready
set timer for hn0 to 0s
New timer is 0s
timer expiration on hn0, state = 1
send RS on hn0, whose state is 2
set timer for hn0 to 4s
New timer is 4s
received RA from fe80::ea4:2ff:fe29:5001 on hn0, state is 2
Processing RA
ndo = 0x1b285594c20
ndo->nd_opt_type = 1
ndo->nd_opt_len = 1
rsid = [hn0:slaac]
stop timer for hn0
there is no timer
root@OPNsense:~ #
Title: Re: Updated to OPNsense 20.7.b_156-amd64, dhcpd6 not starting
Post by: marjohn56 on May 25, 2020, 09:16:56 am
then the file should exist. Can you email me with the remote access details and I'll log in. I'll need notepad++, winscp and putty available.