Lost IPv6 on the router itself after upgrade

[Dronov]

So I waited for 18.1.2 to be ready and upgraded my up-to-date 17.x box. It went to 18.1.1 only, but for LAN clients everything was working fine (including IPv6). Then I decided to upgrade to 18.1.2 and it was "aborted internally". opnsense-update was hanging on pkg-static invocations.

Well, I thought, I've seen something similar when I had my IPv6 misconfigured. And I tried the relevant pkg operations with -4 flag. It worked. Ooops.

Now, everything was working fine (especially regarding the IPv6 for LAN clients and router itself) on 17.x. But after upgrade only router itself has no IPv6 connectivity. The pings and traceroute6 seems to be working, but no actual data is going through (e.g. curl -6 http://google.com/ just times out without receiving anything).

Any advice appreciated on how to debug it further.

[marjohn56]

Can you give us some info about your ISP IPv6 connection, DHCP or Static, PPPoE or IPoE etc.

[dcol]

curl -6 http://google.com/ in the shell just hangs for me using 18.1.2. Using DHCP for IPv6 on the WAN interface.

I don't really use IPv6 for anything, so this post made me aware that IPv6 is probably broken.

[marjohn56]

Static is fine, working here. I'll check v6 dhcp shortly.

[Dronov]

Good, there are at least two of us affected. I am afraid my setup is unnecessary complicated (historical reasons), but hopefully dcol's set up is simpler.

Anyway, my ISP has no IPv6, so I use a tunnel to my own server, which has native IPv6. ISP gives me DSL (FTTC) PPPoE link, but again it's v4 only. Technically, that "tunnel" I have is just an openvpn connection, which encapsulates both IPv4 and IPv6 for simplicity. So everything that leaves my LAN goes through VPN. OpenVPN uses site local feed::/112 for the link itself. VPN server uses a different v6  network for itself. Allocation is static for the router, other LAN hosts mostly use SLAC (DHCPv6 is configured, but mostly unused AFAIK). Naturally, for all that to work I have a routed block, which is used by openvpn connection (iroute-ipv6).

And now there is an unusual part of my set up: I have NPT (NPTv6 in 18.x) to translate my internal network. This is a legacy thing, not really used anymore, just left there because it was working fine. Translation is done for two real/routable addresses, the LAN does NOT use ULA.

Thanks

[dcol]

Yes, my IPv6 is simple with DHCP assignments from the ISP, I think.
Any way to confirm a DHCP assignment from the ISP in the shell?. ipconfig is not supported.
I do have NDP entries for the WAN interface in OPNsense.

[marjohn56]

V6 dhcp is working for me, let's see if we can see why it's not working for you.

In WAN, set debug on for dhcp6c, reboot, DO NOT take the interface down and back up, there's a 50/50 change you'll get multiple dhcp6c clients.

Now, when it's rebooted, from the shell see what's running.

ps -auxw | grep dhcp6c

ps  -auxw | grep rtsold

Have a look at the routing log for messages from rtsold and the dhcpd log for messages from dhcp6c

Once you have those post back what you see.

[dcol]

Here is the info. Didn't see anything in the log that helped

[marjohn56]

Can you post that dhcpd log, dhcp6c is running, so what's it saying?

[dcol]

Feb  9 11:30:45 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb  9 11:30:45 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb  9 11:30:45 firewall dhcpd: All rights reserved.
Feb  9 11:30:45 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb  9 11:30:45 firewall dhcpd: Config file: /etc/dhcpd.conf
Feb  9 11:30:45 firewall dhcpd: Database file: /var/db/dhcpd.leases
Feb  9 11:30:45 firewall dhcpd: PID file: /var/run/dhcpd.pid
Feb  9 11:30:45 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb  9 11:30:45 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb  9 11:30:45 firewall dhcpd: All rights reserved.
Feb  9 11:30:45 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb  9 11:30:45 firewall dhcpd: Wrote 1 leases to leases file.
Feb  9 11:30:45 firewall dhcpd: Listening on BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb  9 11:30:45 firewall dhcpd: Sending on   BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb  9 11:30:45 firewall dhcpd: Sending on   Socket/fallback/fallback-net
Feb  9 11:30:45 firewall dhcpd: Server starting service.
Feb  9 11:30:45 firewall dhcp6c[28694]: Sending Solicit
Feb  9 11:30:46 firewall dhcp6c[28694]: Sending Solicit
Feb  9 11:30:48 firewall dhcp6c[28694]: Sending Solicit
Feb  9 11:30:53 firewall dhcp6c[28694]: Sending Solicit
Feb  9 11:31:01 firewall dhcp6c[28694]: Sending Solicit
Feb  9 11:31:17 firewall dhcp6c[28694]: Sending Solicit
Feb  9 11:31:23 firewall dhcp6c[28694]: exiting
Feb  9 11:31:58 firewall dhcp6c[40829]: failed to open /usr/local/etc/dhcp6cctlkey: No such file or directory
Feb  9 11:31:58 firewall dhcp6c[40829]: failed initialize control message authentication
Feb  9 11:31:58 firewall dhcp6c[40829]: skip opening control port
Feb  9 11:31:59 firewall dhcp6c[40953]: Sending Solicit
Feb  9 11:32:00 firewall dhcp6c[40953]: Sending Request
Feb  9 11:32:00 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb  9 11:32:00 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb  9 11:32:00 firewall dhcpd: All rights reserved.
Feb  9 11:32:00 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb  9 11:32:00 firewall dhcpd: Config file: /etc/dhcpd.conf
Feb  9 11:32:00 firewall dhcpd: Database file: /var/db/dhcpd.leases
Feb  9 11:32:00 firewall dhcpd: PID file: /var/run/dhcpd.pid
Feb  9 11:32:00 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb  9 11:32:00 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb  9 11:32:00 firewall dhcpd: All rights reserved.
Feb  9 11:32:00 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb  9 11:32:00 firewall dhcpd: Wrote 1 leases to leases file.
Feb  9 11:32:00 firewall dhcpd: Listening on BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb  9 11:32:00 firewall dhcpd: Sending on   BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb  9 11:32:00 firewall dhcpd: Sending on   Socket/fallback/fallback-net
Feb  9 11:32:00 firewall dhcpd: Server starting service.
Feb  9 11:32:00 firewall dhcp6c[40953]: dhcp6c Received REQUEST
Feb  9 11:32:00 firewall dhcp6c[40953]: status code for PD-0: success
Feb  9 11:32:00 firewall dhcp6c[40953]: add an address 2001:579:839c:a:21b:21ff:fea6:65f9/64 on igb1
Feb  9 11:32:00 firewall dhcp6c[40953]: status code for NA-0: success
Feb  9 11:32:00 firewall dhcp6c[40953]: add an address 2001:579:3f0f:700:15d:8859:7fb0:f424/128 on igb0
Feb  9 11:32:00 firewall dhcp6c: dhcp6c REQUEST on igb0 - running newipv6
Feb  9 11:32:16 firewall dhcp6c[40953]: Start address release
Feb  9 11:32:16 firewall dhcp6c[40953]: Sending Release
Feb  9 11:32:16 firewall dhcp6c[40953]: remove an address 2001:579:3f0f:700:15d:8859:7fb0:f424/128 on igb0
Feb  9 11:32:16 firewall dhcp6c[40953]: Start address release
Feb  9 11:32:16 firewall dhcp6c[40953]: Sending Release
Feb  9 11:32:16 firewall dhcp6c[40953]: remove an address 2001:579:839c:a:21b:21ff:fea6:65f9/64 on igb1
Feb  9 11:32:17 firewall dhcp6c[40953]: Sending Release
Feb  9 11:32:17 firewall dhcp6c[40953]: Sending Release
Feb  9 11:32:19 firewall dhcp6c[40953]: Sending Release
Feb  9 11:32:19 firewall dhcp6c[40953]: Sending Release
Feb  9 11:32:22 firewall dhcp6c[40953]: Sending Release
Feb  9 11:32:23 firewall dhcp6c[40953]: Sending Release
Feb  9 11:32:29 firewall dhcp6c[40953]: Sending Release
Feb  9 11:32:32 firewall dhcp6c[40953]: Sending Release
Feb  9 11:32:42 firewall dhcp6c[40953]: no responses were received
Feb  9 11:32:47 firewall dhcp6c[40953]: no responses were received
Feb  9 11:32:47 firewall dhcp6c[40953]: exiting
Feb  9 11:32:48 firewall dhcp6c[35464]: failed to open /usr/local/etc/dhcp6cctlkey: No such file or directory
Feb  9 11:32:48 firewall dhcp6c[35464]: failed initialize control message authentication
Feb  9 11:32:48 firewall dhcp6c[35464]: skip opening control port
Feb  9 11:32:49 firewall dhcp6c[35589]: Sending Solicit
Feb  9 11:32:50 firewall dhcp6c[35589]: Sending Solicit
Feb  9 11:32:51 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb  9 11:32:51 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb  9 11:32:51 firewall dhcpd: All rights reserved.
Feb  9 11:32:51 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb  9 11:32:51 firewall dhcpd: Config file: /etc/dhcpd.conf
Feb  9 11:32:51 firewall dhcpd: Database file: /var/db/dhcpd.leases
Feb  9 11:32:51 firewall dhcpd: PID file: /var/run/dhcpd.pid
Feb  9 11:32:51 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb  9 11:32:51 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb  9 11:32:51 firewall dhcpd: All rights reserved.
Feb  9 11:32:51 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb  9 11:32:51 firewall dhcpd: Wrote 1 leases to leases file.
Feb  9 11:32:51 firewall dhcpd: Listening on BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb  9 11:32:51 firewall dhcpd: Sending on   BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb  9 11:32:51 firewall dhcpd: Sending on   Socket/fallback/fallback-net
Feb  9 11:32:51 firewall dhcpd: Server starting service.
Feb  9 11:32:52 firewall dhcp6c[35589]: Sending Solicit
Feb  9 11:32:56 firewall dhcp6c[35589]: Sending Solicit
Feb  9 11:33:04 firewall dhcp6c[35589]: Sending Solicit
Feb  9 11:33:20 firewall dhcp6c[35589]: Sending Solicit
Feb  9 11:33:52 firewall dhcp6c[35589]: Sending Solicit
Feb  9 11:34:57 firewall dhcp6c[35589]: Sending Solicit
Feb  9 11:36:56 firewall dhcp6c[35589]: Sending Solicit
Feb  9 11:38:49 firewall dhcp6c[35589]: Sending Solicit

[marjohn56]

Well you successfully got a PD and IA once at 11:32, beyond that the ISP's BNG is not responding. Have you tried with both Directly Send Solicit On and Off?

[dcol]

Directly Send Solicit On or Off still hangs on curl -6 http://google.com/

[marjohn56]

If all the dhcp6c logs say is sending solicit then forget curl, you're ISP is not responding. With direct solicit off, dhcp6c is launched by rtsold, which happens when the BNG router responds to a RS solicit packet from your router, that's happening, but your isp's BNG is not responding to a dhcp6 solicit, or it does, but very intermittently.

Now, an ISP in the UK used to behave like this, and the solution was to disconnect the modem for about 15 minutes which triggered the BNG into resetting the link, so try that.

Might be worth sniffing the wan and looking at the V6 traffic to confirm it's not responding.

[dcol]

If there is some reason I need IPv6, let me know. Otherwise I will just turn it off and ignore it.
Seems like my ISP doesn't use it and I don't use it for anything internally.
Is IPv6 really necessary?

[marjohn56]

Some, but very few sites are ipv6 only. Might not need it now but some time in the future you will, but that time is years away yet. Ipv4 will be around for the foreseeable future. ;)