OPNsense Forum
Archive => 18.1 Legacy Series => Topic started by: Dronov on February 09, 2018, 02:12:00 pm
-
So I waited for 18.1.2 to be ready and upgraded my up-to-date 17.x box. It went to 18.1.1 only, but for LAN clients everything was working fine (including IPv6). Then I decided to upgrade to 18.1.2 and it was "aborted internally". opnsense-update was hanging on pkg-static invocations.
Well, I thought, I've seen something similar when I had my IPv6 misconfigured. And I tried the relevant pkg operations with -4 flag. It worked. Ooops.
Now, everything was working fine (especially regarding the IPv6 for LAN clients and router itself) on 17.x. But after upgrade only router itself has no IPv6 connectivity. The pings and traceroute6 seems to be working, but no actual data is going through (e.g. curl -6 http://google.com/ just times out without receiving anything).
Any advice appreciated on how to debug it further.
-
Can you give us some info about your ISP IPv6 connection, DHCP or Static, PPPoE or IPoE etc.
-
curl -6 http://google.com/ in the shell just hangs for me using 18.1.2. Using DHCP for IPv6 on the WAN interface.
I don't really use IPv6 for anything, so this post made me aware that IPv6 is probably broken.
-
Static is fine, working here. I'll check v6 dhcp shortly.
-
Good, there are at least two of us affected. I am afraid my setup is unnecessary complicated (historical reasons), but hopefully dcol's set up is simpler.
Anyway, my ISP has no IPv6, so I use a tunnel to my own server, which has native IPv6. ISP gives me DSL (FTTC) PPPoE link, but again it's v4 only. Technically, that "tunnel" I have is just an openvpn connection, which encapsulates both IPv4 and IPv6 for simplicity. So everything that leaves my LAN goes through VPN. OpenVPN uses site local feed::/112 for the link itself. VPN server uses a different v6 network for itself. Allocation is static for the router, other LAN hosts mostly use SLAC (DHCPv6 is configured, but mostly unused AFAIK). Naturally, for all that to work I have a routed block, which is used by openvpn connection (iroute-ipv6).
And now there is an unusual part of my set up: I have NPT (NPTv6 in 18.x) to translate my internal network. This is a legacy thing, not really used anymore, just left there because it was working fine. Translation is done for two real/routable addresses, the LAN does NOT use ULA.
Thanks
-
Yes, my IPv6 is simple with DHCP assignments from the ISP, I think.
Any way to confirm a DHCP assignment from the ISP in the shell?. ipconfig is not supported.
I do have NDP entries for the WAN interface in OPNsense.
-
V6 dhcp is working for me, let's see if we can see why it's not working for you.
In WAN, set debug on for dhcp6c, reboot, DO NOT take the interface down and back up, there's a 50/50 change you'll get multiple dhcp6c clients.
Now, when it's rebooted, from the shell see what's running.
ps -auxw | grep dhcp6c
ps -auxw | grep rtsold
Have a look at the routing log for messages from rtsold and the dhcpd log for messages from dhcp6c
Once you have those post back what you see.
-
Here is the info. Didn't see anything in the log that helped
-
Can you post that dhcpd log, dhcp6c is running, so what's it saying?
-
Feb 9 11:30:45 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb 9 11:30:45 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb 9 11:30:45 firewall dhcpd: All rights reserved.
Feb 9 11:30:45 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb 9 11:30:45 firewall dhcpd: Config file: /etc/dhcpd.conf
Feb 9 11:30:45 firewall dhcpd: Database file: /var/db/dhcpd.leases
Feb 9 11:30:45 firewall dhcpd: PID file: /var/run/dhcpd.pid
Feb 9 11:30:45 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb 9 11:30:45 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb 9 11:30:45 firewall dhcpd: All rights reserved.
Feb 9 11:30:45 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb 9 11:30:45 firewall dhcpd: Wrote 1 leases to leases file.
Feb 9 11:30:45 firewall dhcpd: Listening on BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb 9 11:30:45 firewall dhcpd: Sending on BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb 9 11:30:45 firewall dhcpd: Sending on Socket/fallback/fallback-net
Feb 9 11:30:45 firewall dhcpd: Server starting service.
Feb 9 11:30:45 firewall dhcp6c[28694]: Sending Solicit
Feb 9 11:30:46 firewall dhcp6c[28694]: Sending Solicit
Feb 9 11:30:48 firewall dhcp6c[28694]: Sending Solicit
Feb 9 11:30:53 firewall dhcp6c[28694]: Sending Solicit
Feb 9 11:31:01 firewall dhcp6c[28694]: Sending Solicit
Feb 9 11:31:17 firewall dhcp6c[28694]: Sending Solicit
Feb 9 11:31:23 firewall dhcp6c[28694]: exiting
Feb 9 11:31:58 firewall dhcp6c[40829]: failed to open /usr/local/etc/dhcp6cctlkey: No such file or directory
Feb 9 11:31:58 firewall dhcp6c[40829]: failed initialize control message authentication
Feb 9 11:31:58 firewall dhcp6c[40829]: skip opening control port
Feb 9 11:31:59 firewall dhcp6c[40953]: Sending Solicit
Feb 9 11:32:00 firewall dhcp6c[40953]: Sending Request
Feb 9 11:32:00 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb 9 11:32:00 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb 9 11:32:00 firewall dhcpd: All rights reserved.
Feb 9 11:32:00 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb 9 11:32:00 firewall dhcpd: Config file: /etc/dhcpd.conf
Feb 9 11:32:00 firewall dhcpd: Database file: /var/db/dhcpd.leases
Feb 9 11:32:00 firewall dhcpd: PID file: /var/run/dhcpd.pid
Feb 9 11:32:00 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb 9 11:32:00 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb 9 11:32:00 firewall dhcpd: All rights reserved.
Feb 9 11:32:00 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb 9 11:32:00 firewall dhcpd: Wrote 1 leases to leases file.
Feb 9 11:32:00 firewall dhcpd: Listening on BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb 9 11:32:00 firewall dhcpd: Sending on BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb 9 11:32:00 firewall dhcpd: Sending on Socket/fallback/fallback-net
Feb 9 11:32:00 firewall dhcpd: Server starting service.
Feb 9 11:32:00 firewall dhcp6c[40953]: dhcp6c Received REQUEST
Feb 9 11:32:00 firewall dhcp6c[40953]: status code for PD-0: success
Feb 9 11:32:00 firewall dhcp6c[40953]: add an address 2001:579:839c:a:21b:21ff:fea6:65f9/64 on igb1
Feb 9 11:32:00 firewall dhcp6c[40953]: status code for NA-0: success
Feb 9 11:32:00 firewall dhcp6c[40953]: add an address 2001:579:3f0f:700:15d:8859:7fb0:f424/128 on igb0
Feb 9 11:32:00 firewall dhcp6c: dhcp6c REQUEST on igb0 - running newipv6
Feb 9 11:32:16 firewall dhcp6c[40953]: Start address release
Feb 9 11:32:16 firewall dhcp6c[40953]: Sending Release
Feb 9 11:32:16 firewall dhcp6c[40953]: remove an address 2001:579:3f0f:700:15d:8859:7fb0:f424/128 on igb0
Feb 9 11:32:16 firewall dhcp6c[40953]: Start address release
Feb 9 11:32:16 firewall dhcp6c[40953]: Sending Release
Feb 9 11:32:16 firewall dhcp6c[40953]: remove an address 2001:579:839c:a:21b:21ff:fea6:65f9/64 on igb1
Feb 9 11:32:17 firewall dhcp6c[40953]: Sending Release
Feb 9 11:32:17 firewall dhcp6c[40953]: Sending Release
Feb 9 11:32:19 firewall dhcp6c[40953]: Sending Release
Feb 9 11:32:19 firewall dhcp6c[40953]: Sending Release
Feb 9 11:32:22 firewall dhcp6c[40953]: Sending Release
Feb 9 11:32:23 firewall dhcp6c[40953]: Sending Release
Feb 9 11:32:29 firewall dhcp6c[40953]: Sending Release
Feb 9 11:32:32 firewall dhcp6c[40953]: Sending Release
Feb 9 11:32:42 firewall dhcp6c[40953]: no responses were received
Feb 9 11:32:47 firewall dhcp6c[40953]: no responses were received
Feb 9 11:32:47 firewall dhcp6c[40953]: exiting
Feb 9 11:32:48 firewall dhcp6c[35464]: failed to open /usr/local/etc/dhcp6cctlkey: No such file or directory
Feb 9 11:32:48 firewall dhcp6c[35464]: failed initialize control message authentication
Feb 9 11:32:48 firewall dhcp6c[35464]: skip opening control port
Feb 9 11:32:49 firewall dhcp6c[35589]: Sending Solicit
Feb 9 11:32:50 firewall dhcp6c[35589]: Sending Solicit
Feb 9 11:32:51 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb 9 11:32:51 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb 9 11:32:51 firewall dhcpd: All rights reserved.
Feb 9 11:32:51 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb 9 11:32:51 firewall dhcpd: Config file: /etc/dhcpd.conf
Feb 9 11:32:51 firewall dhcpd: Database file: /var/db/dhcpd.leases
Feb 9 11:32:51 firewall dhcpd: PID file: /var/run/dhcpd.pid
Feb 9 11:32:51 firewall dhcpd: Internet Systems Consortium DHCP Server 4.3.6
Feb 9 11:32:51 firewall dhcpd: Copyright 2004-2017 Internet Systems Consortium.
Feb 9 11:32:51 firewall dhcpd: All rights reserved.
Feb 9 11:32:51 firewall dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb 9 11:32:51 firewall dhcpd: Wrote 1 leases to leases file.
Feb 9 11:32:51 firewall dhcpd: Listening on BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb 9 11:32:51 firewall dhcpd: Sending on BPF/igb1/00:1b:21:a6:65:f9/192.168.1.0/24
Feb 9 11:32:51 firewall dhcpd: Sending on Socket/fallback/fallback-net
Feb 9 11:32:51 firewall dhcpd: Server starting service.
Feb 9 11:32:52 firewall dhcp6c[35589]: Sending Solicit
Feb 9 11:32:56 firewall dhcp6c[35589]: Sending Solicit
Feb 9 11:33:04 firewall dhcp6c[35589]: Sending Solicit
Feb 9 11:33:20 firewall dhcp6c[35589]: Sending Solicit
Feb 9 11:33:52 firewall dhcp6c[35589]: Sending Solicit
Feb 9 11:34:57 firewall dhcp6c[35589]: Sending Solicit
Feb 9 11:36:56 firewall dhcp6c[35589]: Sending Solicit
Feb 9 11:38:49 firewall dhcp6c[35589]: Sending Solicit
-
Well you successfully got a PD and IA once at 11:32, beyond that the ISP's BNG is not responding. Have you tried with both Directly Send Solicit On and Off?
-
Directly Send Solicit On or Off still hangs on curl -6 http://google.com/
-
If all the dhcp6c logs say is sending solicit then forget curl, you're ISP is not responding. With direct solicit off, dhcp6c is launched by rtsold, which happens when the BNG router responds to a RS solicit packet from your router, that's happening, but your isp's BNG is not responding to a dhcp6 solicit, or it does, but very intermittently.
Now, an ISP in the UK used to behave like this, and the solution was to disconnect the modem for about 15 minutes which triggered the BNG into resetting the link, so try that.
Might be worth sniffing the wan and looking at the V6 traffic to confirm it's not responding.
-
If there is some reason I need IPv6, let me know. Otherwise I will just turn it off and ignore it.
Seems like my ISP doesn't use it and I don't use it for anything internally.
Is IPv6 really necessary?
-
Some, but very few sites are ipv6 only. Might not need it now but some time in the future you will, but that time is years away yet. Ipv4 will be around for the foreseeable future. ;)
-
OK, since I am not quite ready to give up my IPv6 connectivity yet, I spent some time looking at tcpdump output (for curl -6 http://google.com/), and here is what I see:
- Outbound packets are leaving with correct(ly NPT'ed) address, which corresponds to an externally visible IPv6 address on the opnsense box itself
- Response packets arrive to the same (correct external IPv6)
- But eventually packets never reach the curl socket, which to me means that either NPT does not catch them or some firewall rules block them directly
I suspect some changes either in NPT or default firewall rules broke my set up, but I have no idea how to troubleshoot it further. At the moment I do not have enough time to turn off NPT, and ultimately I do not want to do this either.
Could someone please give me an advice how to trace those incoming packets further, after I see them on ovpnc interface? Like I said they are clearly either not NPT'ed properly or reject by some new firewall rules.
-
There might be something wrong with the network stack (probably involving PPPoE links). It might not be related, but I'm also having intermittent IPv6 losses (my ISP has IPv6, so I'm not using tunnels). But if the stack gets broken somehow in some cases, might explain your issues (as well as mine).
The first thing to do is for Franco to figure out the culprit and confirm there is a stack issue, so this is the reason I'm not raising any red flags yet.
-
Could someone please give me an advice how to trace those incoming packets further, after I see them on ovpnc interface? Like I said they are clearly either not NPT'ed properly or reject by some new firewall rules.
So what else is working or not working on IPv6, can you ping an address? Do you have ipv6 addresses on the LAN side?
Use the Interfaces->Diagnostics, use both WAN and LAN IPv6 and make sure pings are working from both.
Once i understand what is and what is not working we can look deeper.
-
Well, as far as I can test everything mostly works, except outbound v6 data transmission to the opnsense box itself. LAN is fully v6 enabled and LAN clients have no issues with v6 connectivity. Not a complete list, but some of the major cases I can think of:
- ping6 opnsense works from (any) LAN host
- ping6 google.com works from (any) LAN host
- traceroute6 google.com works from (any) LAN host
- curl -6 http://google.com/ works from (any) LAN host
- curl -6 http://LAN-PLEX-WEB/ works from a LAN host
- ping6 from opnsense to any LAN host works
- traceroute6 from opnsense to any LAN host works
- curl -6 http://LAN-PLEX-WEB/ works from opnbox
- ping6 google.com works from opnsense box
- traceroute6 google.com works from opnsense box
Pings work from command line (using the default source address), and from Interfaces -> Diagnostics:
- v4 ping works from default, LAN, localhost, OpenVPN
- v6 ping works from default, LAN
- v6 ping does NOT work from localhost (sendmsg: No route to host) and OpenVPN (0 packets received, but it's a site local address, not routable)
$ netstat -r
Internet6:
Destination Gateway Flags Netif Expire
::1 link#5 UH lo0
2000::/3 ovpnc2 US ovpnc2
XXXX:YYY:ZZZ::/64 link#2 U igb1
XXXX:YYY:ZZZ::8001 link#2 UHS lo0
fd9d:a224:acef::/1 link#9 U ovpnc2
fe80::%igb0/64 link#1 U igb0
fe80::20d:b9ff:fe4 link#1 UHS lo0
fe80::%igb1/64 link#2 U igb1
fe80::20d:b9ff:fe4 link#2 UHS lo0
fe80::%lo0/64 link#5 U lo0
fe80::1%lo0 link#5 UHS lo0
fe80::%ovpnc1/64 link#8 U ovpnc1
fe80::20d:b9ff:fe4 link#8 UHS lo0
fe80::20d:b9ff:fe4 link#9 UHS lo0
fe80::%pppoe0/64 link#11 U pppoe0
fe80::20d:b9ff:fe4 link#11 UHS lo0
feed::/112 link#9 U ovpnc2
feed::1002 link#9 UHS lo0
XXXX:YYY:ZZZ -- internal prefix, NPT'ed to external one. XXXX:YYY:ZZZ::8001 -- opnsense box address
So the one of few things that does NOT work from opnsense box is curl -6 http://google.com/
Please let me know if I have missed any important pieces.
Thanks.
-
So from the shell, you don't get this
root@OPNsense:~ # curl -6 http://google.com/
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.uk/?gfe_rd=cr&dcr=0&ei=WDSEWpUOy9byB47XgZgN">here</A>.
</BODY></HTML>
-
Right, from the shell on the opnsense box I only get
opnsense@OPNsense:~ % curl -6 http://google.com/
curl: (7) Failed to connect to google.com port 80: Operation timed out
P.S. the same for root user:
root@OPNsense:~ # curl -6 http://google.com/
curl: (7) Failed to connect to google.com port 80: Operation timed out
-
And what V6 rules do you have on the Lan?
-
Unfortunately I an not sure what is the best way to pull that information (please let me know if grep SOMETHING /tmp/rules.debug will be better), here is what I have in GUI: Firewall -> Rules -> LAN (there are mostly default, IIRC):
- Anti-Lockout Rule (it's both v4 and v6 if I am not mistaken)
- Direct LAN IPV6 to Firewall (LAN net -> This Firewall)
- Default allow LAN IPv6 to any rule (LAN net -> any)
Thanks
-
- Direct LAN IPV6 to Firewall (LAN net -> This Firewall)
What's that rule for than?
On my LAN I have one V6 rule, allow any to any.
The WAN rules do all the hard work.
-
Well, to be honest I do not remember why Direct LAN IPV6 to Firewall (LAN net -> This Firewall) rule is there. Perhaps some experiments I've done long time ago. It shouldn't do harm, though. Am I wrong?
At the moment I am thinking of bringing up an opnsense instance in a VM to see if I can reproduce it with minimal customisations, as the last option. Unfortunately, I won't happen any time soon. Any further troubshooting tips greatly appreciated.
-
Well, to be honest I do not remember why Direct LAN IPV6 to Firewall (LAN net -> This Firewall) rule is there. Perhaps some experiments I've done long time ago. It shouldn't do harm, though. Am I wrong?
Rules without a reason worry me. :)
At the very least disable it and find out what, if anything it's doing. I have only ever seen This Firewall rules on the WAN side, then again, that's just in my experience.