IPv6 connectivity loss on 24.1.10

[Ben S]

Turning my OpenVPN client back on has broken it.  Getting desperate, I found this:

Code Select Expand

$ sudo tcpdump -i ovpnc1 net fe80::/10 or net ff02::/16
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ovpnc1, link-type NULL (BSD loopback), capture size 262144 bytes

14:03:16.232230 IP6 fe80::xx.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit

The DHCPv6 packet is being sent over the VPN interface.

Any idea why that would happen?  It makes little sense to me.

[Ben S]

I disabled OpenVPN again and on the next attempt, dhcp6c succeeded.

[franco]

I'm not sure, but with another user we found https://github.com/opnsense/core/commit/eb269e0d4a which coincidentally is about destination ff02::/16. It looks like the older ruleset was allowing dhcp6c to piggyback the state of the other rule, but that's no longer possible.


Cheers,
Franco

[Ben S]

Thanks, I've applied that and will report back.

[tofflock]

Hi

I've been running V24.1.9_3 since 20-Jun & everything appeared to be fine on the surface.  Today, since 24.1.10 became available, I cleaned my glasses properly & looked at the dashboard and realised that I had no IPV6 addresses.  (I do have a public V4 (and no CGNAT) which explains  why I (we) hadn't noticed that anything was awry before.

I rebooted the FW and noted that IPV6 was back ok.  I then set to, writing a logging script to check how long it took to fail.  My public V6 address had gone before I'd finished writing the script  ::) .   So I finished the script, rebooted again, and ran the logging script.  It took precisely 10 minutes to lose the V6 address again (sounds like a 600 sec lease to me).

I then did a backup of the config file, and tried to do the update to 24.1.10_1.  That didn't work.  I'm guessing that part of the system was trying to do the upgrade using IPV6, which had silently gone away.

So I rebooted again, and started the update to 24.1.10_1 before I lost my V6 public address.  The update appeared to go ok.  When it had finished, I did a manual reboot (but not a filter reload - because I wasn't sure what that was referring to) and the system has now been up for just over an hour.  My IPV6 logging script shows that my V6 address hasn't gone awol.

So 24.1.10_1 seems to have fixed whatever was wrong in 24.1.9_3 for me, at least.

I do have another live system, also running 24.1.9_3, but that system has a fixed IPV6 public address, but, unsurprisingly, that system shows absolutely no problem.

HTH

PeterF

[Ben S]

Initially https://github.com/opnsense/core/commit/eb269e0d4a is looking good for me.  I've re-enabled OpenVPN and DHCPv6 still renews ok.  I'm going to reboot and make sure a clean boot runs fine.

[franco]

I'll hotfix this now and then we're hopefully back to 24.1.8 state. The rest of the goodies for IPv6 are stashed away on 24.7 which makes this extra work worth the while eventually.


Cheers,
Franco

[meyergru]

While the patch seems to fix the problem with dhcpv6 renewal for me as well, 24.1.10_1 still takes long on reboot to pick up the IPv6 GUA - it does not start right away as before.

[franco]

If you could look into the different timelines during boot between versions that would help.


Cheers,
Franco

[meyergru]

Here is the log from a reboot with the patched 24.1.10_1. You can see that the IPv6 takes ~2 minutes to get at from the start of the WAN configuration, but IDK why... the IPv4 is ready after ~1 minute. I can also see that the wireguard connections are reconnected before there is even an IPv6 - which is a problem if the counterpart is IPv6 only.

Around 2024-07-12T18:38:50, there is a lot going on that fails, like dyndns updates and HAproxy startup (apparently with a segfault).

[Ben S]

I'll hotfix this now and then we're hopefully back to 24.1.8 state. The rest of the goodies for IPv6 are stashed away on 24.7 which makes this extra work worth the while eventually.

24.1.10_2 has been stable for me overnight, with OpenVPN turned back on.  Thanks for the fix.

Ben

[dwasifar]

I'm not sure this is related, but after the upgrade, OPNSense started rejecting all inbound traffic. I haven't done any research yet, just switched over to my backup machine running an older version, but the log errors were mentioning something about default blocking due to state.  Will post the exact error when I get back to that machine.

EDIT: The error is Default deny / State violation rule, and the version is 24.1.10_2-amd64.  ISP is Comcast Business USA.  Is this related, do you think?

EDIT EDIT: I reverted to 24.1 with a fresh install, and everything is working again with the same config.  I'll pass on 24.1.10_x and wait for next update in hope it will settle down.

EDIT³ - Because I am apparently a glutton for punishment, I upgraded the new working installation to 24.1.10_2 to see what would happen. Result: the same thing. All inbound connections blocked with Default deny / State violation rule.

The more I think about it, the more I conclude this is an unrelated issue, so I'm starting a separate thread for the question.

[UserTom]

Also have issues with IPv6 on the latest OPNsense installations with PPPoE.
The firewall can't communicate to the outside world over IPv6. The clients can. I dont know what happened, but it seems to be broken for me, even with a backup restore on a clean install. This worked fine in the past.

[riborg]

My IPv6 is also broken since a few weeks back.

I get this when tcpdumping:

Code Select Expand

# tcpdump -ve -i pflog0 udp port 546 or udp port 547

08:47:53.564963 rule 91/0(match): pass in on igb0: (class 0xe0, hlim 128, next-header UDP (17) payload length: 179) fe80::2022:ff:fe01:1.dhcpv6-server > fe80::2e0:67ff:fe2c:f570.dhcpv6-client: [udp sum ok] dhcp6 advertise (xid=7206b (IA_NA IAID:0 T1:3600 T2:7200 (IA_ADDR 2001:9b0:130:0:ffff:cd9e:c6ea:d722 pltime:54000 vltime:86400)) (IA_PD IAID:0 T1:3600 T2:7200 (IA_PD-prefix 2001:9b1:23ff:6800::/56 pltime:54000 vltime:86400)) (client-ID hwaddr/time type 1 time 700187364 00e0672cf570) (server-ID hwaddr/time type 1 time 660837239 00505687fc2c) (DNS-server 2001:9b0::53:1 2001:9b0::53:2) (status-code UnspecFail))


The last part, (status-code UnspecFail), doesn't look good.

I have tried all tips in this thread, patching back and forth, reverting to 24.1.8 and 24.1.9, nothing helps. Should I revert even further?

One of the patches doesn't seem to work anyway:

Code Select Expand

# opnsense-patch e94baab85

Found local copy of e94baab85, skipping fetch.
1 out of 1 hunks failed while patching etc/inc/filter.lib.inc

From my ISP i get DHCP6-PD with a /56.

[franco]

Not very useful. If that's what the server says it's probably right (even if for the wrong reasons). Check with your ISP.


Cheers,
Franco