[CALL FOR TESTING] PPPoE restructuring and IPv6 improvements

[nanoguy]

You need to revert them in reverse order I'd say.

That's exactly what I did and that's the end result.

[doktornotor]

Well, then don't do it reversed, 

[nanoguy]

Tried the old trick of pretending to change something to both LAN and WAN but saving with no changes, tried to apply and revert the patches again, now IPv6 works as usual. I have no idea why

LE: IPv6 connectivity dropped after a few minutes.

[franco]

Is this on PPPoE or without? I can't tell exactly. There is something going on with dhcp6c as well, but all of this is hard to tell with 24.7.1/24.7.2 kernel being as flaky as it is. Tomorrow's 24.7.3 should make that much better.

In case it's dhcp6c try this:

# opnsense-revert -r 24.7.1 dhcp6c

(reboot needed)


Cheers,
Franco

[franco]

PS: That being said I'm refraining from merging this due to IPv4 connectivity side effect for non-PPPoE. The first patch would make matters worse for these poor people relying on it.

[seacycle]

@seacycle

Thanks for reaching out. TLDR: you're looking for https://github.com/opnsense/core/commit/947e61b1a5

# opnsense-patch 947e61b1a5

…

Can you:

1. Confirm that the patch works? I'd add that to 24.7.3 of course.
2. Confirm that the configuration suggestions works as well on your end?

Please don't go, we need you for this.

The patch does appear to work with the convoluted 6rd-over-pppoe-over-a-tagged-vlan that CenturyLink requires, thanks! (There are some issues with radvd not picking up the prefix reliably without a manual restart, but that pre-dates 24.7.)

[franco]

Ok, thanks for confirming. This one will be in 24.7.3 today.

Interesting note about radvd. Are you using LAN interfaces to track the WAN? Just by memory I think it's true that for 6RD/6to4 there is no event to automatically reconfigure clients. Happy to fix that, but ideally with a ticket on GitHub.

I'd still like clarification on the second half:

The important thing going forward is that these types of setups will no longer work on 25.1 in the way they are currently performed. In 25.1 you will have to delete 6RD from your WAN and create a separate "WAN6" interface from your "port" where PPPoE is running on (something like igb1 for example). There you chose IPv4 None and IPv6 6RD and it should work as before. That being said, the same should already work on all known OPNsense versions but it was favoured by be convenient two-in-one WAN configuration which, again, has been a source of great confusion for at least a decade.

Can you:

[...]
2. Confirm that the configuration suggestions works as well on your end?

Thanks,
Franco

[seacycle]

Interesting note about radvd. Are you using LAN interfaces to track the WAN? Just by memory I think it's true that for 6RD/6to4 there is no event to automatically reconfigure clients. Happy to fix that, but ideally with a ticket on GitHub.

Yes, tracking the WAN interface.  I can put together a ticket.

I'd still like clarification on the second half:

The important thing going forward is that these types of setups will no longer work on 25.1 in the way they are currently performed. In 25.1 you will have to delete 6RD from your WAN and create a separate "WAN6" interface from your "port" where PPPoE is running on (something like igb1 for example). There you chose IPv4 None and IPv6 6RD and it should work as before. That being said, the same should already work on all known OPNsense versions but it was favoured by be convenient two-in-one WAN configuration which, again, has been a source of great confusion for at least a decade.

Can you:

[...]
2. Confirm that the configuration suggestions works as well on your end?

I agree that the two-in-one WAN configuration feels like an abstraction-too-far from the actual topology of the plumbing underneath, leading to confusion. I had thought that the stf tunnel setup would make more sense as an independent "Other Types" interface configuration; make an stf device, assign it to WAN6 and configure as 6rd there. Ultimately, it doesn't seem coupled to the pppoe interface by anything than by the routing table, as far as I understand.

I'm not 100% sure I follow how I would set up what you describe on 24.7, but would like to try it out.  If I have:

igc1 - hardware interface connected to the provider's ONT (unassigned)
vlan01 - pppoe packates need to be tagged with vlan 201, so this is on igc1 (unassigned)
pppoe1 - on top of vlan01 (assigned as WAN)

igc1 and vlan01 are not currently assigned. Are you saying I'd drop the v6 configuration from WAN (pppoe1), assign vlan01 or igc1 to WAN6, and configure the 6rd from that?

[franco]

> igc1 and vlan01 are not currently assigned. Are you saying I'd drop the v6 configuration from WAN (pppoe1), assign vlan01 or igc1 to WAN6, and configure the 6rd from that?

Correct. I believe it would be vlan01. This should confirm it for you from the console:

# ifconfig wan_stf

It seems a bit strange to split IPv4 and IPv6 interfaces but the reality is that it does this anyway even if it looks unified so the two interface approach should be fully compatible.


Cheers,
Franco

[seacycle]

> igc1 and vlan01 are not currently assigned. Are you saying I'd drop the v6 configuration from WAN (pppoe1), assign vlan01 or igc1 to WAN6, and configure the 6rd from that?

Correct. I believe it would be vlan01. This should confirm it for you from the console:

# ifconfig wan_stf

Trial on 24.7.3_1:

• Removed the 6rd configuration from WAN (assigned to the pppoe interface), save, apply.  The 6rd prefix is removed from the routing table, but the wan_stf interface remains partially configured (it has the tv4rbr address still configured, but no inet6 address) and is the tsill default v6 route.
• Reboot, flushes out the wan_stf interface and the associated default v6 route.
• Assign the vlan interface under the pppoe interface to WAN6, enable. IPv4 configured as none, IPv6 configured as 6rd, save apply.  No stf interface gets configured.  Also notice the old 6RD gateway associated with WAN is still listed, marked as defunct. Delete.
• Reboot, still no stf interface.
• Repeat, but use the hardware igc interface under the vlan interface (under the pppoe interface), same result.
• Remove the WAN6 assignment, re-configure 6RD on the WAN assignment (as it was originally), and the stf interface returns with a correct 6rd configuration.

Searching logs for WAN6 I see this a bunch of times:

Code: [Select]

/interfaces.php: The command '/sbin/ifconfig 'opt6_stf' inet6 description 'WAN6 (opt6)' up' returned exit code '1', the output was 'ifconfig: interface opt6_stf does not exist'
So it seems like a code path to create the stf interface is getting missed in this scenario, but later code paths expecting to find it existing is being hit.

With 6rd configured on the WAN assignment (pppoe), the stf interface looks like:

Code: [Select]

wan_stf: flags=1004041<UP,RUNNING,LINK2,LOWER_UP> metric 0 mtu 1280
description: WAN (wan)
options=0
inet6 2602:<redacted>:: prefixlen 24
groups: stf
v4net 0.0.0.0/32 -> tv4br 205.171.2.64
nd6 options=103<PERFORMNUD,ACCEPT_RTADV,NO_DAD>


[franco]

Thanks for testing. All of this is so silly it makes my head hurt.

You're probably seeing a log line that is this:

# opnsense-log | grep "not configuring 6RD tunnel"

So what this means is that 6RD is tied to an IPv4 mode and that should have been validated a long time ago. But it also means 6RD and 6TO4 go through the tunnel after all.

The good news is we don't have to worry about "IPv4 connectivity" knob in this case since we latch on by IP address and not device and so leave the 6RD/6to4 where it currently is. I'll add the required validation and that should be the end of it.


Thank you a lot for these insights,
Franco

[franco]

PS: I was wrong and I'm glad we cleared that up.

[franco]

Commit for reference only: https://github.com/opnsense/core/commit/c17d021b4c5

[franco]

Ok here are the revised patches for 24.7.3:

https://github.com/opnsense/core/commit/6b28dae2

I've added dhclient compatibility for the "IPv4 connectivity" abuse situation by non-PPP setups. I've also fixed a stupid error WRT reading the actual setting correctly. This should address my current concerns and the reason why this fix didn't land in 24.7.3 so far. Although compatibility has been put back for 24.7 I still intend to remove support for non-PPP types in order to come up with an easier fix (because in this case we are not dealing with a volatile PPP interface),

https://github.com/opnsense/core/commit/82ea39b04c

Second commit remained as is.

# opnsense-revert opnsense && opnsense-patch 6b28dae2 82ea39b04c

Since 6b28dae2 is probably more likely to introduce issues it's my goal to bring it into 24.7.4. Some non-PPP testing is appreciated. 24.7.4 will also by default correctly deal and validate with 6RD/6to4 again.

82ea39b04c can follow in 24.7.5

Happy testing and thanks in advance.


Cheers,
Franco

[DiskWizard001]

Currently I have DHCP v6 being shut down due to strange behavior.