OPNsense Forum

Archive => 20.7 Legacy Series => Topic started by: cdine on November 28, 2020, 12:47:23 am

Title: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on November 28, 2020, 12:47:23 am
I'm hoping this can be a place folks can work on the current 6RD default gateway issue on OPNsense that has apparently been present since around version 20.1 from what I've read.

I think the best background and most complete set of information so far is in the commentary of the following GitHub issue. The issue is currently closed, because it never got claimed by someone to do the work.. because of that, I thought starting a fresh forum thread may be best for continued conversation until a concrete problem and fix is determined (I have no interest in discussing or arguing about the project's choice to close issues in the way they do, which caused some spiciness on that issue).

GitHub Issue:
https://github.com/opnsense/core/issues/3903

Other forum posts about this issue & version mentioned:
https://forum.opnsense.org/index.php?topic=14789.0 - 19.7.5
https://forum.opnsense.org/index.php?topic=18571.0 - 20.7
https://forum.opnsense.org/index.php?topic=19691.0 - 20.1 (mentioned that 6RD worked in 19.7)

There may be others, these are just the ones I have come across in my search.

I have not done any development on OPNSense, so I don't quite know where to start in terms of making my own test builds and the like, but I'd be willing to put in effort to get a local dev/test environment in place to be able to work on this.

For now, I can offer up logs of my own setup and even offer access to a dedicated OPNSense install for this issue, which is not in production (i.e. it can be messed with/broken and that's fine). My setup is Centurylink Small Business Fiber (GPON), and their PPPoE / 6rd setup is identical between residential and small business.

I've ran in to this issue on 20.1, 20.7, and currently 21.1.a_201 (the latest development version as of this posting) - I am currently using the OpenSSL variant.

To add some of my own debug info, here's sections of system.log (obtained via `clog -f /var/log/system.log`) when I apply my WAN interface configuration:

IPv4 Configuration Type = PPPoE; IPv6 Configuration Type = None


I'm including this to show that the messages about the pppoe0 interface not existing (order of operations maybe?) are present regardless - initially I thought it could be important, but I doubt it.

Code: [Select]
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure openvpn_prepare (,pppoe0)
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure openvpn_prepare (execute task : openvpn_prepare(,pppoe0))
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: /interfaces.php: The command '/sbin/ifconfig 'pppoe0' inet6 -accept_rtadv' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: /interfaces.php: The command `/sbin/ifconfig 'pppoe0' up' failed to execute
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: /interfaces.php: The command '/usr/sbin/ngctl msg 'pppoe0': setautosrc 1' returned exit code '71', the output was 'ngctl: send msg: No such file or directory'
Nov 27 15:28:34 OPNsense kernel: ng0: changing name to 'pppoe0'
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: /interfaces.php: ROUTING: entering configure using 'opt2'
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure ipsec (,opt2)
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure ipsec (execute task : ipsec_configure_do(,opt2))
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure dhcp ()
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure dns ()
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure dns (execute task : dnsmasq_configure_do())
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure dns (execute task : unbound_configure_do())
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: /interfaces.php: ROUTING: entering configure using defaults
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure monitor ()
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure monitor (execute task : dpinger_configure_do())
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (,opt2)
Nov 27 15:28:35 OPNsense kernel: pflog0: promiscuous mode disabled
Nov 27 15:28:35 OPNsense kernel: pflog0: promiscuous mode enabled
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : dyndns_configure_do(,opt2))
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : ntpd_configure_defer())
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : opendns_configure_do())
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : openssh_configure_do(,opt2))
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : unbound_configure_do(,opt2))
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : vxlan_configure_interface())
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : webgui_configure_do(,opt2))

IPv4 Configuration Type = PPPoE; IPv6 Configuration Type = 6rd Tunnel


Code: [Select]
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure openvpn_prepare (,pppoe0)
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure openvpn_prepare (execute task : openvpn_prepare(,pppoe0))
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: The command '/sbin/ifconfig 'pppoe0' inet6 -accept_rtadv' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: The command `/sbin/ifconfig 'pppoe0' up' failed to execute
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: The command '/usr/sbin/ngctl msg 'pppoe0': setautosrc 1' returned exit code '71', the output was 'ngctl: send msg: No such file or directory'
Nov 27 15:36:21 OPNsense kernel: ng0: changing name to 'pppoe0'
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: The interface IPv4 address '' on interface 'pppoe0' is invalid, not configuring 6RD tunnel
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: entering configure using 'opt2'
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: IPv6 default gateway set to opt2
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: skipping IPv6 default route
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure ipsec (,opt2)
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure ipsec (execute task : ipsec_configure_do(,opt2))
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure dhcp ()
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure dns ()
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure dns (execute task : dnsmasq_configure_do())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure dns (execute task : unbound_configure_do())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: entering configure using defaults
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: IPv6 default gateway set to opt2
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: skipping IPv6 default route
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure monitor ()
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure monitor (execute task : dpinger_configure_do())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: The WAN_CL_6RD monitor address is empty, skipping.
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (,opt2)
Nov 27 15:36:21 OPNsense kernel: pflog0: promiscuous mode disabled
Nov 27 15:36:21 OPNsense kernel: pflog0: promiscuous mode enabled
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : dyndns_configure_do(,opt2))
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : ntpd_configure_defer())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : opendns_configure_do())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : openssh_configure_do(,opt2))
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : unbound_configure_do(,opt2))
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : vxlan_configure_interface())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : webgui_configure_do(,opt2))


As others have noted, my correct IPv6 gateway does get written to both /tmp/opt2_stf_defaultgwv6 and /tmp/opt2_stf_routerv6.


Possibly/probably relevant code locations for this:


So, there's what I have so far - does anyone else have suggestions on next steps, or additional debug information they have gathered on the issue? As I mentioned, I'm happy to do whatever I can to help here, including provide access to this test opnsense install and the like.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on November 28, 2020, 07:05:46 am
So I've spent some more time on this today, I don't have a major win to report yet but thought I'd share my progress (and minor findings... unless I'm just totally misunderstanding the internal gateway model architecture, which is also possible!).

First off, it's a lot easier to get started with development than I expected, kudos to the OPNsense folks for making that a quite easy process to jump in to, just follow the single page here and you can start messing around with things in the core repository in under 5 minutes.

https://wiki.opnsense.org/development/workflow.html#packages

tl;dr: git clone https://github.com/opnsense/core ; cd core ; make mount  8)

Ok, so once I did that, I basically started adding log_error() calls everywhere that I thought maybe relevant, particularly around any handling of gateway objects.

In doing so, I found out the following - and again, while it seems interesting to me, I may be totally off-base in my understanding of how the internal gateway models are supposed to work, so consider this purely exploratory so far. All configuration is still the same from post above.

$gateway['gateway'] is empty/null when this conditional checks if it's present: https://github.com/opnsense/core/blob/21.1.a/src/etc/inc/system.inc#L494

This is due to $gateways not containing a gateway item for the 6RD interface in this object: https://github.com/opnsense/core/blob/21.1.a/src/etc/inc/system.inc#L486

I verified that by examining the contents of the all gateway objects object at the time, by adding a new debug variable with the result of $gateways->getGateways(). Here's the state of the 6rd interface when I checked it:

Code: [Select]
    [000020000000004] => Array
        (
            [interface] => opt2
            [weight] => 1
            [ipprotocol] => inet6
            [name] => WAN_CL_6RD
            [descr] => Interface WAN_CL_6RD Gateway
            [monitor_disable] => 1
            [if] => pppoe0
            [dynamic] => 1
            [virtual] => 1
            [priority] => 254
        )



It also seems possibly worth noting that the Gateways.php code uses the temp files /tmp/[IF]_router[FSUFFIX] - but that code doesn't ever seem to do anything with the /tmp/[IF]_defaultgw[FSUFFIX] files. I'm not yet positive what code is responsible for writing those files for the 6rd interfaces.


I tried to work backwards in releases to where I could reproduce the known-good condition, possibly believed to be in something between 19.0 and 19.5, but I actually didn't succeed in doing so. I tried the following, for each release (git tag) of the core repository, live-mounted on my dev system:


Example command to grab a branch:

Code: [Select]
git checkout tags/19.7.4 -b 19.7.4-branch
tags/19.7
tags/19.7.1
tags/19.7.2
tags/19.7.3
tags/19.7.4
tags/19.7.5
tags/21.1.a
tags/21.1.a

Also master as of tonight (d70a1aae03a716508aca9439a7db0f1abbe34957)

Sadly, while the interface/gateway code definitely changed whenever I checked out a different tag (verified by different log messages, as those have changed quite a bit over the 19.x releases), I was unable to get to a state that configured a default inet6 gateway at all upon applying the interface configuration. I wonder if there may be some limitation of the live mounted development/debug process here, vs needing to use a complete system image - there's probably something else that's different which my above methodology just isn't catching.

So, that's where I'm at right now. Happy to continue on this, and I welcome suggestions from anybody with ideas for what to try next :)
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mimugmail on November 28, 2020, 07:39:36 am
I have no idea about 6rd and also cant reproduce therefore. All I can say is that Gateway Code got a big rewrite with 19.7 release to add Gateway priorities. Just check the major release notes.

I will follow your quest and happy to help where possible
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on November 29, 2020, 04:13:53 am
Thanks @mimugmail! Based on the files I've identified (and others in core), would you expect any need to do a full system build to test, vs. just making changes in a live-mounted source tree? I'm going to try some more hunting with that method, but if there's something I may be missing with that method it'd be good to know.

Perhaps another things you know a bit more about the /tmp/[IF]_router[FSUFFIX] and /tmp/[IF]_defaultgw[FSUFFIX] files in general - their origin and use, and how they differ? I imagine they may just be duplicated due to some legacy reasons, but I'm not quite sure.

Thanks again.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mimugmail on November 29, 2020, 06:35:06 am
You dont even need to make mount. You can edit files directly in /usr/local/opnsense or /usr/local/www

To revert just force a refresh like pkg install  -f opnsense

Difference between Router and defaultgw COULD be the option to mark a gateway as upstream, so it's choose as default when best prio, or dont use it, If it's an internal gateway like a core switch.

If you live in CEST you can reach core devs in IRC, way easier for such quick questions
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mimugmail on December 25, 2020, 07:58:32 pm
Are you still working on this? Easiest thing would be to test against 19.1
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on December 27, 2020, 03:11:52 am
Hi, I hadn't after hitting a wall testing all the 19.7 versions, but I still have the test VM up so I'll see about giving it a try with 19.1 versions soon. Thanks for the suggestion.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on December 28, 2020, 01:58:55 am
So I gave this a shot again with tags/19.1 and even back to 18.7, both with no luck. To be honest, I'm not sure my testing strategy is good here - there's just a lot about the system that I'm not familiar with unfortunately.

Here's the script that I've been using to test (but I've also been checking the routing table manually and inspecting the web UI): https://gist.github.com/craSH/5f3996f04387522f3daaf8ee214d8754
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mimugmail on December 28, 2020, 06:16:33 am
But did you check manually if interface config is correct? Usually restoring config and couple of reboots should show if everything is working or not
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on January 02, 2021, 08:06:55 pm
After @mimugmail suggested trying a clean 19.1.x install, I did so this morning and can report that the bug is NOT present in that version. Simply adding the 6rd config to my WAN interface does result in it electing the proper default inet6 gateway, and this is reflected by both netstat -nr -f inet6 | grep default, as well as the opnsense gateway UI.

I may try my previous approach of live-mounting the git repo for different versions from this one onward to see if I can get it to break that way, in which case I'd be closer to being able to actually identifying the code change that introduced the bug! It's a bit of time/effort though and I'm not sure I'll get to it this weekend - but I wanted to share my status so folks can know it's still being worked on.

Here's some screenshots of the various configs + outputs with 19.1.4:

Code: [Select]
root@OPNsense:~ # netstat -nr -f inet6 | grep default
default                           2602:61:7181:4c00::cdab:240   UGS     wan_stf

(https://s.tlr.im/s/fmvdk7u3.png)
(https://s.tlr.im/s/fwhw4rfz.png)
(https://s.tlr.im/s/plmdkec2.png)
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mimugmail on January 02, 2021, 08:44:39 pm
You can update to latest 19.1 and it will work, after 19.7 it will break
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: k42m on January 23, 2021, 01:39:05 am
Is there currently a fix or workaround?
Setting the default route manually after every reboot is a bit... let's call it inefficient.

I'm a software dev (limited knowledge on system programing though...) and I'd love to help ;)
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mimugmail on January 23, 2021, 06:37:06 am
The biggest problem is that every dev which enough knowledge doesn't have 6RD available to test.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mhofer on January 23, 2021, 09:00:33 pm
Hi, I also have this problem so I did some research today.

The logs show that the IPv6 default route is just ignored:
Code: [Select]
Jan 23 20:24:05 gate1 opnsense[75480]: /interfaces.php: ROUTING: entering configure using defaults
Jan 23 20:24:05 gate1 opnsense[75480]: /interfaces.php: ROUTING: IPv4 default gateway set to opt2
Jan 23 20:24:05 gate1 opnsense[75480]: /interfaces.php: ROUTING: setting IPv4 default route to 144.2.xxx.xxx
Jan 23 20:24:05 gate1 opnsense[75480]: /interfaces.php: ROUTING: keeping current default gateway '144.2.xxx.xxx'
Jan 23 20:24:05 gate1 opnsense[75480]: /interfaces.php: ROUTING: IPv6 default gateway set to opt2
Jan 23 20:24:05 gate1 opnsense[75480]: /interfaces.php: ROUTING: skipping IPv6 default route

I believe the issue is the following lines were missed during the gateway handling rewrite of release 19.7:
https://github.com/opnsense/core/blob/stable/19.1/src/etc/inc/gwlb.inc#L879-L889

As a dirty workaround I did the following:
Code: [Select]
diff --git a/src/opnsense/mvc/app/library/OPNsense/Routing/Gateways.php b/src/opnsense/mvc/app/library/OPNsense/Routing/Gateways.php
index b1efd58d5..bc733d472 100644
--- a/src/opnsense/mvc/app/library/OPNsense/Routing/Gateways.php
+++ b/src/opnsense/mvc/app/library/OPNsense/Routing/Gateways.php
@@ -261,6 +261,9 @@ class Gateways
                         $gwkey = $this->newKey($thisconf['priority'], !empty($thisconf['defaultgw']));
                         // gateway should only contain a valid address, make sure its empty
                         unset($thisconf['gateway']);
+                        if ($ifcfg['ipaddrv6'] == '6rd' && file_exists("/tmp/{$thisconf['interface']}_stf_routerv6")) {
+                            $thisconf['gateway'] = trim(@file_get_contents("/tmp/{$thisconf['interface']}_stf_routerv6"));
+                        }
                         $this->cached_gateways[$gwkey] = $thisconf;
                     } elseif (empty($thisconf['virtual'])) {
                         // skipped dynamic gateway from config, add to $dynamic_gw to handle defunct

Now the default gateway is added correctly again for the 6rd setup.

Unfortunately this is still not enough for my connection as my new ISP only gives me a single /64 which leads in the following unsolved issue:
https://github.com/opnsense/core/issues/4025
I was unable to solve that one in a proper way - it may be FreeBSD related...

Hope someone else can use this information.

Regards
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on January 23, 2021, 10:37:57 pm
Wow, nice work. The code diverged a lot, but I think I'm seeing the actual issue with Gateways.php: it tries to read the wrong _routerv6 file, because the interface for 6RD is "xxx_stf" and not "xxx".

Let me try to fix this. :)


Cheers,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on January 23, 2021, 11:13:42 pm
This should be it https://github.com/opnsense/core/commit/f4e2728b81

# opnsense-patch f4e2728b81

Really hard to find without a test setup so again thank you very much!

The /64 is trickier because in normal routing you want a bigger net on the WAN and a smaller one on the LAN but in this case you only have one and you need to give it to LAN so it thinks it's upstream connected... I will look into it as time permits.


Cheers,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mhofer on January 24, 2021, 12:27:49 pm
Hi Franco

Thanks a lot for the quick investigation!

Unfortunately, your patch does not yet work.
The reason is that the WAN interface is igb0, but the STF interface gets created with the wan interface key (opt2) as prefix instead of igb0, so it is ccalled opt2_stf and /tmp/opt2_stf_routerv6.
However the code currently checks for /tmp/igb0_stf_routerv6.

For now i patched it as follows to make it work:
Code: [Select]
diff --git a/src/opnsense/mvc/app/library/OPNsense/Routing/Gateways.php b/src/opnsense/mvc/app/library/OPNsense/Routing/Gateways.php
index d17bc382c..4a88ba6cf 100644
--- a/src/opnsense/mvc/app/library/OPNsense/Routing/Gateways.php
+++ b/src/opnsense/mvc/app/library/OPNsense/Routing/Gateways.php
@@ -245,6 +245,13 @@ class Gateways
                         }
                         $gwkey = $this->newKey($thisconf['priority'], !empty($thisconf['defaultgw']));
                         $this->cached_gateways[$gwkey] = $thisconf;
+                    } elseif (file_exists("/tmp/{$ifname}{$isuffix}_router{$fsuffix}")) {
+                        $thisconf['gateway'] = trim(@file_get_contents("/tmp/{$ifname}{$isuffix}_router{$fsuffix}"));
+                        if (empty($thisconf['monitor_disable']) && empty($thisconf['monitor'])) {
+                            $thisconf['monitor'] = $thisconf['gateway'];
+                        }
+                        $gwkey = $this->newKey($thisconf['priority'], !empty($thisconf['defaultgw']));
+                        $this->cached_gateways[$gwkey] = $thisconf;
                     } elseif (!empty($ifcfg['gateway_interface']) || substr($ifcfg['if'], 0, 5) == "ovpnc") {
                         // XXX: ditch ovpnc in a major upgrade in the future, supersede with interface setting
                         //      gateway_interface

An alternative might be to change the code where the /tmp file is written.

Regards,
Marcel
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: madj42 on January 24, 2021, 06:05:04 pm
Kind of a stupid question but figured ask anyway.  I am assuming these patches should work on 21.1 as it is now?  I'm waiting for the issue to be completely figured out but wanted to make sure before trying to apply it.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on January 24, 2021, 08:00:30 pm
Hi Marcel,

You are right. We tried to rename the interfaces/files once but the interface name limit can be too short, see: https://github.com/opnsense/core/issues/3222

So here is the missing link with the fallback to config interface name:

https://github.com/opnsense/core/commit/866ec6df74

@madj42: I want this routing fix in the final 21.1 next week. The /64 tracking fix I cannot quickly make promises but Marcel already signalled to help with the remote testing.


Cheers,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: madj42 on January 24, 2021, 09:57:42 pm
I'm not sure I can help with the 64 tracking issue but what I can help with is testing this against my instance I have running on 21.1 production channel.  Would I be alright implementing these two patches together on that?   Assuming I need both from what it looks like?

Trying to offer up another test system since my provider uses RD (I hate it and I wish they would change).

Thank you both for the help in solving this!  I have been using HE for awhile to get around this pain point and would like to switch back.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: madj42 on January 24, 2021, 10:28:59 pm
Decided to bite the bullet and just do it as it could be reversed.  Disabled my HE tunnel and plugged in the RD settings
  Fired right up and I'm up and running.  I'll reboot tonight to test that out.  Thank you both again and if I can help testing anything, else I would love to.  So far this version of 21.1 has been rock solid.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mhofer on January 24, 2021, 10:29:34 pm
I can confirm on opnsense-20.7.8, with:
Code: [Select]
opnsense-patch f4e2728b81 866ec6df74the ipv6 default route for 6rd is recognized again!
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on January 25, 2021, 01:22:41 pm
Super, thanks for confirming! Backported to 21.1 via https://github.com/opnsense/core/commit/30645c77fa

A snapshot update goes out to 21.1-RC1 in about an hour that will already include the fix. 21.1 follows on Thursday according to the current schedule. :)


Cheers,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mhofer on January 25, 2021, 04:09:04 pm
Hi

I think I have a proper solution for my /64 issue.
Instead of setting the calculated /64 subnet length on the _stf interface, I set the original ISP provided subnet length.
And change the gateway to be inside the ISP provided prefix instead of the calculated /64.

https://github.com/mhofer117/core/commit/bbad554d3c

Together with the patches from franco, it is working great so far with connectivity on LAN and from the firewall itself.

@madj42 are you willing to test this on your setup too?
Code: [Select]
opnsense-patch -a mhofer117 bbad554d3c

Regards,
Marcel
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on January 25, 2021, 04:58:31 pm
Hi Marcel,

Looks clever: keep WAN address but widen the prefix anyway -- upstream gateway won't care and ISP prefix routing will do the rest from firewall perspective. :)

Will you open a PR? Can add this to master branch (development version) of 21.1 surely. If madj42 confirms we will add to production version also.


Thank you,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mhofer on January 25, 2021, 05:27:07 pm
I have just created the PR: https://github.com/opnsense/core/pull/4635
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: madj42 on January 25, 2021, 06:44:24 pm
@franco, assuming you just want me to patch my system and report back that it doesn't break anything on my end even though I'm not having the same issue as mhofer?

If so, I'll patch it and reboot after work.

UPDATE: I had some time on a break and patched the system.  Patched the system and rebooted.  All appears to be working but I'd like to see if I notice anything overnight.  I'll be sure to update this post tomorrow.

Also once this is added in, I'm assuming this will be in the development channel and not in the 21.1 production one until Thursday?  Didn't know if you wanted me to test something in the development channel after it's merged just to be sure there are no issues when the production release is released.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on January 25, 2021, 07:25:18 pm
Yes, indeed, that is valuable feedback to rule out a regression on a similar but unaffected setup. Ideally this can go to 21.1 release (not development) now if it works. Since 6RD is in a twilight state with 20.7 anyway and we do not affect 20.7.x users anymore there is a window of opportunity to simply ship an improved 21.1 image.

I'm not sure when you want to update this thread tomorrow as we are going to be kicking the final build around 11 UTC, but appreciate the heads-up anytime.

Again, thanks for this perfect last minute community patch session! :)


Cheers,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: mhofer on January 25, 2021, 07:53:14 pm
Thanks alot madj42 for testing!
And Thank you very much franco for the quick and constructive patch session =)

Will be nice to see 6rd fully functional again in the next release.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: madj42 on January 27, 2021, 01:13:41 am
I apologize, the entire day was busy.  I think we are good as there haven't been any issues here on my end.  Did the latest push to the production channel of 21.1 include the fix?

Thanks again!
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on January 27, 2021, 09:02:15 am
No worries. The RC1 repo is at 21.1.r1_38 which includes the default route fix. The /64 fix follows with the actual 21.1 update.


Cheers,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: madj42 on January 27, 2021, 07:49:32 pm
Thanks franco.  Appreciate the help and response!
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on January 29, 2021, 02:02:24 am
Awesome to see this fixed, thanks to everyone who jumped in to help.

I can confirm 21.1 resolves this issue on my Centurylink fiber connection, with fully functional IPv6 from the gateway itself and via IPv6 track interface configuration on my inside interfaces.

I'm not sure if it's related to this or not - but in the interfaces -> overview display for an inside interface setup to track the 6rd interface, it does not display anything in the "IPv6 address" row. Checking the same interface via ifconfig on the shell does show the proper IP address configured. I do not have this behavior on an interface with a V6 address statically assigned.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on January 29, 2021, 07:15:37 am
@cdine thanks for confirming.

Would you mind sending me an ifconfig dump via PM? Quickest way to get it solved since I can't emulate. :)


Cheers,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on January 29, 2021, 08:58:52 pm
@franco sure thing, and I can do better than that! I think it was lost in the thread discussion but I have a dedicated test/dev instance setup on with PPPoE/6rd configured that I can give devs access to, I'll DM you with the ifconfig dump and login info.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on January 30, 2021, 09:41:10 am
Hi cdine,

Thanks very much. Next week I have time to take a look around and respond.

Someone else reported the same issue though so I think I found a mistake in my refactoring:

https://github.com/opnsense/core/issues/4651#issuecomment-770177933


Cheers,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on January 30, 2021, 06:19:21 pm
@franco - sounds good. I can keep the VM up for however long it's useful, for this 6rd stuff or whatever else.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on February 02, 2021, 02:52:03 pm
Hi cdine,

Yay, the missing address was already fixed by 33544858c0a but then I noticed that IPv6 address wasn't showing and found a very old bug (already annotated with XXX in the meantime but lacking the setup to fix) that misses the correct IPv6 interface and its addresses so that is how 090dd89aa1e was created and now WAN overview is correctly showing the IPv6 address and missing the link-local (not supported by stf(4) it seems).

# opnsense-patch -l
33544858c0a interfaces: unhide primary IPv6 #4651
090dd89aa1e interfaces: finally fix IPv6 misalignment in

Both patches are active and should also hit 21.1.1. Unless you see anything else I don't need the VM anymore.


Thanks a lot,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on February 03, 2021, 03:37:07 am
Thanks @Franco - those patches didn't actually resolve it for me. To be clear, the issue I am seeing is the ipv6 address on the LAN interface which is configured to track the WAN interface. The WAN interface has been showing it's address properly.

The VM is still up/available, with those 2 patches applied, feel free to hop on if it is useful.
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on February 03, 2021, 09:09:50 am
I assume you ran the patches manually, but they were already installed so that removed them... Reapplied them now and it's fine again -- both on dashboard and overview page.


Cheers,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on February 03, 2021, 09:23:49 am
Oh :D I didn't realize you had hopped on there and ran that. Very cool! I confirm it looks good to me!
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: franco on February 03, 2021, 09:39:42 am
Ok, me happy, case closed... all backported to upcoming 21.1.1. Thanks again <3


Cheers,
Franco
Title: Re: 6RD Default Route Issue (19.7.5, 20.1, 20.7, 21.1.a_201)
Post by: cdine on February 03, 2021, 09:41:26 am
Quite happy to help.