Messages

[Links:]

Just replying to also voice my desire and support of this feature.

The Tailscale team has done some great work with this upstream in FreeBSD/pf with the FreeBSD Foundation's support. They called out OPNSense by name, so I do hope this makes its way in to OPNSense once it makes its way in to FreeBSD stable. From the looks of things, it is not yet there - see discussion in the review link below.

There appears to have also been an issue opened in the OPNSense repo asking for support of this, but it was auto-closed.

Links:

• Review thread with (well, sort of) recent discussion: https://reviews.freebsd.org/D11137
• Parent bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219803
• Project page for the work: https://www.freebsd.org/status/report-2024-07-2024-09/eim-nat/
• Tailscale blog post about the work as shared by TheDragon: https://www.freebsd.org/status/report-2024-07-2024-09/eim-nat/
• OPNSense GitHub issue asking for support of this feature: https://github.com/opnsense/core/issues/8384

Quite happy to help.

Oh :D I didn't realize you had hopped on there and ran that. Very cool! I confirm it looks good to me!

Thanks @Franco - those patches didn't actually resolve it for me. To be clear, the issue I am seeing is the ipv6 address on the LAN interface which is configured to track the WAN interface. The WAN interface has been showing it's address properly.

The VM is still up/available, with those 2 patches applied, feel free to hop on if it is useful.

@franco - sounds good. I can keep the VM up for however long it's useful, for this 6rd stuff or whatever else.

@franco sure thing, and I can do better than that! I think it was lost in the thread discussion but I have a dedicated test/dev instance setup on with PPPoE/6rd configured that I can give devs access to, I'll DM you with the ifconfig dump and login info.

Awesome to see this fixed, thanks to everyone who jumped in to help.

I can confirm 21.1 resolves this issue on my Centurylink fiber connection, with fully functional IPv6 from the gateway itself and via IPv6 track interface configuration on my inside interfaces.

I'm not sure if it's related to this or not - but in the interfaces -> overview display for an inside interface setup to track the 6rd interface, it does not display anything in the "IPv6 address" row. Checking the same interface via ifconfig on the shell does show the proper IP address configured. I do not have this behavior on an interface with a V6 address statically assigned.

[19.1.4]

After @mimugmail suggested trying a clean 19.1.x install, I did so this morning and can report that the bug is NOT present in that version. Simply adding the 6rd config to my WAN interface does result in it electing the proper default inet6 gateway, and this is reflected by both netstat -nr -f inet6 | grep default, as well as the opnsense gateway UI.

I may try my previous approach of live-mounting the git repo for different versions from this one onward to see if I can get it to break that way, in which case I'd be closer to being able to actually identifying the code change that introduced the bug! It's a bit of time/effort though and I'm not sure I'll get to it this weekend - but I wanted to share my status so folks can know it's still being worked on.

Here's some screenshots of the various configs + outputs with 19.1.4:

Code Select Expand


root@OPNsense:~ # netstat -nr -f inet6 | grep default
default                           2602:61:7181:4c00::cdab:240   UGS     wan_stf


So I gave this a shot again with tags/19.1 and even back to 18.7, both with no luck. To be honest, I'm not sure my testing strategy is good here - there's just a lot about the system that I'm not familiar with unfortunately.

Here's the script that I've been using to test (but I've also been checking the routing table manually and inspecting the web UI): https://gist.github.com/craSH/5f3996f04387522f3daaf8ee214d8754

Hi, I hadn't after hitting a wall testing all the 19.7 versions, but I still have the test VM up so I'll see about giving it a try with 19.1 versions soon. Thanks for the suggestion.

Thanks @mimugmail! Based on the files I've identified (and others in core), would you expect any need to do a full system build to test, vs. just making changes in a live-mounted source tree? I'm going to try some more hunting with that method, but if there's something I may be missing with that method it'd be good to know.

Perhaps another things you know a bit more about the /tmp/[IF]_router[FSUFFIX] and /tmp/[IF]_defaultgw[FSUFFIX] files in general - their origin and use, and how they differ? I imagine they may just be duplicated due to some legacy reasons, but I'm not quite sure.

Thanks again.

[core]

So I've spent some more time on this today, I don't have a major win to report yet but thought I'd share my progress (and minor findings... unless I'm just totally misunderstanding the internal gateway model architecture, which is also possible!).

First off, it's a lot easier to get started with development than I expected, kudos to the OPNsense folks for making that a quite easy process to jump in to, just follow the single page here and you can start messing around with things in the core repository in under 5 minutes.

https://wiki.opnsense.org/development/workflow.html#packages

tl;dr: git clone https://github.com/opnsense/core ; cd core ; make mount  8)

Ok, so once I did that, I basically started adding log_error() calls everywhere that I thought maybe relevant, particularly around any handling of gateway objects.

In doing so, I found out the following - and again, while it seems interesting to me, I may be totally off-base in my understanding of how the internal gateway models are supposed to work, so consider this purely exploratory so far. All configuration is still the same from post above.

$gateway['gateway'] is empty/null when this conditional checks if it's present: https://github.com/opnsense/core/blob/21.1.a/src/etc/inc/system.inc#L494

This is due to $gateways not containing a gateway item for the 6RD interface in this object: https://github.com/opnsense/core/blob/21.1.a/src/etc/inc/system.inc#L486

I verified that by examining the contents of the all gateway objects object at the time, by adding a new debug variable with the result of $gateways->getGateways(). Here's the state of the 6rd interface when I checked it:

Code Select Expand


    [000020000000004] => Array
        (
            [interface] => opt2
            [weight] => 1
            [ipprotocol] => inet6
            [name] => WAN_CL_6RD
            [descr] => Interface WAN_CL_6RD Gateway
            [monitor_disable] => 1
            [if] => pppoe0
            [dynamic] => 1
            [virtual] => 1
            [priority] => 254
        )




It also seems possibly worth noting that the Gateways.php code uses the temp files /tmp/[IF]_router[FSUFFIX] - but that code doesn't ever seem to do anything with the /tmp/[IF]_defaultgw[FSUFFIX] files. I'm not yet positive what code is responsible for writing those files for the 6rd interfaces.


I tried to work backwards in releases to where I could reproduce the known-good condition, possibly believed to be in something between 19.0 and 19.5, but I actually didn't succeed in doing so. I tried the following, for each release (git tag) of the core repository, live-mounted on my dev system:


Example command to grab a branch:

Code Select Expand

git checkout tags/19.7.4 -b 19.7.4-branch

tags/19.7
tags/19.7.1
tags/19.7.2
tags/19.7.3
tags/19.7.4
tags/19.7.5
tags/21.1.a
tags/21.1.a

Also master as of tonight (d70a1aae03a716508aca9439a7db0f1abbe34957)

Sadly, while the interface/gateway code definitely changed whenever I checked out a different tag (verified by different log messages, as those have changed quite a bit over the 19.x releases), I was unable to get to a state that configured a default inet6 gateway at all upon applying the interface configuration. I wonder if there may be some limitation of the live mounted development/debug process here, vs needing to use a complete system image - there's probably something else that's different which my above methodology just isn't catching.

So, that's where I'm at right now. Happy to continue on this, and I welcome suggestions from anybody with ideas for what to try next :)

[GitHub Issue:]

I'm hoping this can be a place folks can work on the current 6RD default gateway issue on OPNsense that has apparently been present since around version 20.1 from what I've read.

I think the best background and most complete set of information so far is in the commentary of the following GitHub issue. The issue is currently closed, because it never got claimed by someone to do the work.. because of that, I thought starting a fresh forum thread may be best for continued conversation until a concrete problem and fix is determined (I have no interest in discussing or arguing about the project's choice to close issues in the way they do, which caused some spiciness on that issue).

GitHub Issue:
https://github.com/opnsense/core/issues/3903

Other forum posts about this issue & version mentioned:
https://forum.opnsense.org/index.php?topic=14789.0 - 19.7.5
https://forum.opnsense.org/index.php?topic=18571.0 - 20.7
https://forum.opnsense.org/index.php?topic=19691.0 - 20.1 (mentioned that 6RD worked in 19.7)

There may be others, these are just the ones I have come across in my search.

I have not done any development on OPNSense, so I don't quite know where to start in terms of making my own test builds and the like, but I'd be willing to put in effort to get a local dev/test environment in place to be able to work on this.

For now, I can offer up logs of my own setup and even offer access to a dedicated OPNSense install for this issue, which is not in production (i.e. it can be messed with/broken and that's fine). My setup is Centurylink Small Business Fiber (GPON), and their PPPoE / 6rd setup is identical between residential and small business.

I've ran in to this issue on 20.1, 20.7, and currently 21.1.a_201 (the latest development version as of this posting) - I am currently using the OpenSSL variant.

To add some of my own debug info, here's sections of system.log (obtained via `clog -f /var/log/system.log`) when I apply my WAN interface configuration:

IPv4 Configuration Type = PPPoE; IPv6 Configuration Type = None

• PPPoE Username/Password: Filled out w/ CenturyLink details
• Everything else empty/default

I'm including this to show that the messages about the pppoe0 interface not existing (order of operations maybe?) are present regardless - initially I thought it could be important, but I doubt it.

Code Select Expand


Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure openvpn_prepare (,pppoe0)
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure openvpn_prepare (execute task : openvpn_prepare(,pppoe0))
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: /interfaces.php: The command '/sbin/ifconfig 'pppoe0' inet6 -accept_rtadv' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: /interfaces.php: The command `/sbin/ifconfig 'pppoe0' up' failed to execute
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: /interfaces.php: The command '/usr/sbin/ngctl msg 'pppoe0': setautosrc 1' returned exit code '71', the output was 'ngctl: send msg: No such file or directory'
Nov 27 15:28:34 OPNsense kernel: ng0: changing name to 'pppoe0'
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: /interfaces.php: ROUTING: entering configure using 'opt2'
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure ipsec (,opt2)
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure ipsec (execute task : ipsec_configure_do(,opt2))
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure dhcp ()
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure dns ()
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure dns (execute task : dnsmasq_configure_do())
Nov 27 15:28:34 OPNsense opnsense-devel[86324]: plugins_configure dns (execute task : unbound_configure_do())
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: /interfaces.php: ROUTING: entering configure using defaults
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure monitor ()
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure monitor (execute task : dpinger_configure_do())
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (,opt2)
Nov 27 15:28:35 OPNsense kernel: pflog0: promiscuous mode disabled
Nov 27 15:28:35 OPNsense kernel: pflog0: promiscuous mode enabled
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : dyndns_configure_do(,opt2))
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : ntpd_configure_defer())
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : opendns_configure_do())
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : openssh_configure_do(,opt2))
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : unbound_configure_do(,opt2))
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : vxlan_configure_interface())
Nov 27 15:28:35 OPNsense opnsense-devel[86324]: plugins_configure newwanip (execute task : webgui_configure_do(,opt2))


IPv4 Configuration Type = PPPoE; IPv6 Configuration Type = 6rd Tunnel

• PPPoE Username/Password: Filled out w/ CenturyLink details
• 6RD prefix: 205.171.2.64
• 6RD Border Relay: 205.171.2.64
• 6RD IPv4 Prefix length: 0 bits (default)
• 6RD IPv4 Prefix address: Auto-detect (default)
• Everything else empty/default

Code Select Expand


Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure openvpn_prepare (,pppoe0)
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure openvpn_prepare (execute task : openvpn_prepare(,pppoe0))
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: The command '/sbin/ifconfig 'pppoe0' inet6 -accept_rtadv' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: The command `/sbin/ifconfig 'pppoe0' up' failed to execute
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: The command '/usr/sbin/ngctl msg 'pppoe0': setautosrc 1' returned exit code '71', the output was 'ngctl: send msg: No such file or directory'
Nov 27 15:36:21 OPNsense kernel: ng0: changing name to 'pppoe0'
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: The interface IPv4 address '' on interface 'pppoe0' is invalid, not configuring 6RD tunnel
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: entering configure using 'opt2'
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: IPv6 default gateway set to opt2
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: skipping IPv6 default route
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure ipsec (,opt2)
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure ipsec (execute task : ipsec_configure_do(,opt2))
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure dhcp ()
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure dns ()
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure dns (execute task : dnsmasq_configure_do())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure dns (execute task : unbound_configure_do())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: entering configure using defaults
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: IPv6 default gateway set to opt2
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: ROUTING: skipping IPv6 default route
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure monitor ()
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure monitor (execute task : dpinger_configure_do())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: /interfaces.php: The WAN_CL_6RD monitor address is empty, skipping.
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (,opt2)
Nov 27 15:36:21 OPNsense kernel: pflog0: promiscuous mode disabled
Nov 27 15:36:21 OPNsense kernel: pflog0: promiscuous mode enabled
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : dyndns_configure_do(,opt2))
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : ntpd_configure_defer())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : opendns_configure_do())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : openssh_configure_do(,opt2))
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : unbound_configure_do(,opt2))
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : vxlan_configure_interface())
Nov 27 15:36:21 OPNsense opnsense-devel[7331]: plugins_configure newwanip (execute task : webgui_configure_do(,opt2))



As others have noted, my correct IPv6 gateway does get written to both /tmp/opt2_stf_defaultgwv6 and /tmp/opt2_stf_routerv6.


Possibly/probably relevant code locations for this:

• https://github.com/opnsense/core/blob/master/src/etc/rc.newwanip#L112
• https://github.com/opnsense/core/blob/master/src/etc/inc/interfaces.inc#L2709 (definition of interface_6rd_configure())

So, there's what I have so far - does anyone else have suggestions on next steps, or additional debug information they have gathered on the issue? As I mentioned, I'm happy to do whatever I can to help here, including provide access to this test opnsense install and the like.

[...]

When I ping something like 1.1.1.1, my VLAN gateway response with a positive ping result. I'm not sure but guessing that I have a NAT issue.

I'm at the same spot - everything on the Wireguard side is working, but I cannot get traffic to route to it via policy routing, and the opnsense gateway responds to pings/etc when I would expect those packets to traverse Wireguard, for example:

Code Select Expand


$ netstat -nr -f inet
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.0.0.1        0.0.0.0         UG        0 0          0 ens192
10.0.0.0        0.0.0.0         255.255.255.0   U         0 0          0 ens192


$ ping 8.8.8.8 -c1
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.201 ms

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.201/0.201/0.201/0.000 ms


Have others who ran in to this figured out what the issue is? I haven't seen this behavior using similar setups with OpenVPN and the like.

Just jumping in to say that I'm in the same situation here with Centurylink's residential GPON gigabit service. I have dynamic IPs, and have had them administratively remove the VLAN 201 requirement, but otherwise my configuration is the same as described by others here. I'm on a new install of OPNsense, installed as 20.7 and currently on 20.7.1.

@madj42 would you mind sharing details about your gif tunnel setup? I previously had a static IP block for this line but I never setup the GIF tunnel myself.