Hi,
IssueI have a GRE tunnel configured between IPv4 hosts with IPv6 addresses inside of it. I noticed that the netmask parameter set on the GIF (https://github.com/opnsense/core/blob/2862eaf0cc43321ba6574ca331e8066c6ae8d7bd/src/www/interfaces_gif_edit.php#L223) or GRE (https://github.com/opnsense/core/blob/2862eaf0cc43321ba6574ca331e8066c6ae8d7bd/src/www/interfaces_gre_edit.php#L196) configuration page was completely ignored here (https://github.com/opnsense/core/blob/f9ea49ae317b6e65e5f7a6fa12d6bab7d766bfab/src/etc/inc/interfaces.inc#L592-L596). As far as I remember, this behavior occurred even prior to 22.7_4.
If this is a mistake I can prepare a PR so that you could distribute a fix in the next release (e.g. 22.7.1). Otherwise, if this was made intentionally, does anybody know why the 128 prefix length was hardcoded to the GIF/GRE tunnel config?
ImplicationI'm interested in BGP working over IPv6 tunnel addresses. While the tunnel endpoints are pingable and static routes work correctly, the hosts won't establish a BGP session on GUA IPv6 or won't copy routes to kernel tables on LL IPv6. After a short debugging I noticed that the BGP daemon considered the other endpoint as an invalid nexthop if a tunnel was configured with a 128 prefix length.
Possible fixI tested changing this code
mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " " . escapeshellarg($gre['tunnel-remote-addr']) . " prefixlen 128");
to the following
mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " prefixlen " . $gre['tunnel-remote-net']);and for me it has absolutely fixed the BGP over IPv6-in-IPv4 GRE issue.
In my view, the final solution could even be conditional to the prefix length:
- 128 - ifconfig IFACE inet6 LOCAL REMOTE prefixlen 128
- other - ifconfig IFACE inet6 LOCAL prefixlen NET
Are you using a net size greater than /64?
In general this looks like it should be fixed indeed since we take the parameter for the net size anyway we can enforce it and the standard of /64 vs. /128 shouldn't break anything.
Cheers,
Franco
@franco Thank you for a fast reply. I'm using /126 for IPv6 inside the tunnel which provides for 4 addresses just like /30 in IPv4.
Thanks for explaining. Did you maybe mean to propose:
mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " " . escapeshellarg($gre['tunnel-remote-addr']) . " prefixlen " . $gre['tunnel-remote-net']);
?
As far as can see IPv4 does the same.
Cheers,
Franco
Flashback to 2014 sure seems funny now:
https://github.com/pfsense/pfsense/commit/cdeaf91e14e6e2
;)
Cheers,
Franco
It's not as straight-forward as I hoped:
# /sbin/ifconfig gre2 inet6 23::333 1::4 prefixlen 64
ifconfig: ioctl (SIOCAIFADDR): Invalid argument
This works but is distinctively different from the previous...
# /sbin/ifconfig gre2 inet6 23::333 prefixlen 64
Adding a destination address to the ifconfig command requires prefixlen 128
So maybe as a compromise we could make a second mode by inputting the "::" or "0.0.0.0" remote address which omits this destination from the ifconfig command as you suggested?
Cheers,
Franco
@franco Thank you for digging into the issue.
First of all, no, we don't need any remote address when prefixlen is less than 128. Otherwise, as you mentioned, ifconfig returns an error and won't assign the address. So the code I posted is correct:
mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " prefixlen " . $gre['tunnel-remote-net']);
Secondly, it's worth discussing why/when we need to set a remote address. In my view, the situation is quite straightforward:
- If we choose a point-to-point mode, it means /128 should be used. When using /128 the kernel routing table needs a separate entry to be configured for a remote side, otherwise there will be no route to the gateway error. A user should note that BGP won't work in this mode (at least without special changes to the config).
- If we choose a broadcast mode, it means /126 and less should be used (not sure about /127). This case provides for at least 4 addresses: 0 - network, 1 - the first side, 2 - the second side, 3 - broadcast. Then the kernel routing table already knows that the gateway is accessible on the interface, thus no need to configure a remote address route separately. In this mode BGP works flawlessly.
As far as the 2014 discussion is concerned, it is absolutely correct that 128 prefix length is enforced by the kernel
in the point-to-point mode, i.e. when we explicitly set a remote address. But the ones who developed that part of the code seem to have missed
the broadcast mode I described above.
Thirdly, when it comes to your suggestion, I believe we can't input "::" or "0.0.0.0" to the remote field as this is the value used as a gateway address substitute for "dynamic" on the single gateway configuration page. At least it won't work if one (like me) wants to use the tunnel remote address as a default gateway for IPv6.
So my suggestion would be:
if ($gre['tunnel-remote-net']) == 128) {
mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " " . escapeshellarg($gre['tunnel-remote-addr']) . " prefixlen 128");
} else {
mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " prefixlen " . $gre['tunnel-remote-net']);
}
This code can be further simplified to:
mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . ($gre['tunnel-remote-net'] == 128 ? " " . escapeshellarg($gre['tunnel-remote-addr']) : "") . " prefixlen " . $gre['tunnel-remote-net']);
The problem is most people have 64 set for their IPv6 remote net since that is the default proposed by the GUI.
I've changed the default here but we do have a historic mess...
https://github.com/opnsense/core/commit/111a2560fb7b
BTW, in IPv6 network and broadcast address are not part of the address range so /127 are two valid addresses to use.
You are right about the gateway address issue. I think we don't have much choice but to add a new checkbox for this.
Cheers,
Franco
@franco Even if most people have 64 set in the web interface, their routing tables do not have a /64 route since the prefix length value has (always/for a long time) been ignored. Changing the code from what it was (128 hardcoded) to what I propose will ensure they have a correct /64 route, which by default includes the remote side of the tunnel. So in my view, nothing will break, and no need to change the defaults.
Just to illustrate what I mean:
- Up to now. Say you have 2a01::1 local, 2a01::2 remote and /64 prefix length set in the web interface. Then you have the following in your kernel routing table.
ipv6 2a01::2 link#10 UH NaN 1476 gre0 TEST_GRE_IFACE
ipv6 2a01::1 link#10 UHS NaN 16384 lo0 Loopback
- After the change I propose is implemented. Same config in the web interface. Then you have the following in your kernel routing table.
ipv6 2a01::/64 link#10 U NaN 1476 gre0 TEST_GRE_IFACE
ipv6 2a01::1 link#10 UHS NaN 16384 lo0 Loopback
Please, pay attention to absence of /64 in the first case and presence of the H flag (not sure what it means).
I'm not convinced yet but if that works that would be good. Maybe Maurice has a setup to help test this.
Cheers,
Franco
@franco I will tell you more, this issue is also relevant to IPv4. Say you have 172.17.34.1 local, 172.17.34.2 remote and 24 netmask in the web interface. Then you have the following in your routing rable.
ipv4 172.17.34.1 link#13 UHS NaN 16384 lo0 Loopback
ipv4 172.17.34.2 link#13 UH NaN 1476 gre1 TEST_GRE_IFACE
"ifconfig IFACE inet LOCAL REMOTE netmask MASK" actually ignores the netmask whatever is set. Unfortunately, I haven't found a way to run ifconfig without remote address for inet family. Seems like a FreeBSD kernel limitation. But this is another story.
Update: Furthermore, if I try to assign an IPv4 alias to IPv4-in-IPv4 GRE tunnel, I also get an error:
/firewall_virtual_ip.php: The command '/sbin/ifconfig 'gre1' inet '172.17.35.1'/'24' alias ' returned exit code '1', the output was 'ifconfig: ioctl (SIOCAIFADDR): Destination address required'
I've never used GRE and haven't used 6in4 GIF for quite a few years. That being said, what you have figured out makes sense to me (mostly ;)).
Quote from: franco on August 04, 2022, 10:29:46 AM
BTW, in IPv6 network and broadcast address are not part of the address range so /127 are two valid addresses to use.
Yep.
Quote from: franco on August 04, 2022, 10:29:46 AM
You are right about the gateway address issue. I think we don't have much choice but to add a new checkbox for this.
Do we actually need a gateway address? We have the 'Dynamic gateway policy' option in the interface settings specifically for tunnel interfaces without a gateway address. I'm using this with other tunnel interfaces (WireGuard, Tayga). @vnxme, could you try whether this works with GRE and GIF?
Quote from: vnxme on August 04, 2022, 10:47:53 AM
Say you have 2a01::1 local, 2a01::2 remote and /64 prefix length set in the web interface.
But what if local and remote address are not actually in the same /64? Say you have 2001:db8:1::1 local, 2001:db8:2::1 remote and the default /64 prefix length configured in the UI. This currently works because the /64 is ignored and a host route for the remote address is created instead. If we now configure the interface with 2001:db8:1::1/64, the remote address might become unreachable.
It might be better to write some migration code which changes the prefix length of existing setups to 128. Or, if 'Dynamic gateway policy' works, ...
Quote from: franco on August 04, 2022, 08:20:26 AM
So maybe as a compromise we could make a second mode by inputting the "::" or "0.0.0.0" remote address which omits this destination from the ifconfig command as you suggested?
... do this. Though just leaving the remote address field empty to enable the new mode would seem more intuitive to me.
Quote from: vnxme on August 04, 2022, 11:49:08 AM
"ifconfig IFACE inet LOCAL REMOTE netmask MASK" actually ignores the netmask whatever is set.
It doesn't ignore it. When you check the interface config with ifconfig IFACE, you can see that the netmask is applied correctly (inet LOCAL --> REMOTE netmask MASK).
Quote from: vnxme on August 04, 2022, 11:49:08 AM
Unfortunately, I haven't found a way to run ifconfig without remote address for inet family.
That's a known FreeBSD limitation. The destination address of tunnel interfaces can't be omitted in the IPv4 stack, but it can in the IPv6 stack. We've had to deal with this when working on NAT64 (Tayga).
Cheers
Maurice
@Maurice Thank you for reply.
QuoteDo we actually need a gateway address? We have the 'Dynamic gateway policy' option in the interface settings specifically for tunnel interfaces without a gateway address. I'm using this with other tunnel interfaces (WireGuard, Tayga). @vnxme, could you try whether this works with GRE and GIF?
Not sure I understand what you actually mean by 'dynamic gateway policy'. I have 'dynamic gateway switching' enabled. I have 'dynamic' set as a gateway address in my default gateway configured on the GRE interface. I have no addresses/gateways configured separately in interface settings, i.e. IPv4/IPv6 configuration type set to none. Could you please elaborate what you suggest I should try?
QuoteBut what if local and remote address are not actually in the same /64? Say you have 2001:db8:1::1 local, 2001:db8:2::1 remote and the default /64 prefix length configured in the UI. This currently works because the /64 is ignored and a host route for the remote address is created instead. If we now configure the interface with 2001:db8:1::1/64, the remote address might become unreachable.
In my view, this assumption is conceptually wrong and I doubt there is any ISP using such a config. But you are right in thinking about ordinary users who might have misconfigured their tunnels because of the fact OpnSense ignored the prefix length. Normally, you would have to set /128 prefix if you need to connect endpoints from completely different network segments.
A solution here could be to change the prefix length from whatever was configured to /128 on migration. Then let users decide which option they prefer: an IPv4-like point-to-point mode with a /128 prefix length or a broadcast mode with a custom prefix length.
Quote... do this. Though just leaving the remote address field empty to enable the new mode would seem more intuitive to me.
Can't fully agree with you. As I have already explained above, the remote address is the value used as a gateway address substitute for "dynamic" on the single gateway configuration page. If the remote address is empty, the 'dynamic' gateway won't work out of the box. Nevertheless, I agree it is possible to set the gateway manually.
QuoteIt doesn't ignore it. When you check the interface config with ifconfig IFACE, you can see that the netmask is applied correctly (inet LOCAL --> REMOTE netmask MASK).
You are right stating ifconfig outputs this value as a part of interface status summary. What I'm talking about is the kernel routing table, and for this particular purpose the netmask is ignored.
QuoteThat's a known FreeBSD limitation. The destination address of tunnel interfaces can't be omitted in the IPv4 stack, but it can in the IPv6 stack. We've had to deal with this when working on NAT64 (Tayga).
Good to know you confirm my understanding, thanks. I tried to dig into the FreeBSD sources of ifconfig, but I'm not a C-guy to fully understand the code. I tracked the issue to the fact that getaddr/setaddr functions for inet and inet6 address families inside ifconfig are significantly different (imagine how naïve I was expecting them to be more or less similar ;D).
@franco @Maurice Then what shall we do with that? An additional drop-down box to select between point-to-point and broadcast? Or force migrate everybody to /128 and let them change it later? Or make an empty remote address set the broadcast mode?
Quote from: vnxme on August 04, 2022, 07:21:03 PM
Could you please elaborate what you suggest I should try?
'Dynamic gateway policy' is a checkbox in the interface settings (the last one, saying "This interface does not require an intermediate system to act as a gateway"). It allows creating routes where the next-hop is the interface itself instead of a gateway address. This might or might not work for GRE and GIF. You could test it by manually configuring the interface and routes; something like this:
ifconfig gre3 create
ifconfig gre3 tunnel 192.0.2.1 198.51.100.1
ifconfig gre3 inet6 2001:db8:1::1/126
route -6 add 2001:db8:2::/64 -interface gre3
Quote from: vnxme on August 04, 2022, 07:21:03 PM
As I have already explained above, the remote address is the value used as a gateway address substitute for "dynamic" on the single gateway configuration page. If the remote address is empty, the 'dynamic' gateway won't work out of the box.
See above about the dynamic gateway policy. And if this shouldn't work here, then ...
Quote from: vnxme on August 04, 2022, 07:21:03 PM
Nevertheless, I agree it is possible to set the gateway manually.
... this.
Quote from: vnxme on August 04, 2022, 07:21:03 PM
What I'm talking about is the kernel routing table, and for this particular purpose the netmask is ignored.
That's correct and it's standard FreeBSD behaviour for IPv4 tunnels. You'll have to add that route manually.
Quote from: vnxme on August 04, 2022, 07:21:03 PM
Then what shall we do with that? [...] Or make an empty remote address set the broadcast mode?
That would be my choice. Shouldn't brake any existing setups, doesn't require migration code an seems intuitive.
Cheers
Maurice
Thanks for the great input from both of you.
Quote from: vnxme on August 04, 2022, 07:21:03 PM
@franco @Maurice Then what shall we do with that? An additional drop-down box to select between point-to-point and broadcast? Or force migrate everybody to /128 and let them change it later? Or make an empty remote address set the broadcast mode?
Aha! So according to the theory voiced here we test if the remote net fits the local address. If it does apply prefixlen as given, if not apply 128 as before? Should both be working according to you and no new knobs and migrations necessary. I like how we could solve this locally in the GRE configuration code.
Cheers,
Franco
@Maurice Thank you for explaining the 'dynamic gateway policy'.
I'm not sure that I can properly test this edge case as it barely suits my setup. I have an IPv6-in-IPv4 GRE tunnel between two OpnSense virtual machines, one of which is a VPS. The VPS has a native dual-stack connection and a routed /56 prefix. On the other side a default IPv6 route is set via the VPS and I use active gateway monitoring to switch to another upstream gateway. If I created a default route where the next-hop is the interface itself, I wouldn't be able to monitor my connection status and switch gateways automatically.
QuoteThat would be my choice. Shouldn't brake any existing setups, doesn't require migration code an seems intuitive.
Making an empty remote address set the broadcast mode likely requires changes to both interfaces.inc and GIF/GRE configuration pages. Besides, we should take into account that the 'dynamic' gateway configuration might be broken and it even shouldn't be allowed on such an interface.
QuoteAha! So according to the theory voiced here we test if the remote net fits the local address. If it does apply prefixlen as given, if not apply 128 as before? Should both be working according to you and no new knobs and migrations necessary. I like how we could solve this locally in the GRE configuration code.
@franco, like your suggestion! Let me know if you expect me to prepare a pull request or test your patch.
Patch is here: https://github.com/opnsense/core/commit/d69ff4a16a89
I've also added https://github.com/opnsense/core/commit/889420b652b and it needs both...
# opnsense-patch 889420b652b d69ff4a16a89
So if this works I'm not so sure about this anymore although it seems more true to the original implementation:
https://github.com/opnsense/core/commit/111a2560fb7
Cheers,
Franco
@franco Great, will test shortly. Shouldn't we also make a change to the GIF part similar to https://github.com/opnsense/core/commit/d69ff4a16a89 ?
Yes, we will keep GIF/GRE implementation as close as possible. But GRE test first. :)
Cheers,
Franco
Tested the patch on 2 machines, switched between /128 and /126 and vice versa. Works fine in both modes, no errors in log. BGP also works fine when /126 is configured.
Yay, awesome :)
Neat solution! Before releasing it, I'd suggest thoroughly testing whether switching from this ...
ifconfig gif0 inet6 2001:db8::2 2001:db8::1 prefixlen 128
... to this ...
ifconfig gif0 inet6 2001:db8::2 prefixlen 64
... has any weird side effects. Probably not, but you never know. This is the typical tunnelbroker.net setup.
Cheers
Maurice
Yep, will do.
What Ad mentioned regarding this is for VTI we seem to just calculate a matching subnet size between local and remote which also works fine to get rid of the hardcoded prefixlen 128 .. that means omitting the selected remote net on the GUI altogether, but I'm not sure if we should be going this far here.
Cheers,
Franco
That could result in a very large subnet (e. g. if local and remote address are not in the same /64). I don't think we should ditch the /128 mode entirely.
Cheers
Maurice
Yup, we could consider /64 to be the breaking point of that method, but since we have a net size selection the current approach seems more reasonable and we could also add a validation to the page so we can get rid of implicit prefixlen 128 in the future (it continues to work but use is discouraged unless specifically configured).
Cheers,
Franco