GRE/GIF IPv6 tunnel netmask ignored on interface configuration

[vnxme]

Hi,

Issue
I have a GRE tunnel configured between IPv4 hosts with IPv6 addresses inside of it. I noticed that the netmask parameter set on the GIF or GRE configuration page was completely ignored here. As far as I remember, this behavior occurred even prior to 22.7_4.

If this is a mistake I can prepare a PR so that you could distribute a fix in the next release (e.g. 22.7.1). Otherwise, if this was made intentionally, does anybody know why the 128 prefix length was hardcoded to the GIF/GRE tunnel config?

Implication
I'm interested in BGP working over IPv6 tunnel addresses. While the tunnel endpoints are pingable and static routes work correctly, the hosts won't establish a BGP session on GUA IPv6 or won't copy routes to kernel tables on LL IPv6. After a short debugging I noticed that the BGP daemon considered the other endpoint as an invalid nexthop if a tunnel was configured with a 128 prefix length.

Possible fix
I tested changing this code

Code Select Expand

        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " " . escapeshellarg($gre['tunnel-remote-addr']) . " prefixlen 128");

to the following

Code Select Expand


        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " prefixlen " . $gre['tunnel-remote-net']);
and for me it has absolutely fixed the BGP over IPv6-in-IPv4 GRE issue.

In my view, the final solution could even be conditional to the prefix length:

• 128 - ifconfig IFACE inet6 LOCAL REMOTE prefixlen 128
• other - ifconfig IFACE inet6 LOCAL prefixlen NET

[franco]

Are you using a net size greater than /64?

In general this looks like it should be fixed indeed since we take the parameter for the net size anyway we can enforce it and the standard of /64 vs. /128 shouldn't break anything.


Cheers,
Franco

[vnxme]

@franco Thank you for a fast reply. I'm using /126 for IPv6 inside the tunnel which provides for 4 addresses just like /30 in IPv4.

[franco]

Thanks for explaining. Did you maybe mean to propose:

mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " " . escapeshellarg($gre['tunnel-remote-addr']) . " prefixlen " . $gre['tunnel-remote-net']);

?

As far as can see IPv4 does the same.


Cheers,
Franco

[franco]

Flashback to 2014 sure seems funny now:

https://github.com/pfsense/pfsense/commit/cdeaf91e14e6e2

;)


Cheers,
Franco

[franco]

It's not as straight-forward as I hoped:

# /sbin/ifconfig gre2 inet6 23::333 1::4 prefixlen 64
ifconfig: ioctl (SIOCAIFADDR): Invalid argument

This works but is distinctively different from the previous...

# /sbin/ifconfig gre2 inet6 23::333 prefixlen 64

Adding a destination address to the ifconfig command requires prefixlen 128

So maybe as a compromise we could make a second mode by inputting the "::" or "0.0.0.0" remote address which omits this destination from the ifconfig command as you suggested?


Cheers,
Franco

[vnxme]

@franco Thank you for digging into the issue.

First of all, no, we don't need any remote address when prefixlen is less than 128. Otherwise, as you mentioned, ifconfig returns an error and won't assign the address. So the code I posted is correct:

Code Select Expand


        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " prefixlen " . $gre['tunnel-remote-net']);


Secondly, it's worth discussing why/when we need to set a remote address. In my view, the situation is quite straightforward:

• If we choose a point-to-point mode, it means /128 should be used. When using /128 the kernel routing table needs a separate entry to be configured for a remote side, otherwise there will be no route to the gateway error. A user should note that BGP won't work in this mode (at least without special changes to the config).
• If we choose a broadcast mode, it means /126 and less should be used (not sure about /127). This case provides for at least 4 addresses: 0 - network, 1 - the first side, 2 - the second side, 3 - broadcast. Then the kernel routing table already knows that the gateway is accessible on the interface, thus no need to configure a remote address route separately. In this mode BGP works flawlessly.

As far as the 2014 discussion is concerned, it is absolutely correct that 128 prefix length is enforced by the kernel in the point-to-point mode, i.e. when we explicitly set a remote address. But the ones who developed that part of the code seem to have missed the broadcast mode I described above.

Thirdly, when it comes to your suggestion, I believe we can't input "::" or "0.0.0.0" to the remote field as this is the value used as a gateway address substitute for "dynamic" on the single gateway configuration page. At least it won't work if one (like me) wants to use the tunnel remote address as a default gateway for IPv6.

[vnxme]

So my suggestion would be:

Code Select Expand


if ($gre['tunnel-remote-net']) == 128) {
        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " " . escapeshellarg($gre['tunnel-remote-addr']) . " prefixlen 128");
} else {
        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " prefixlen " . $gre['tunnel-remote-net']);
}


This code can be further simplified to:

Code Select Expand


        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . ($gre['tunnel-remote-net'] == 128 ? " " . escapeshellarg($gre['tunnel-remote-addr']) : "") . " prefixlen " . $gre['tunnel-remote-net']);


[franco]

The problem is most people have 64 set for their IPv6 remote net since that is the default proposed by the GUI.

I've changed the default here but we do have a historic mess...

https://github.com/opnsense/core/commit/111a2560fb7b

BTW, in IPv6 network and broadcast address are not part of the address range so /127 are two valid addresses to use.

You are right about the gateway address issue. I think we don't have much choice but to add a new checkbox for this.


Cheers,
Franco

[vnxme]

@franco Even if most people have 64 set in the web interface, their routing tables do not have a /64 route since the prefix length value has (always/for a long time) been ignored. Changing the code from what it was (128 hardcoded) to what I propose will ensure they have a correct /64 route, which by default includes the remote side of the tunnel. So in my view, nothing will break, and no need to change the defaults.

Just to illustrate what I mean:

• Up to now. Say you have 2a01::1 local, 2a01::2 remote and /64 prefix length set in the web interface. Then you have the following in your kernel routing table.

Code Select Expand


ipv6 2a01::2 link#10 UH NaN 1476 gre0 TEST_GRE_IFACE
ipv6 2a01::1 link#10 UHS NaN 16384 lo0 Loopback


• After the change I propose is implemented. Same config in the web interface. Then you have the following in your kernel routing table.

Code Select Expand


ipv6 2a01::/64 link#10 U NaN 1476 gre0 TEST_GRE_IFACE
ipv6 2a01::1 link#10 UHS NaN 16384 lo0 Loopback


Please, pay attention to absence of /64 in the first case and presence of the H flag (not sure what it means).

[franco]

I'm not convinced yet but if that works that would be good. Maybe Maurice has a setup to help test this.


Cheers,
Franco

[vnxme]

@franco I will tell you more, this issue is also relevant to IPv4. Say you have 172.17.34.1 local, 172.17.34.2 remote and 24 netmask in the web interface. Then you have the following in your routing rable.

Code Select Expand

ipv4 172.17.34.1 link#13 UHS NaN 16384 lo0 Loopback
ipv4 172.17.34.2 link#13 UH NaN 1476 gre1 TEST_GRE_IFACE

"ifconfig IFACE inet LOCAL REMOTE netmask MASK" actually ignores the netmask whatever is set. Unfortunately, I haven't found a way to run ifconfig without remote address for inet family. Seems like a FreeBSD kernel limitation. But this is another story.

Update: Furthermore, if I try to assign an IPv4 alias to IPv4-in-IPv4 GRE tunnel, I also get an error:

Code Select Expand

/firewall_virtual_ip.php: The command '/sbin/ifconfig 'gre1' inet '172.17.35.1'/'24' alias ' returned exit code '1', the output was 'ifconfig: ioctl (SIOCAIFADDR): Destination address required'

[Maurice]

I've never used GRE and haven't used 6in4 GIF for quite a few years. That being said, what you have figured out makes sense to me (mostly ;)).

BTW, in IPv6 network and broadcast address are not part of the address range so /127 are two valid addresses to use.

Yep.

You are right about the gateway address issue. I think we don't have much choice but to add a new checkbox for this.

Do we actually need a gateway address? We have the 'Dynamic gateway policy' option in the interface settings specifically for tunnel interfaces without a gateway address. I'm using this with other tunnel interfaces (WireGuard, Tayga). @vnxme, could you try whether this works with GRE and GIF?

Say you have 2a01::1 local, 2a01::2 remote and /64 prefix length set in the web interface.

But what if local and remote address are not actually in the same /64? Say you have 2001:db8:1::1 local, 2001:db8:2::1 remote and the default /64 prefix length configured in the UI. This currently works because the /64 is ignored and a host route for the remote address is created instead. If we now configure the interface with 2001:db8:1::1/64, the remote address might become unreachable.

It might be better to write some migration code which changes the prefix length of existing setups to 128. Or, if 'Dynamic gateway policy' works, ...

So maybe as a compromise we could make a second mode by inputting the "::" or "0.0.0.0" remote address which omits this destination from the ifconfig command as you suggested?

... do this. Though just leaving the remote address field empty to enable the new mode would seem more intuitive to me.

"ifconfig IFACE inet LOCAL REMOTE netmask MASK" actually ignores the netmask whatever is set.

It doesn't ignore it. When you check the interface config with ifconfig IFACE, you can see that the netmask is applied correctly (inet LOCAL --> REMOTE netmask MASK).

Unfortunately, I haven't found a way to run ifconfig without remote address for inet family.

That's a known FreeBSD limitation. The destination address of tunnel interfaces can't be omitted in the IPv4 stack, but it can in the IPv6 stack. We've had to deal with this when working on NAT64 (Tayga).

Cheers
Maurice

[vnxme]

@Maurice Thank you for reply.

Do we actually need a gateway address? We have the 'Dynamic gateway policy' option in the interface settings specifically for tunnel interfaces without a gateway address. I'm using this with other tunnel interfaces (WireGuard, Tayga). @vnxme, could you try whether this works with GRE and GIF?

Not sure I understand what you actually mean by 'dynamic gateway policy'. I have 'dynamic gateway switching' enabled. I have 'dynamic' set as a gateway address in my default gateway configured on the GRE interface. I have no addresses/gateways configured separately in interface settings, i.e. IPv4/IPv6 configuration type set to none. Could you please elaborate what you suggest I should try?

But what if local and remote address are not actually in the same /64? Say you have 2001:db8:1::1 local, 2001:db8:2::1 remote and the default /64 prefix length configured in the UI. This currently works because the /64 is ignored and a host route for the remote address is created instead. If we now configure the interface with 2001:db8:1::1/64, the remote address might become unreachable.

In my view, this assumption is conceptually wrong and I doubt there is any ISP using such a config. But you are right in thinking about ordinary users who might have misconfigured their tunnels because of the fact OpnSense ignored the prefix length. Normally, you would have to set /128 prefix if you need to connect endpoints from completely different network segments.
A solution here could be to change the prefix length from whatever was configured to /128 on migration. Then let users decide which option they prefer: an IPv4-like point-to-point mode with a /128 prefix length or a broadcast mode with a custom prefix length.

... do this. Though just leaving the remote address field empty to enable the new mode would seem more intuitive to me.

Can't fully agree with you. As I have already explained above, the remote address is the value used as a gateway address substitute for "dynamic" on the single gateway configuration page. If the remote address is empty, the 'dynamic' gateway won't work out of the box. Nevertheless, I agree it is possible to set the gateway manually.

It doesn't ignore it. When you check the interface config with ifconfig IFACE, you can see that the netmask is applied correctly (inet LOCAL --> REMOTE netmask MASK).

You are right stating ifconfig outputs this value as a part of interface status summary. What I'm talking about is the kernel routing table, and for this particular purpose the netmask is ignored.

That's a known FreeBSD limitation. The destination address of tunnel interfaces can't be omitted in the IPv4 stack, but it can in the IPv6 stack. We've had to deal with this when working on NAT64 (Tayga).

Good to know you confirm my understanding, thanks. I tried to dig into the FreeBSD sources of ifconfig, but I'm not a C-guy to fully understand the code. I tracked the issue to the fact that getaddr/setaddr functions for inet and inet6 address families inside ifconfig are significantly different (imagine how naïve I was expecting them to be more or less similar ;D).

@franco @Maurice Then what shall we do with that? An additional drop-down box to select between point-to-point and broadcast? Or force migrate everybody to /128 and let them change it later? Or make an empty remote address set the broadcast mode?

[Maurice]

Could you please elaborate what you suggest I should try?

'Dynamic gateway policy' is a checkbox in the interface settings (the last one, saying "This interface does not require an intermediate system to act as a gateway"). It allows creating routes where the next-hop is the interface itself instead of a gateway address. This might or might not work for GRE and GIF. You could test it by manually configuring the interface and routes; something like this:

Code Select Expand


ifconfig gre3 create
ifconfig gre3 tunnel 192.0.2.1 198.51.100.1
ifconfig gre3 inet6 2001:db8:1::1/126
route -6 add 2001:db8:2::/64 -interface gre3


As I have already explained above, the remote address is the value used as a gateway address substitute for "dynamic" on the single gateway configuration page. If the remote address is empty, the 'dynamic' gateway won't work out of the box.

See above about the dynamic gateway policy. And if this shouldn't work here, then ...

Nevertheless, I agree it is possible to set the gateway manually.

... this.

What I'm talking about is the kernel routing table, and for this particular purpose the netmask is ignored.

That's correct and it's standard FreeBSD behaviour for IPv4 tunnels. You'll have to add that route manually.

Then what shall we do with that? [...] Or make an empty remote address set the broadcast mode?

That would be my choice. Shouldn't brake any existing setups, doesn't require migration code an seems intuitive.

Cheers
Maurice