Messages

Hi,

Say you have a VXLAN interface configured on the Interfaces -> Other Types -> VXLAN page and assigned to some user-friendly interface. Now you would like to change its VNI or addresses. Edit and Save buttons work correctly, the changed settings are properly kept. But when you press the Apply button, the interface isn't effectively reconfigured - ifconfig reports the VNI and/or the addresses it had before the change. Besides, the following error is logged:

Code Select Expand

/usr/local/sbin/pluginctl: The command '/sbin/ifconfig 'vxlan1' vxlanid '1234' vxlanlocal '1.2.3.4' vxlanremote '5.6.7.8' ' returned exit code '1', the output was 'ifconfig: VXLAN_CMD_SET_VNI: Device busy'

Thus, it seems the only effective way to reconfigure VXLAN interfaces in 22.7_4 is a reboot.

P.S. I understand you must be tired of my bug reports submitted in the last couple of days. Just have a little time to make OpnSense better. Hope for understanding ;)

Tested the patch on 2 machines, switched between /128 and /126 and vice versa. Works fine in both modes, no errors in log. BGP also works fine when /126 is configured.

@franco Great, will test shortly. Shouldn't we also make a change to the GIF part similar to https://github.com/opnsense/core/commit/d69ff4a16a89 ?

@Maurice Thank you for explaining the 'dynamic gateway policy'.

I'm not sure that I can properly test this edge case as it barely suits my setup. I have an IPv6-in-IPv4 GRE tunnel between two OpnSense virtual machines, one of which is a VPS. The VPS has a native dual-stack connection and a routed /56 prefix. On the other side a default IPv6 route is set via the VPS and I use active gateway monitoring to switch to another upstream gateway. If I created a default route where the next-hop is the interface itself, I wouldn't be able to monitor my connection status and switch gateways automatically.

That would be my choice. Shouldn't brake any existing setups, doesn't require migration code an seems intuitive.

Making an empty remote address set the broadcast mode likely requires changes to both interfaces.inc and GIF/GRE configuration pages. Besides, we should take into account that the 'dynamic' gateway configuration might be broken and it even shouldn't be allowed on such an interface.

Aha! So according to the theory voiced here we test if the remote net fits the local address. If it does apply prefixlen as given, if not apply 128 as before? Should both be working according to you and no new knobs and migrations necessary. I like how we could solve this locally in the GRE configuration code.

@franco, like your suggestion! Let me know if you expect me to prepare a pull request or test your patch.

@Maurice Thank you for reply.

Do we actually need a gateway address? We have the 'Dynamic gateway policy' option in the interface settings specifically for tunnel interfaces without a gateway address. I'm using this with other tunnel interfaces (WireGuard, Tayga). @vnxme, could you try whether this works with GRE and GIF?

Not sure I understand what you actually mean by 'dynamic gateway policy'. I have 'dynamic gateway switching' enabled. I have 'dynamic' set as a gateway address in my default gateway configured on the GRE interface. I have no addresses/gateways configured separately in interface settings, i.e. IPv4/IPv6 configuration type set to none. Could you please elaborate what you suggest I should try?

But what if local and remote address are not actually in the same /64? Say you have 2001:db8:1::1 local, 2001:db8:2::1 remote and the default /64 prefix length configured in the UI. This currently works because the /64 is ignored and a host route for the remote address is created instead. If we now configure the interface with 2001:db8:1::1/64, the remote address might become unreachable.

In my view, this assumption is conceptually wrong and I doubt there is any ISP using such a config. But you are right in thinking about ordinary users who might have misconfigured their tunnels because of the fact OpnSense ignored the prefix length. Normally, you would have to set /128 prefix if you need to connect endpoints from completely different network segments.
A solution here could be to change the prefix length from whatever was configured to /128 on migration. Then let users decide which option they prefer: an IPv4-like point-to-point mode with a /128 prefix length or a broadcast mode with a custom prefix length.

... do this. Though just leaving the remote address field empty to enable the new mode would seem more intuitive to me.

Can't fully agree with you. As I have already explained above, the remote address is the value used as a gateway address substitute for "dynamic" on the single gateway configuration page. If the remote address is empty, the 'dynamic' gateway won't work out of the box. Nevertheless, I agree it is possible to set the gateway manually.

It doesn't ignore it. When you check the interface config with ifconfig IFACE, you can see that the netmask is applied correctly (inet LOCAL --> REMOTE netmask MASK).

You are right stating ifconfig outputs this value as a part of interface status summary. What I'm talking about is the kernel routing table, and for this particular purpose the netmask is ignored.

That's a known FreeBSD limitation. The destination address of tunnel interfaces can't be omitted in the IPv4 stack, but it can in the IPv6 stack. We've had to deal with this when working on NAT64 (Tayga).

Good to know you confirm my understanding, thanks. I tried to dig into the FreeBSD sources of ifconfig, but I'm not a C-guy to fully understand the code. I tracked the issue to the fact that getaddr/setaddr functions for inet and inet6 address families inside ifconfig are significantly different (imagine how naïve I was expecting them to be more or less similar ;D).

@franco @Maurice Then what shall we do with that? An additional drop-down box to select between point-to-point and broadcast? Or force migrate everybody to /128 and let them change it later? Or make an empty remote address set the broadcast mode?

[Update:]

@franco I will tell you more, this issue is also relevant to IPv4. Say you have 172.17.34.1 local, 172.17.34.2 remote and 24 netmask in the web interface. Then you have the following in your routing rable.

Code Select Expand

ipv4 172.17.34.1 link#13 UHS NaN 16384 lo0 Loopback
ipv4 172.17.34.2 link#13 UH NaN 1476 gre1 TEST_GRE_IFACE

"ifconfig IFACE inet LOCAL REMOTE netmask MASK" actually ignores the netmask whatever is set. Unfortunately, I haven't found a way to run ifconfig without remote address for inet family. Seems like a FreeBSD kernel limitation. But this is another story.

Update: Furthermore, if I try to assign an IPv4 alias to IPv4-in-IPv4 GRE tunnel, I also get an error:

Code Select Expand

/firewall_virtual_ip.php: The command '/sbin/ifconfig 'gre1' inet '172.17.35.1'/'24' alias ' returned exit code '1', the output was 'ifconfig: ioctl (SIOCAIFADDR): Destination address required'

@franco Even if most people have 64 set in the web interface, their routing tables do not have a /64 route since the prefix length value has (always/for a long time) been ignored. Changing the code from what it was (128 hardcoded) to what I propose will ensure they have a correct /64 route, which by default includes the remote side of the tunnel. So in my view, nothing will break, and no need to change the defaults.

Just to illustrate what I mean:

• Up to now. Say you have 2a01::1 local, 2a01::2 remote and /64 prefix length set in the web interface. Then you have the following in your kernel routing table.

Code Select Expand


ipv6 2a01::2 link#10 UH NaN 1476 gre0 TEST_GRE_IFACE
ipv6 2a01::1 link#10 UHS NaN 16384 lo0 Loopback


• After the change I propose is implemented. Same config in the web interface. Then you have the following in your kernel routing table.

Code Select Expand


ipv6 2a01::/64 link#10 U NaN 1476 gre0 TEST_GRE_IFACE
ipv6 2a01::1 link#10 UHS NaN 16384 lo0 Loopback


Please, pay attention to absence of /64 in the first case and presence of the H flag (not sure what it means).

So my suggestion would be:

Code Select Expand


if ($gre['tunnel-remote-net']) == 128) {
        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " " . escapeshellarg($gre['tunnel-remote-addr']) . " prefixlen 128");
} else {
        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " prefixlen " . $gre['tunnel-remote-net']);
}


This code can be further simplified to:

Code Select Expand


        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . ($gre['tunnel-remote-net'] == 128 ? " " . escapeshellarg($gre['tunnel-remote-addr']) : "") . " prefixlen " . $gre['tunnel-remote-net']);


@franco Thank you for digging into the issue.

First of all, no, we don't need any remote address when prefixlen is less than 128. Otherwise, as you mentioned, ifconfig returns an error and won't assign the address. So the code I posted is correct:

Code Select Expand


        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " prefixlen " . $gre['tunnel-remote-net']);


Secondly, it's worth discussing why/when we need to set a remote address. In my view, the situation is quite straightforward:

• If we choose a point-to-point mode, it means /128 should be used. When using /128 the kernel routing table needs a separate entry to be configured for a remote side, otherwise there will be no route to the gateway error. A user should note that BGP won't work in this mode (at least without special changes to the config).
• If we choose a broadcast mode, it means /126 and less should be used (not sure about /127). This case provides for at least 4 addresses: 0 - network, 1 - the first side, 2 - the second side, 3 - broadcast. Then the kernel routing table already knows that the gateway is accessible on the interface, thus no need to configure a remote address route separately. In this mode BGP works flawlessly.

As far as the 2014 discussion is concerned, it is absolutely correct that 128 prefix length is enforced by the kernel in the point-to-point mode, i.e. when we explicitly set a remote address. But the ones who developed that part of the code seem to have missed the broadcast mode I described above.

Thirdly, when it comes to your suggestion, I believe we can't input "::" or "0.0.0.0" to the remote field as this is the value used as a gateway address substitute for "dynamic" on the single gateway configuration page. At least it won't work if one (like me) wants to use the tunnel remote address as a default gateway for IPv6.

@franco Thank you for a fast reply. I'm using /126 for IPv6 inside the tunnel which provides for 4 addresses just like /30 in IPv4.

[Issue]

Hi,

Issue
I have a GRE tunnel configured between IPv4 hosts with IPv6 addresses inside of it. I noticed that the netmask parameter set on the GIF or GRE configuration page was completely ignored here. As far as I remember, this behavior occurred even prior to 22.7_4.

If this is a mistake I can prepare a PR so that you could distribute a fix in the next release (e.g. 22.7.1). Otherwise, if this was made intentionally, does anybody know why the 128 prefix length was hardcoded to the GIF/GRE tunnel config?

Implication
I'm interested in BGP working over IPv6 tunnel addresses. While the tunnel endpoints are pingable and static routes work correctly, the hosts won't establish a BGP session on GUA IPv6 or won't copy routes to kernel tables on LL IPv6. After a short debugging I noticed that the BGP daemon considered the other endpoint as an invalid nexthop if a tunnel was configured with a 128 prefix length.

Possible fix
I tested changing this code

Code Select Expand

        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " " . escapeshellarg($gre['tunnel-remote-addr']) . " prefixlen 128");

to the following

Code Select Expand


        mwexec("/sbin/ifconfig {$gre['greif']} inet6 " . escapeshellarg($gre['tunnel-local-addr']) . " prefixlen " . $gre['tunnel-remote-net']);
and for me it has absolutely fixed the BGP over IPv6-in-IPv4 GRE issue.

In my view, the final solution could even be conditional to the prefix length:

• 128 - ifconfig IFACE inet6 LOCAL REMOTE prefixlen 128
• other - ifconfig IFACE inet6 LOCAL prefixlen NET

@franco Good to know. Thank you for your help!

@franco Tested the solution you proposed in this post. After opnsense-patch 7aaa6a263b1 bb4743a7322 the issue is resolved on 22.7_4.

Will you release this patch officially?

@franco Tested 22.1.10_4 with the same config last night:

• No GRE tunnel flapping reported in console
• No entries added to the general log every 2.5 min on WAN DHCP renew
• No BGP errors

Thus, the issue seems to be specific to 22.7.

@franco Thank you for attention to this topic. I had some issues with 22.1 early this year and decided to skip that version. If it was of any value to you, I could have 22.1 installed and tested on the same config as well.