OPNsense Forum

Archive => 22.1 Legacy Series => Topic started by: gromit on March 18, 2022, 07:34:59 PM

Title: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: gromit on March 18, 2022, 07:34:59 PM
I just upgraded to 22.1.3 from 22.1.2_1 and upon rebooting into the new version noticed my Hurricane Electric Tunnel Broker was not working properly.  The gateway was no longer listed in the "Gateways" widget on the dashboard.  Looking further in System: Gateways: Single I saw that the HE gateway was greyed out with a status of "Pending" and a priority of "defunct (upstream)".

I deleted my HE tunnel and recreated it, but it still did not work.

The change log for this release leads off with this statement:

QuoteThis update includes groundwork for interface handling improvements making the boot more flexible in complex interface assignment scenarios involving GIF, GRE and bridge devices.

Could my HE Tunnel Broker problems be related to this?

I had a recollection that my gif0 previously had the HE tunnel IPv6 endpoints defined on it and this was no longer the case after upgrading.  I manually applied these to gif0 (as per HE's FreeBSD instructions) and manually defined an IPv6 default route and that has at least got IPv6 going again for me.

Is there a change in the way HE Tunnel Broker needs to be set up under 22.1.3??

Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: franco on March 18, 2022, 08:08:05 PM
In this particular case we did not alter config.xml contents for the GIF/GRE/bridge changes so I'm not sure about what happened there. It could still be related for other reasons. Do you have a system log of the boot sequence you can share? It should give a clearer picture of what could be wrong.


Cheers,
Franco
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: gromit on March 18, 2022, 09:13:11 PM
I couldn't see any complaints in the boot sequence, and I do remember that IPv6 was initially working after boot when I tested via the serial console.  It was only when looking at the WebGUI I noticed the HE tunnel missing from the "Gateways" widget and so I went looking about to troubleshoot.

It could be that in the "troubleshooting" I broke things further.  :(

Anyway, I just recreated the IPv6 gateway again and this time it appears to be working.  :)

I'll give it another reboot to see if it stays put this time...
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: gromit on March 18, 2022, 09:38:59 PM
Alas, after the reboot there are still problems with the gateway.  :(

This time around, I noticed this in the boot sequence:

Reconfiguring IPv4 on em0
>>> Invoking start script 'freebsd'
gif0: link state changed to DOWN
in6_purgeaddr: err=65, destination address delete failed
gif0: link state changed to UP
in6_purgeaddr: err=65, destination address delete failed


(em0 is my WAN link.)

The resultant tunnel has only IPv4 endpoints defined:

gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
description: HE
options=80000<LINKSTATE>
tunnel inet 73.99.XXX.XXX --> 216.YYY.YYY.2
inet6 fe80::2eb:caff:fec0:5c4%gif0 prefixlen 64 scopeid 0x18
groups: gif
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


When I manually add the IPv6 endpoints to gif0 then the gateway works again.
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: oli on March 19, 2022, 11:54:55 AM
Just want to confirm that I have the same problem. so @gromit you are not alone.
Though I also did not find a solution yet
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: RedVortex on March 20, 2022, 11:22:29 PM
Same here. I'm unable to bring HE tunnel up unless I manually put the IPv6 addresses manually and also manually add a route afterwards.

I recreated the interface and gif and it's not working. I only see the gif0 with ipv4 only configs.

My tunnel was working in 22.1.2. Seems like HE (gif) is broken in 22.1.3
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: oli on March 21, 2022, 07:13:15 AM
How do you "manually put the IPv6 addresses" ?

When I set "IPv6 Configuration Type" to "Static IPv6" on the Tunnel-Interface then the UI complains "Cannot assign an IP configuration type to a tunnel interface"
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: kurthw on March 21, 2022, 07:14:03 AM
Can cofirm that the HE Tunnel (Gateway) is broken  :(. Strangely enough, I still have an external IP v6 address....
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: franco on March 21, 2022, 07:52:07 AM
Any system log from the boot process? This is strange but doesn't seem to indicate what could be wrong.

in6_purgeaddr: err=65, destination address delete failed

And where is the "missing IPv6" configured?


Cheers,
Franco
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: franco on March 21, 2022, 09:38:22 AM
Ok, I could reproduce this and can report that the problem was previously fixed in the code by running device configuration twice to bring it to a working state. How about we don't do that and use the following instead?

https://github.com/opnsense/core/commit/61500f6790

# opnsense-patch 61500f6790


Cheers,
Franco
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: RedVortex on March 21, 2022, 02:11:07 PM
Thanks for the patch. I was able to bring the HE tunnel up and the gateway was there too after patching and save/apply on the interface and gif (i think had to save gif as well for it to work).

I'll do a reboot test later on tonight and make sure other interfaces are not negatively affected by this patch and that HE automatically gets back up after reboot too.
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: franco on March 21, 2022, 02:48:39 PM
> save/apply on the interface

Should be good then after reboot too as this executes the same code. :)


Cheers,
Franco
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: kurthw on March 21, 2022, 05:17:31 PM
Thank you franco for the quick solution. I can confirm that the patch works.

Under System > Gateways > Single the disabled entry still had to be re-enabled and ipv6-test.com reports a connection via HE.net.

In the meantime I have rebooted 2 times and the setting remains active again as with 22.1.2. However WITHOUT setting "Upstream Gateway" as it can be found in the manual https://docs.opnsense.org/manual/how-tos/ipv6_tunnelbroker.html#step-2-configure-the-gif-tunnel-as-a-new-interface.
(https://rofd.net/Pics/Overview.png)

(https://rofd.net/Pics/WANV6_TunnelV6.png)
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: franco on March 21, 2022, 07:49:27 PM
kurthw Thanks for confirming. :)

"Upstream gateway" is required when you want to use it as a default route. If there is only one (IPv6) gateway it will automatically promote itself to default anyway (see "active" annotation in the list).


Cheers,
Franco
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: oli on March 21, 2022, 11:10:42 PM
Thx, patch works for me, too.  8)
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: Napsterbater on March 22, 2022, 01:52:38 AM
Even with "opnsense-patch 61500f6790" my HE GIF Tunnel is completely broken.

Seems an address is not getting assigned per the INTERFACES -> OVERVIEW screen.
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: franco on March 22, 2022, 08:41:16 AM
More details please.
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: Napsterbater on March 22, 2022, 12:36:04 PM
Tell me what details do you need. What specific log outputs do you need. Because nothing I found in the web UI logs. Tell me anything useful.

All I know is the IP addresses are not being assigned to the interface. That's the gateway can't be created. This all started on 22.1.3
I have applied the patch, I've deleted the GIF interface I've deleted interface assignment. I've tried to recreate the gateway. Nothing works. System reboots don't work either.
I have a DHCP WAN with a Static Virtual IP that the GIF is attached to.
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: franco on March 22, 2022, 01:32:36 PM
It's really the same issue you describe and you said you applied the patch, but maybe you applied it twice which removes it again or a local modification (opnsense-patch run) conflicted with it. Can you post the actual opnsense-patch output?


Cheers,
Franco
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: Napsterbater on March 22, 2022, 10:11:07 PM
Napsterbater@car1:~ % sudo opnsense-patch
Napsterbater@car1:~ %


Napsterbater@car1:~ % sudo opnsense-patch 61500f6790
Found local copy of 61500f6790, skipping fetch.
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 61500f6790969ba1c426c27322ba8879dd96bd5a Mon Sep 17 00:00:00 2001
|From: Franco Fichtner <franco@opnsense.org>
|Date: Mon, 21 Mar 2022 09:24:09 +0100
|Subject: [PATCH] interfaces: suspicious code is suspicious
|
|I'm not sure why interface_bring_down() is needed when both
|IPv4 and IPV6 are empty.  It sort of means "handle this the
|hard way when doing tunnel configurations" althoug the code
|disagrees about the historic comment and the code that was
|introduced... "set to none" vs. "ipaddr <> none" and later
|"empty(ipaddr)" to match the comment.  In the grand scheme of
|things this does not matter at all...
|
|So in 22.1.3 we removed the inline configuration of GIF and GRE
|which causes interface configuration to strip the addresses
|added by device configuration instead of refixing it on the
|fly (executing code twice all the time).  The code flow was
|always correct but in practice tripping over itself so now try
|a more sensible approach by stripping addresses when we have
|assignments going on individually for IPv4 and IPv6.
|
|PR: https://forum.opnsense.org/index.php?topic=27553.0
|---
| src/etc/inc/interfaces.inc | 14 ++++++--------
| 1 file changed, 6 insertions(+), 8 deletions(-)
|
|diff --git a/src/etc/inc/interfaces.inc b/src/etc/inc/interfaces.inc
|index 4ab49d680a..79876b1ba3 100644
|--- a/src/etc/inc/interfaces.inc
|+++ b/src/etc/inc/interfaces.inc
--------------------------
Patching file etc/inc/interfaces.inc using Plan A...
Reversed (or previously applied) patch detected!  Assuming -R.Hunk #1 succeeded at 2200 with fuzz 1 (offset 33 lines).
done
All patches have been applied successfully.  Have a nice day.


"Reversed (or previously applied) patch detected!  Assuming -R.Hunk #1 succeeded at 2200 with fuzz 1 (offset 33 lines).
done"
So it was installed?
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: Napsterbater on March 22, 2022, 10:18:29 PM
After re applying the patch due to removal when posting previous output. And rebooting again. It does seem to be back to working.

No idea why it didn't the first time.
Title: Re: HE Tunnel Broker broke after 22.1.3 upgrade (gateway stopped working)
Post by: gromit on April 08, 2022, 10:07:38 PM
Quote from: franco on March 21, 2022, 07:52:07 AM
Any system log from the boot process? This is strange but doesn't seem to indicate what could be wrong.

in6_purgeaddr: err=65, destination address delete failed

And where is the "missing IPv6" configured?


Cheers,
Franco

Sorry for having spaced on replying to this thread.  In my case, I believe what might have been causing the issue was that, after setting up the HE Tunnel, I also (for reasons too lengthy to go into here) defined a static IPv6 address on my WAN link (em0).  I didn't have any IPv6 gateway defined for this, but I think this was messing up configuration of gif0 during boot.

My "fix" was to set "IPv6 Configuration Type" to "None" on the WAN interface.  Boot-up configuration of the HE Tunnel worked after that.