OPNsense Forum
Archive => 21.1 Legacy Series => Topic started by: Jiffy on May 27, 2021, 09:08:38 pm
-
** EDIT **
The workaround below is no longer needed.
A new FRR package has fixed it.
** EDIT **
** EDIT **
Routing was actually broken.
Reverting to an older version of FRR resolved this.
# opnsense-revert -r 21.1.5 frr7
Franco has mentioned that the FRR package in 21.1.6 will be replaced to prevent others from having this issue.
** EDIT **
Hi,
I'm Running OPNsense in a Proxmox VM.
During lunch today, I shut it down, took a snapshot, powered it up and upgraded to 21.1.6.
At that point everything worked except for the ipsec tunnel, the tunnel was up, OSPF neighbors were there and the correct routes were installed too, it just wasn't working.
I couldn't connect to anything nor could I ping anything.
I even went as far as installing an "any any" rule in both directions on my ipsec interface, no joy.
No other changes were made, I had to bring the tunnel back up so I restored the snapshot.
After the restore everything was fine again.
I can upgrade it again, but is there anything else I can check?
Is there something I can do/test/report that will help you help me?
Thank you,
Jiffy
-
Just to rule that one out: Did you reboot after applying the update?
-
Important info would be your previous good version and whether you use OpenSSL or LibreSSL.
There are only two small fixes in IPsec configuration code that shouldn't cause breakage and so far we haven't seen or heard of it. The relevant patches are:
https://github.com/opnsense/core/commit/4758dd89d
https://github.com/opnsense/core/commit/ca3d33f7c6
Both are easily reverted using opnsense-patch 4758dd89d or opnsense-patch ca3d33f7c6. Rerunning the patch brings it back.
Of course as marcquark says make sure to test this cleanly by reboot to rule out any side effects like firewall rules not reloading.
Cheers,
Franco
-
Hi,
Yes, I did reboot, twice. I don't blame you for asking.
I'll try to revert those patches and report back.
The previous good version is 21.1.5 and I'm running it with OpenSSL.
Thank you,
Jiffy
-
Hi,
I've upgraded and reverted those patches, still no luck.
I'll snapshot this and restore the 21.1.5 snapshot for now.
Standing by for further instructions.
Thank you,
Jiffy
-
Hi Jiffy,
Let's try a broader approach on the broken state:
# opnsense-revert -r 21.1.5 opnsense
If it won't work with the old core package we know it's not the configuration code.
Cheers,
Franco
-
Hi, Franco,
Copy and pasted the output of the revert into the attached revert-log.txt.
Still not working.
I'm not using OSPF6, RIP or BGP, so I'm guessing these are safe to ignore:
*** OPNsense\Quagga\OSPF6 Migration failed, check log for details
*** OPNsense\Quagga\RIP Migration failed, check log for details
*** OPNsense\Quagga\BGP Migration failed, check log for details
Thank you,
Jiffy
-
Hi,
just some feedback from my update experience yesterday. Two OPNsense systems in a carp cluster with:
5x IKEv1
2x IKEv2
35 currently active Mobile IKEv2 clients.
Most of them aes-gcm and some aes-cbc, all of them with sha256. No issues to report, all tunnel and client connections working well after after the update. But i had one little issue today, we got an power outtage and after reboot two tunnels stopped working with authentication failure, only thing that helped was open phase 1 and press apply/save again, authentication errors stopped. Just restarting the connection wasnt enough.
-
Hi, Cerberus,
I just tried that, didn't work.
My tunnel is up and OSPF neighbors are there, it just won't pass traffic.
Franco,
I started a tcpdump on a node on the inside (LAN) of the firewall and had someone ping it from a node on the other side of the tunnel.
Traffic is coming in but not leaving.
From inside, traceroute stops at the firewall. There aren't any denies in the firewall logs.
Thank you,
Jiffy
-
Only possibility left is StrongSwan to be honest...
# opnsense-revert -r 21.1.5 strongswan
Cheers,
Franco
-
I was in a similar situation and it came down to an overscoped NAT rule. Weren't there changes to NAT code aswell?
-
Reverted strongswan, still no luck.
-
I was in a similar situation and it came down to an overscoped NAT rule. Weren't there changes to NAT code aswell?
If 21.1.5 core with 21.1.6 throws the same error what could possibly be changed at all?
So uhh maybe kernel?
# opnsense-update -kr 21.1.5
We are running out of options assuming the health audit does not report issues we are overlooking... that's what it's there for. :)
Cheers,
Franco
-
I'm kinda in the same boat, three client machine's I can't get IPsec back up and running after upgrading last night!
But for me I also have struggle with No-IP Update's, had to set the password's, afterwards it (No-IP Update) [seemed] worked again...
EDIT: see my seperate Topic (https://forum.opnsense.org/index.php?topic=23301.0) regarding the DynDNS issue which I had to open as it only update'd the Cached-Ip but not the registry at No-IP.com
-
Reverted kernel, still no good.
-
I've reverted to the 21.1.5 snapshot, rebooted, upgraded to 21.1.6, rebooted and no errors during the health audit.
***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 21.1.6 (amd64/OpenSSL) at Fri May 28 12:13:38 EDT 2021
>>> Check installed kernel version
Version 21.1.6 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 21.1.6 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" has 67 dependencies to check.
Checking packages: ..................................................................... done
***DONE***
-
Same here without the revert:
***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 21.1.6 (amd64/OpenSSL) at Fri May 28 18:24:26 CEST 2021
>>> Check installed kernel version
Version 21.1.6 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 21.1.6 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" has 67 dependencies to check.
Checking packages: ..................................................................... done
***DONE***
-
Frankly I'm out of ideas what this could be? Not saying there isn't a problem but it is hiding really well.
Cheers,
Franco
-
I've added static routes to System->Routes->Configuration
Now I can get to things on the other side of the tunnel.
These were normally accessible via OSPF
They (remote ip addresses) are also showing up under Routing->Diagnostics->General and Routing->Diagnostics->OSPF
but when I disable the statics, they are no longer reachable.
Jiffy
-
Last guess of the day:
# opnsense-revert -r 21.1.5 frr7
Someone proposed a patch fixing an issue but apparently neither the reporter nor the patch lived up to the expectations and the patch was forgotten ;(
Cheers,
Franco
-
# opnsense-revert -r 21.1.5 frr7
That fixed it.
Thank you,
Jiffy
-
My issue was chained to the os-dyndns failure...
-
Ok, we will replace the FRR package in 21.1.6 to avoid more people running into this issue.
Cheers,
Franco
-
Great, thank you for all your help, Franco.
Jiffy
-
We also ran into this issue on one of our opnsense instances.
Can the opnsense-revert -r 21.1.5 frr7 be run after a complete update to 21.1.6 or do we first have to downgrade the "base-system" to 21.1.5?
-
Yes, revert is a targeted revert of a particular package for such situations so you would revert only the FRR package and leave the rest at 21.1.6 since it doesn't pose any issues.
In any case check for updates now. It should already advertise the fixed package as a separate update (along with security fix for expat library).
Cheers,
Franco
-
Hi,
I took the latest update and can confirm it is still working.
Thank you again,
Jiffy
-
Thanks for the quick report!
-
same here, did the update and everything started working again. Thanks for the quick solution.
-
Just saw this thread on the back of the FRR patch that apparently broke things for this user. I am on the other side of the fence, where this patch actually fixes things for me.
Can the original poster please describe how the tunnels runnins OSPF were configured? Was this IPSec + GRE or VTI? And specifically, what netmasks were configured on both ends of the tunnel?
-
Just saw this thread on the back of the FRR patch that apparently broke things for this user. I am on the other side of the fence, where this patch actually fixes things for me.
Can the original poster please describe how the tunnels runnins OSPF were configured? Was this IPSec + GRE or VTI? And specifically, what netmasks were configured on both ends of the tunnel?
If you updated only recently, you already have a fixed version
-
It was more a question in terms of researching the error by the patch author.
Cheers,
Franco