[SOLVED] 21.1.6 possibly broke ipsec

[Jiffy]

** EDIT **
The workaround below is no longer needed.
A new FRR package has fixed it.
** EDIT **

** EDIT **
Routing was actually broken.
Reverting to an older version of FRR resolved this.

# opnsense-revert -r 21.1.5 frr7

Franco has mentioned that the FRR package in 21.1.6 will be replaced to prevent others from having this issue.
** EDIT **

Hi,

I'm Running OPNsense in a Proxmox VM.
During lunch today, I shut it down, took a snapshot, powered it up and upgraded to 21.1.6.
At that point everything worked except for the ipsec tunnel, the tunnel was up, OSPF neighbors were there and the correct routes were installed too, it just wasn't working.
I couldn't connect to anything nor could I ping anything.
I even went as far as installing an "any any" rule in both directions on my ipsec interface, no joy.
No other changes were made, I had to bring the tunnel back up so I restored the snapshot.
After the restore everything was fine again.
I can upgrade it again, but is there anything else I can check?
Is there something I can do/test/report that will help you help me?

Thank you,
Jiffy

[marcquark]

Just to rule that one out: Did you reboot after applying the update?

[franco]

Important info would be your previous good version and whether you use OpenSSL or LibreSSL.

There are only two small fixes in IPsec configuration code that shouldn't cause breakage and so far we haven't seen or heard of it. The relevant patches are:

https://github.com/opnsense/core/commit/4758dd89d
https://github.com/opnsense/core/commit/ca3d33f7c6

Both are easily reverted using opnsense-patch 4758dd89d or opnsense-patch ca3d33f7c6. Rerunning the patch brings it back.

Of course as marcquark says make sure to test this cleanly by reboot to rule out any side effects like firewall rules not reloading.


Cheers,
Franco

[Jiffy]

Hi,

Yes, I did reboot, twice. I don't blame you for asking.
I'll try to revert those patches and report back.

The previous good version is 21.1.5 and I'm running it with OpenSSL.

Thank you,
Jiffy

[Jiffy]

Hi,

I've upgraded and reverted those patches, still no luck.
I'll snapshot this and restore the 21.1.5 snapshot for now.
Standing by for further instructions.

Thank you,
Jiffy

[franco]

Hi Jiffy,

Let's try a broader approach on the broken state:

# opnsense-revert -r 21.1.5 opnsense

If it won't work with the old core package we know it's not the configuration code.


Cheers,
Franco

[Jiffy]

Hi, Franco,

Copy and pasted the output of the revert into the attached revert-log.txt.
Still not working.

I'm not using OSPF6, RIP or BGP, so I'm guessing these are safe to ignore:
*** OPNsense\Quagga\OSPF6 Migration failed, check log for details
*** OPNsense\Quagga\RIP Migration failed, check log for details
*** OPNsense\Quagga\BGP Migration failed, check log for details

Thank you,
Jiffy

[Cerberus]

Hi,

just some feedback from my update experience yesterday. Two OPNsense systems in a carp cluster with:

5x IKEv1
2x IKEv2
35 currently active Mobile IKEv2 clients.

Most of them aes-gcm and some aes-cbc, all of them with sha256. No issues to report, all tunnel and client connections working well after after the update. But i had one little issue today, we got an power outtage and after reboot two tunnels stopped working with authentication failure, only thing that helped was open phase 1 and press apply/save again, authentication errors stopped. Just restarting the connection wasnt enough.

[Jiffy]

Hi, Cerberus,

I just tried that, didn't work.
My tunnel is up and OSPF neighbors are there, it just won't pass traffic.

Franco,

I started a tcpdump on a node on the inside (LAN) of the firewall and had someone ping it from a node on the other side of the tunnel.
Traffic is coming in but not leaving.
From inside, traceroute stops at the firewall. There aren't any denies in the firewall logs.

Thank you,
Jiffy

[franco]

Only possibility left is StrongSwan to be honest...

# opnsense-revert -r 21.1.5 strongswan


Cheers,
Franco

[marcquark]

I was in a similar situation and it came down to an overscoped NAT rule. Weren't there changes to NAT code aswell?

[Jiffy]

Reverted strongswan, still no luck.

[franco]

I was in a similar situation and it came down to an overscoped NAT rule. Weren't there changes to NAT code aswell?

If 21.1.5 core with 21.1.6 throws the same error what could possibly be changed at all?

So uhh maybe kernel?

# opnsense-update -kr 21.1.5

We are running out of options assuming the health audit does not report issues we are overlooking... that's what it's there for. :)


Cheers,
Franco

[mircsicz]

I'm kinda in the same boat, three client machine's I can't get IPsec back up and running after upgrading last night!

But for me I also have struggle with No-IP Update's, had to set the password's, afterwards it (No-IP Update) [seemed] worked again...

EDIT: see my seperate Topic regarding the DynDNS issue which I had to open as it only update'd the Cached-Ip but not the registry at No-IP.com

[Jiffy]

Reverted kernel, still no good.