IPsec tunnel won't work after latest update

[Nicolassimond]

Hello,

We have updated the OPNSense at our office to the latest release yesterday (auto-update in fact) and one of our two IPSEC tunnel won't let data goes through.

Both IPSEC tunnel goes to two up-to-date Watchguard's firewall at two different locations.

We have the same configuration on both site-to-site tunnels:
IPv4 IKEv2
Phase 1:  main   AES (128 bits) + SHA256 + DH Group 20   Mutual PSK
Phase 2: ESP IPv4 tunnel   AES256 + SHA256+ DH Group 20

The first tunnel work as expected:

Code Select Expand

State
INSTALLED
Routed

Stats
Time : 459
Bytes in : 70596
Bytes out : 59200


But the second won't accept traffic anymore:

Code Select Expand

State
INSTALLED
Routed

Stats
Time : 459
Bytes in : 0
Bytes out : 19228

We restarted both firewall, deleted the IPSEC config completly then made it from scratch, rolled back to the previous version on the opnsense firewall but nothing works.
We don't see the traffic comming to the watchguard or being rejected at all, it seems that nothing is going outside the opnsense firewall.

Any idea?

[franco]

What's a "latest" update? What is the last known good version? There haven't been much disruptive ipsec changes lately except for a strongswan update in 22.1.1 which is more of a black box update from our point of view, but it's unlikely something like this went undiscovered for weeks.


Cheers,
Franco

[Nicolassimond]

Hello,

Last is: OPNsense 22.1.7_1-amd64
FreeBSD 13.0-STABLE
OpenSSL 1.1.1o 3 May 2022

The last working was the update before, we did the update yesterday (the firewall have a cron to auto-update, so I don't monitor the version number closely to be honest).

[franco]

I make no bets on what sort of auto-update cron frequency and mirror you have so you will have to do better.

If we talk about 22.1.6 -> 22.1.7_1 or 22.1.7 -> 22.1.7_1 I have no clue what you are looking for in terms of regressions. Might just be intermittent or the other end of the tunnel.


Thanks,
Franco

[Nicolassimond]

The cron update the firewall every 1 hour, that why I said it was the release just before (this one is on our test labs to check if we are going to resell opnsense firewalls)

We already check the other side, I don't think it comes from here, as we have the same watchguard on another location, with same update and same tunnels settings (except for the networks and public ip of course) that works flawlessly, this where it gets strange.

[franco]

I'm sure there's an issue somewhere but suspecting an auto-update somebody put there and neglecting release notes information to double-check doesn't help get to the bottom of this. There's a number of other approaches that could be used here I'm sure.


Cheers,
Franco

[Nicolassimond]

The auto-update is suspected because it was working just before, you will agree that is has something to do with it.

Anyway, what could we check? To get rid of this?

Best regards,

[franco]

You could use opnsense-revert to test your theory:

# opnsense-revert -r 22.1.6 opnsense


Cheers,
Franco

[Nicolassimond]

I will test this afternoon, as I don't have access to the firewall right now.
But this won't reverse kernel and all the packages?

[franco]

Just opnsense core package. Kernel revert is a different command and also hasn't been changed since 22.1.5 anyway.


Cheers,
Franco

[Nicolassimond]

So, it won't change anything, I have already done this as written in my first post.

Where can I get the traffics logs and ipsec logs from command line? Web interface shows everything empty for ipsec (and as I said, I don't have any incoming traffic inside the tunnel on the watchguard so I can't see any error on it).

[franco]

> So, it won't change anything, I have already done this as written in my first post.

No you didn't mention "opnsense-revert" specifically. And if it doesn't then as I said you're not looking for the right thing here suspecting updates.

My last post here.


Cheers,
Franco

[Nicolassimond]

I asked what else I could look 5th post ago and didn't get any feedback on this.
I didn't know there was another way beside "opnsense-revert" to actually revert to previous version, so I didn't write further indeed.

If I get through the professional service that we paid for this firewall (and others) will I get the same answers?

[franco]

If you have support hours we'll likely schedule a anydesk session to look at the problem directly. Doesn't matter if it's update-related, or configuration issue or other end troubleshooting.


Cheers,
Franco

[Nicolassimond]

It's solved, it was unplugging it for the lunch and restart it, that's all...

But we have another bug leading to 100%cpu usage by Python when we check the logs of ipsec.


You should really get into a ticketing system for MSP and professionals, apart from the "anydesk" remote session like all the other vendors with full log analysis and everything.

The problem, even if we pay for support hours by default when buying your product, it that legally, I can't let you connect to our systems and customer's one without a NDA with each of your staff that may connect, which is a bit complicated you will agree.