Multiple problems after upgrade to 22.1.10

[clarknova]

I have two pairs of firewalls running OPNsense 22.1.8_1. Today I tried to update the back up of both pairs using option 12 via ssh. Part way through the upgrade the shell stopped responding and I eventually power cycled one of them. Now I see errors on the console during the boot sequence and the console never arrives at the OPNsense shell. The host responds to ping requests, but sshd and nginx do not respond. Console images attached.

Is this a known issue? What is the recommended recovery procedure?

[franco]

Partial package update. Since we are switching from Python 3.8 to 3.9 it looks like at least Jinja2 package for Python 3.9 was never installed. Power-cycling from mid-update is often problematic.

I'd still encourage you to try the update again and if it should hang let me know where that would happen..


Thanks,
Franco

[clarknova]

I can boot in single-user mode and get a shell. Is there a way to recover from this? Or do I have to do a clean install with a config restore? I'm off site with idrac access, so I'd prefer to recover it if there's a way.

[clarknova]

I got my backup firewall back online doing the following:

• Boot from latest installer iso (22.1.2)
• Import configuration from disk
• Update from console

I then placed the primary firewall in persistent CARP maintenance mode and used option 12 on the console to update. It went exactly as it did on the backup firewall, to my best recollection. I've attached a couple of screenshots. It stalls indefinitely at "Configuring system logging...". Then the "sonewconn...Listen queue overflow" errors start coming.

[clarknova]

Despite the console being generally unresponsive, I can still reload the web dashboard. The only things that appear out of the ordinary are that some applets aren't loading and the configd service is stopped. Screenshot attached.

[franco]

Does SSH work too or just single user mode?

I'd recommend trying to bootstrap everything not knowing what partial state the system is in:

# opnsense-bootstrap


Cheers,
Franco

[clarknova]

ssh is still working, but bootstrapping appears to have failed.

Code: [Select]

# opnsense-bootstrap
This utility will attempt to turn this installation into the latest
OPNsense 22.1 release.  All packages will be deleted, the base
system and kernel will be replaced, and if all went well the system
will automatically reboot.

Proceed with this action? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
process with pid 77073 still holds the lock
process with pid 77073 still holds the lock
process with pid 77073 still holds the lock
process with pid 77073 still holds the lock
process with pid 77073 still holds the lock
process with pid 77073 still holds the lock
pkg-static: Cannot get an exclusive lock on a database, it is locked by another process

# ps aux|grep 77073
root    77073    0.0  0.6 185028 156320 v0  S+   12:29        0:34.41 pkg-static upgrade -y

# kill -15 77073

At this point I was able to run bootstrap successfully. The firewall booted to 21.1.10 and all appears to be working well. Is there a down side to using opnsense-bootstrap for upgrades? Seems to be more reliable than console option 12.

edit: I see I have to manually reinstall packages. I guess I'll see if the package config survived.
edit2: Package config appears to be intact. I don't know why the upgrade has failed on 3 hosts, but the outcome is good.

[franco]

The "lock" on the package database is a bit suspicious. If the configurations are more or less identical it could happen in all of them.

Did you mean "plugins" for "packages"? We can consider adding plugin reinstall to opnsense-bootstrap, at least for the first tier as additional repositories such as Sunny Valley can be reinstalled but will only be available for its plugins after they have been installed too.


Cheers,
Franco

[clarknova]

Yes, I meant I had to reinstall missing plugins, but the plugin configuration all appears to be as it was after installing them.

If a plugin reinstall option could be added to opnsense-bootstrap that could be helpful in situations such as this.

[franco]

Fair enough, made a reminder ticket https://github.com/opnsense/update/issues/82


Cheers,
Franco