configd service not starting after upgrade to 21.1.9

[mgeoffriau]

Hey folks, looking for some help getting pointed in the right direction.

Background - I've got a number of OPNsense instances running in various applications, but I'll be up front and admit I have one particular bare-metal install that I have failed to keep updated as the resulting interruption is annoying and if the upgrade goes bad it's a pain to recover. But, after an (unrelated) issue on Friday made some network maintenance necessary, I decided to knock out the upgrades on this instance to bring it back to current version.

It was on 21.1.1 when I started (if I remember correctly) and the first upgrade (to 21.1.7, I believe) went fine. The next upgrade to 21.1.9 is where issues began. When the upgrade completed and the hardware (HP T620 Plus thin client) rebooted, I couldn't access it remotely, and no routing resumed so I'm assuming it was in some bad state. I took it back to my bench, connected a monitor, and it booted up just fine. Tried another power cycle and it was fine again. So I took it back to the network closet, reconnected it, and booted it back up.

Couple minutes later, it's back online so I log in to check things out and prep the next upgrade. That's when I notice that many parts of the GUI load extremely slowly, and in the Services pane of the dashboard it shows that configd is not running. Clicking to start it is not successful.

My first thought was to SSH in and see what was up (and maybe follow some tips from this thread: https://forum.opnsense.org/index.php?topic=14323.0) but when I attempted to, I just got "Access denied" responses for my creds. SSH was working just prior to the upgrade, so I don't think the SSH failure is a configuration problem. As far as I can tell, it's still routing just fine and most other services are running and operational.

Looking for any recommendations on next steps. Thanks in advance.

[newsense]

If having console access see if you can get in, and if internet is working try to complete the update.

Else reinstall using the currently supported 24.7 version and import the config file

[meyergru]

Well, it is kind of obvious what you intended, but it is also obvious that in addition to the hassle of having to go through all of the 7 major upgrades you tried to avoid in the first place, you now have to do all of them years after most people faced specific problems with migrating from one version to the next. I imagine very few people could remember them that long.

Searching the forum probably will not help in this situation  (it would have if you kept up with updates). It might help if you read all of the release notes - including those of the minor updates, which often fix migration problems.

As an alternative, I would try to import a saved configuration into a fresh 24.7.3 installation, potentially only importing one section after another, in the hope that not too much has changed in the config syntax.

If you want to keep downtimes low, as opposed to skipping years of updates altogether, I would suggest to either use a business license or update to major versions a few weeks later after the rough edges have been ironed out.

BTW: IDK if 21.1 already had ZFS, but if you use it for the new installation, from 24.7.3 on, you can also use snapshots from the GUI to roll back in case anything goes wrong with an upgrade.

[mgeoffriau]

Appreciate both replies.

I can definitely bring it back to my bench to access the console directly, but I wouldn't have an easy way of giving it internet access at the same time, unfortunately. So the only real purpose would be diagnostic/manual repair.

I very much considered just reimaging to 24.7 and restoring a config (even before encountering an issue, since it would save so much time), but my main concern was the compatibility of a config backup file over that large of a gap. I searched a bit but couldn't find anything definitive regarding forward-compatibility of backups.

I suppose it wouldn't hurt anything to test. I could reimage on 24.7, try the backup restore, and if things go sideways, just reimage back to 21.1 or 21.7 where it seemed like things were fine and restore my existing backups from those versions.

[mgeoffriau]

So this afternoon I snagged one of my backup devices, loaded OPNsense 24.7, restored my backup config...and it seemed like everything just worked? No apparent issues so I swapped that out with the production router, upgraded to 24.7.3, added my plugins back in (mostly*), tested a few things to confirm, and then started reimaging the prod device to 24.7 and restored my backup. Swapped them back, upgraded prod to 24.7.3 and added my plugins, and it looks like everything is solid. And I've got my backup ready again on the current version should I need to swap back out for any reason.

* I had forgotten that os-dyndns was deprecated and was sad to see it was finally removed. Will have to do some reading to see what a good dynamic DNS client replacement would be.