Upgrade from 23.7.6 to 23.7.7 failing

[Rogero]

Tried to upgrade when the release came out and am receiving the following error:

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 23.7.6 at Tue Oct 31 11:26:52 PDT 2023
Fetching changelog information, please wait... done
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 851 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (10 candidates): .......... done
Processing candidates (10 candidates): .......... done
Checking integrity...Assertion failed: (strcmp(uid, p->uid) != 0), function pkg_conflicts_check_local_path, file pkg_jobs_conflicts.c, line 315.
Child process pid=79470 terminated abnormally: Abort trap
***DONE***

The health audit is 4115 lines long and too large to attach. Lot's of missing files and checksum mismatches.

Additionally, cannot get ssh to start. Likely related but not sure.

Any help would be appreciated. Let me know what additional info you would like.

[cookiemonster]

symptoms of hard disk failures

[Rogero]

Tried to load smart tools on the box and it needs 23.7.7 which I don't have.

[franco]

Could just be a broken database.

I'd recommend a full bootstrap, but if you can make a snapshot and/or back up your configuration.

# opnsense-bootstrap

Your data will not be harmed, but everything will be reinstalled cleanly. Sometimes this operation is very heavy on the disk and if the disk is already showing signs of decay it could push it over the edge... hence the precaution.


Cheers,
Franco

[Rogero]

Tried running the opnsense-bootstrap command and it fails with a message:
"pkg: sqllite error while executing DELETE FROM packages WHERE id +3222 in file pkgdb.c:2296: database disk image is malformed"

Any solution to this other that a fresh install?

Thanks,
Roger

[franco]

Hi Roger,

Ok, the databases live here:

# ls /var/db/pkg/*.sqlite

Maybe move to a backup location and bootstrap will be more successful?


Cheers,
Franco

[Rogero]

Thanks but too late. Bootstrap nuked my credentials before it failed so I couldn't log in any more. It was still passing traffic at least so I could download a new iso file and rebuild the device. Restored xml backup and it is mostly back to it's original setup.

[franco]

Bummer that the database had such a crucial role and that part failed because it got damaged.

Not having seen the health audit I'm unsure if you have UFS or ZFS and if more file integrity errors were on the disk. If it's a physical disk with UFS power outages could have disrupted the file system or if it's ZFS the disk or main memory may not be very healthy. Just as an extra point of caution.


Cheers,
Franco

[Rogero]

Hi Franco,
I had a power outage a week or so ago which might have been the cause of the db corruption. Since rebuilding I have installed the smart utilities and the SSD is clean. The worst part is that opnsense-bootstrap process deletes the user accounts before it gets to the db so once it failed, it was DOA. My only remaining issue is restoring the config, did not restore my firewall rules and I cannot remember what I had set up for zerotier.

Thanks for your assistance,
Roger