[SOLVED] pkg broke during 17.1 upgrade

[ejprice]

Hi folks,

I've just completed an upgrade from 16.7 to 17.1 and it seems pkg is broken. Here is the output I'm getting:

Shared object "libssl.so.7" not found, required by "pkg"

So I tried to force it to reinstall with pkg-static:
pkg-static install -f pkg

pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
Updating OPNsense repository catalogue...
pkg-static: http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/17.1/latest/meta.txz: No address record
repository OPNsense has no meta file, using default settings
pkg-static: http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/17.1/latest/packagesite.txz: No address record
Unable to update repository OPNsense
All repositories are up-to-date.
pkg-static: Repository OPNsense cannot be opened. 'pkg update' required

pkg-1.9.3_2 is locked and may not be modified

pkg-1.9.3_2 is locked and may not be modified

So far I've tried changing mirrors, doing an update with pkg-static but I'm still stuck. Any ideas?

Thanks in advance!

[franco]

Unlock pkg first...

# pkg-static unlock pkg
# pkg-static upgrade


Cheers,
Franco

[ejprice]

Thank you! Hadn't thought of that. Still a FreeBSD noob 🙄

Unfortunately, the upgrade still didn't fix the problem. Here is the output:

root@ppt-fw:~ # pkg-static upgrade
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
Updating OPNsense repository catalogue...
pkg-static: http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/17.1/latest/meta.txz: No address record
repository OPNsense has no meta file, using default settings
pkg-static: http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/17.1/latest/packagesite.txz: No address record
Unable to update repository OPNsense
All repositories are up-to-date.
pkg-static: Repository OPNsense cannot be opened. 'pkg update' required
Checking for upgrades (0 candidates): 100%
Processing candidates (0 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.
root@ppt-fw:~ # pkg
Shared object "libssl.so.7" not found, required by "pkg"

Any other thoughts? It is not a production system so I can simply backup the config and do a fresh install but that seems like cheating to me.

Thanks!

[franco]

Geez, pkg can be stubborn. Maybe better follow its suggestion.

# pkg-static install -f pkg

If that for some reason doesn't work, it can also be re-bootstrapped, I think it goes like this:

# pkg-static remove pkg
# pkg bootstrap

[ejprice]

Thank you for the quick reply!

Geez, pkg can be stubborn. Maybe better follow its suggestion.

# pkg-static install -f pkg

Indeed I did that first on the initial error and many times since. It, unfortunately gives the same error (see below).

If that for some reason doesn't work, it can also be re-bootstrapped, I think it goes like this:

# pkg-static remove pkg
# pkg bootstrap

pkg bootstrap doesn't want to work either.

root@ppt-fw:~ # pkg bootstrap
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.opnsense.org/FreeBSD:11:amd64/17.1/latest, please wait...
pkg: Error fetching http://pkg.opnsense.org/FreeBSD:11:amd64/17.1/latest/Latest/pkg.txz: No address record
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.

This has been the same error I've been getting when I tried pkg-static upgrade and pkg-static install -f pkg

I have changed mirrors so that is not the issue.

[franco]

Oh sorry, I completely overlooked this. DNS resolution isn't working. Try to force e.g. 8.8.8.8 from the GUI as a DNS server to fix this temporarily.

[ejprice]

No problem at all! I really appreciate the help.

I think it's really dead 😢. Name resolution does not work at all from the shell. I confirmed the Google DNS server is in /etc/resolv.conf. I confirmed the system can traceroute to the Google NS. So just for fun and because I had nothing to lose, I edited the /usr/local/etc/pkg/repos/origin.conf file and hardcoded the IP address of the OPNSense repository.

I received a connection timeout upon trying pkg bootstrap even though we're bypassing DNS. Traceroute completes successfully. Telneting the pkg repo on port 80 does not complete. I've disabled Squid and paused the port 80 redirect. I've also tried https which should bypass squid and the pf rules since SSL bump wasn't configured. I'd say this is indicative of a much larger problem.

Thoughts?

[franco]

Oh dear. Without working network the chances of firmware recovery are slim. At this point you should consider reinstalling the system (installer on 17.1 with import configuration + guided installation to retain your settings) or if its a nano system recover your config from the media, maybe also from a install memstick.


Cheers,
Franco

[ejprice]

That was my thought exactly. Originally it was a learning opportunity and a challenge. Now is is just a pain 😡

Thank you for all the help! I've learned some good stuff about OPNSense and FreeBSD.

[franco]

I think for the future there is something that needs to be addressed... major upgrades should unlock packages before upgrading. Or at least we should double check which effect pkg-lock has on pkg-upgrade -f:

https://github.com/opnsense/update/issues/10

Over the years, I trashed a good share of installs during testing, or wiping the occasional production box that really just needed to be upgraded.


Cheers,
Franco

[franco]

The fix will be part of 17.1.1. Packages are unlocked prior to the major upgrade of all packages as they should.

This may have consequences for manually installed packages, but we need to make sure we provide a correct platform where it can actually be installed again.


Cheers,
Franco

[ejprice]

The fix will be part of 17.1.1. Packages are unlocked prior to the major upgrade of all packages as they should.

This may have consequences for manually installed packages, but we need to make sure we provide a correct platform where it can actually be installed again.


Cheers,
Franco

I would humbly say that manually installed packages are outside the scope of the OPNSense team's responsibility anyway. If you're savvy enough to install unsupported packages, you get to maintain them and keep both pieces if it breaks ;-) Just my 2 cents.

Thanks again for the help!

[franco]

Should still avoid bricking during a transition. We always try to retain/enforce a good known state. In any case: won't happen again!

[HappyUser]

Sorry to be the bearer of bad news but I got this when I tried to upgrade to 18.1.

Updating OPNsense repository catalogue...
pkg-static: Repository OPNsense load error: access repo file(/var/db/pkg/repo-OPNsense.sqlite) failed: No such file or directory
pkg-static: http://mirrors.dmcnet.net/opnsense/FreeBSD:11:amd64/17.7/latest/meta.txz: No address record
repository OPNsense has no meta file, using default settings
pkg-static: http://mirrors.dmcnet.net/opnsense/FreeBSD:11:amd64/17.7/latest/packagesite.txz: No address record
Unable to update repository OPNsense
Error updating repositories!

[franco]

"No address record" means for some reason or another the mirror can't be reached.


Cheers,
Franco