[SOLVED] Update to 22.1.5 didn't finish.

[tofflock]

Hi

I upgraded my local FW to 22.1.5 a couple of days ago and that went fine.  Then I started the same upgrade on my remote FW, and that didn't complete.  It appears to have failed getting base-22.1.5-amd64.txz, and then didn't get the new kernel.  It also hasn't rebooted, yet.  It won't be long before there's a brownout which'll cause a reboot. The end of the update log (full copy attached) is here:

Code: [Select]

.
.
/var/cache/pkg/py38-charset-normalizer-2.0.12.txz
The cleanup will free 59 MiB
Deleting files: .......... done
All done
Nothing to do.
Starting web GUI...done.
Generating RRD graphs...done.
Fetching base-22.1.5-amd64.txz: ..............................pgrep:
Cannot open pidfile `/tmp/opnsense-fetch.pid.8WIINU': No such file or directory
[fetch: transfer timed out] failed, no signature found
***DONE***
How should I go about finishing off this update safely (& remotely) please?

TIA

PeterF

[franco]

Try to poke the update again. If it isn't a persistent problem it should work on a subsequent try.. make sure to account for the reboot that's going to happen once the base and kernel are being picked up.


Cheers,
Franco

[tofflock]

Hi Franco

Try to poke the update again. If it isn't a persistent problem it should work on a subsequent try.. make sure to account for the reboot that's going to happen once the base and kernel are being picked up.

I tried the update again and as you anticipated (I think) it said there were updates for the base and kernel packages.  These appeared to get installed ok but with no status ouput to the log window.  The system rebooted ok.

After it had rebooted, I did a set of audit tests:
CONNECTIVITY Audit was ok for IPV4 (IPV6 is not available from my ISP)
Then I checked for updates, and it reported all packages were up to date
However, the HEALTH Audit produced some very strange results (file attached) with the final package check reporting

Code: [Select]

.
.
Checking packages: .
zip-3.0_1 has no upstream equivalent
***DONE***for every single package (67).
I also noticed that the list of plugins was truncated to only those that I'd installed.
I checked for updates again, the plugin list was fully populated, and a subsequent HEALTH audit then looked normal.
I'm a bit puzzled by the "failed" HEALTH audit, but everything appears to be working fine.

Thanks for your help

PeterF

[franco]

Hi Peter,

That's sort of weird. Once you check for updates the heath check likely works...but after connectivity audit it doesn't?

If that's the case I really wasn't expecting FreeBSD package manager would forget its good data due to not reaching its mirror via IPv6?


Cheers,
Franco

[tofflock]

Hi Franco

If that's the case I really wasn't expecting FreeBSD package manager would forget its good data due to not reaching its mirror via IPv6?

I've done a bit of digging to try & understand what is going on.
1  The CONNECTIVITY audit is implemented by this script:

Code: [Select]

cat /usr/local/opnsense/scripts/firmware/connection.sh

#!/bin/sh

# Copyright (C) 2021 Franco Fichtner <franco@opnsense.org>
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
# INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
# AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
# AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
# OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

LOCKFILE="/tmp/pkg_upgrade.progress"
TEE="/usr/bin/tee -a"

: > ${LOCKFILE}

URL=$(opnsense-update -M)
POPT="-c4 -s1500"

HOST=${URL#*://}
HOST=${HOST%%/*}
IPV4=$(host -t A ${HOST} | head -n 1 | cut -d\  -f4)
IPV6=$(host -t AAAA ${HOST} | head -n 1 | cut -d\  -f5)

echo "***GOT REQUEST TO AUDIT CONNECTIVITY***" >> ${LOCKFILE}
echo "Currently running $(opnsense-version) at $(date)" >> ${LOCKFILE}
if [ -n "${IPV4}" -a -z "${IPV4%%*.*}" ]; then
        echo "Checking connectivity for host: ${HOST} -> ${IPV4}" | ${TEE} ${LOCKFILE}
        (ping ${POPT} ${IPV4} 2>&1) | ${TEE} ${LOCKFILE}
        echo "Checking connectivity for repository (IPv4): ${URL}" | ${TEE} ${LOCKFILE}
        (pkg -4 update -f 2>&1) | ${TEE} ${LOCKFILE}
else
        echo "No IPv4 address could be found for host: ${HOST}" | ${TEE} ${LOCKFILE}
fi
if [ -n "${IPV6}" -a -z "${IPV6%%*:*}" ]; then
        echo "Checking connectivity for host: ${HOST} -> ${IPV6}" | ${TEE} ${LOCKFILE}
        (ping6 ${POPT} ${IPV6} 2>&1) | ${TEE} ${LOCKFILE}
        echo "Checking connectivity for repository (IPv6): ${URL}" | ${TEE} ${LOCKFILE}
        (pkg -6 update -f 2>&1) | ${TEE} ${LOCKFILE}
else
        echo "No IPv6 address could be found for host: ${HOST}" | ${TEE} ${LOCKFILE}
fi
echo '***DONE***' >> ${LOCKFILE}
2  After the script has executed "pkg -4 update -f" (in my IPV4-only environment) in the IPV4 section, there exists an SQLite 3.x database file:

Code: [Select]

/var/db/pkg/repo-OPNsense.sqlite
3  When the second (IPV6) half runs (executing"pkg -6 update -f"), the database file has been deleted if there is no IPV6 connectivity.  The rest of the files in the same directory remain untouched.

Perhaps it would be better if this script only executed the pkg command if there was the appropriate network connectivity?

I've taken the liberty to modify the script - it's shown below, and also attached.  I'll confess I had to learn how to safely get the return code from the first command in a pipe (see **1).

Code: [Select]

#!/bin/sh

# Copyright (C) 2021 Franco Fichtner <franco@opnsense.org>
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
# INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
# AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
# AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
# OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

LOCKFILE="/tmp/pkg_upgrade.progress"
TEE="/usr/bin/tee -a"

: > ${LOCKFILE}

URL=$(opnsense-update -M)
POPT="-c4 -s1500"

HOST=${URL#*://}
HOST=${HOST%%/*}
IPV4=$(host -t A ${HOST} | head -n 1 | cut -d\  -f4)
IPV6=$(host -t AAAA ${HOST} | head -n 1 | cut -d\  -f5)

echo "***GOT REQUEST TO AUDIT CONNECTIVITY***" >> ${LOCKFILE}
echo "Currently running $(opnsense-version) at $(date)" >> ${LOCKFILE}
if [ -n "${IPV4}" -a -z "${IPV4%%*.*}" ]; then
        echo "Checking connectivity for host: ${HOST} -> ${IPV4}" | ${TEE} ${LOCKFILE}
        exec 4>&1
        PRES=`{ { ping ${POPT} ${IPV4} 2>&1; printf $? 1>&3; } | ${TEE} ${LOCKFILE} 1>&4; } 3>&1`
        [ $PRES = 0 ] && echo "Checking connectivity for repository (IPv4): ${URL}" | ${TEE} ${LOCKFILE}
        [ $PRES = 0 ] && (pkg -4 update -f 2>&1) | ${TEE} ${LOCKFILE}
else
        echo "No IPv4 address could be found for host: ${HOST}" | ${TEE} ${LOCKFILE}
fi
if [ -n "${IPV6}" -a -z "${IPV6%%*:*}" ]; then
        echo "Checking connectivity for host: ${HOST} -> ${IPV6}" | ${TEE} ${LOCKFILE}
        exec 4>&1
        PRES=`{ { ping6 ${POPT} ${IPV6} 2>&1; printf $? 1>&3; } | ${TEE} ${LOCKFILE} 1>&4; } 3>&1`
        [ $PRES = 0 ] && echo "Checking connectivity for repository (IPv6): ${URL}" | ${TEE} ${LOCKFILE}
        [ $PRES = 0 ] && (pkg -6 update -f 2>&1) | ${TEE} ${LOCKFILE}
else
        echo "No IPv6 address could be found for host: ${HOST}" | ${TEE} ${LOCKFILE}
fi
echo '***DONE***' >> ${LOCKFILE}
I've also installed the new script on my firewall and it performs in my IPV4-only world just like the original, except that the database file doesn't get deleted.  I cannot check in an IPV6 environment, but visual inspection says there's a good chance it should work. 

**1
https://unix.stackexchange.com/questions/14270/get-exit-status-of-process-thats-piped-to-another

HTH

Peter

[franco]

Hi Peter,

Thanks for looking into it. Truth be told we can not know if the repository update will work or not despite all the safeguards. A robust strategy would be to add a recovery in this case, but then again I don't think it's very helpful if pkg doesn't keep its old database around on a failure. Maybe we can do a backup and move it back even to the state that it was before the connectivity audit? I'll take a look.


Cheers,
Franco

[franco]

This seems to do the trick

https://github.com/opnsense/core/commit/2a3f201d38


Cheers,
Franco

[tofflock]

This seems to do the trick

https://github.com/opnsense/core/commit/2a3f201d38

It does indeed!  It took me a little while to figure it out until I read the man page for pkg.conf.

Many thanks for the help.  Can we mark this as closed now?

PeterF

[franco]

Looks a little weird if I edit your original post, but you can edit it and add "[SOLVED]" in front of the title if you want to. This change will also be in 22.1.6. Thanks for helping to get to the bottom of it.

Interestingly enough the problem did always exist even previously but was a lot harder to diagnose when pkg would be allowed to chose if it used IPv6 or IPv4 by itself. I'm happy with the new state, however, as we can diagnose connection issues in both address families without tainting the working system.


Cheers,
Franco