Cannot Upgrade to 19.1 from 18.7.10_4

[kuerious]

now I am just annoyed, mostly, at the lack of logs rather than the upgrade failing

Hear hear. Logs are the first place I went looking for clues, to no avail. Other FOSS applications I use (even https://dietpi.com/) have a straightforward and easy-to-use "here's the error log, go to this address _____ and submit a report". You'd think a competitive product like OPNsense would as well.

[franco]

For reasons of perspective let me chime in here:

The worst thing is that users modify their installs in unknown ways, not include this in their "steps to reproduce" and then also don't provide console output on the upgrade (serial capture, VGA video with a mobile phone) as we always suggest supervised upgrades and we're left to guess what went wrong while also workarounds are mentioned elsewhere in the forum.

Anyone can complain, anyone can add logging, but it won't solve complaints and unless there is a time machine nothing will change about the current situation.

[kuerious]

You are correct, to a point.

For example, I - for one - don't "modify" something like OPN as I have no programmatic knowledge. As I stated in my earlier posts, not only did I follow the rules as laid out for us but I also scoured the forums for the knowledge to save myself before even asking for help.

I'm certain there are plenty of people who visit the forums once, make loud complaints to anyone involved, and then never respond or post again. In that sense, I am not one of those people. I posted asking for assistance and agreed with another user's point (while also making a suggestion for an enhancement that would allow an end-user to find the answers they need).

It would seem you took it as criticism. And not even as constructive criticism. Perhaps a PM or other format to ask your audience what they mean ... instead of publicly posting a complaint which makes no headway in an end user's (or two) issue(s).

[franco]

I'm not taking it as criticism. I merely question the rationale behind it.

We do not need logging changes for a system that was in place since end of 2016 that now "suddenly" breaks. The likelihood of an external source for this behaviour is high so I'd like to know this first before deciding how useful logging really is.

I've asked for serial console output for a number of times now, I got nothing. I have hints of how to reproduce, but it would take a few hours of time that nobody is going to be sponsoring so I would rather like to spend it on something more productive for the project like shipping fixes that already exist for 19.1.2 in a timely fashion.


Cheers,
Franco

[franco]

PS: The workaround described here https://forum.opnsense.org/index.php?topic=9521.msg51688#msg51688 is more effective than any discussion on the matter so far. We have yet to hear from someone saying this didn't work.

[dp]

Franco (and others),

I too am having this same issue. For me it is a physical and for evaluating purposes so it is not in production. As for the console output I can confirm there isn't any. No errors, no real responses, just what has been mentioned, downloading, installing and reboot. I can film it if you like.

Now to that end is there any options to run the upgrade command from a prompt with a switch(s) set to get debug commands piped to a file? Although the work around to install and restore the config exists, if there is a bug lurking somewhere it would be great to find it now instead of it possibly showing up later in a uglier/widespread way.

As for my setup it is a stock install with three interfaces: Inside, Outside and Management. I can even upload my config if you like. Like I said nothing in the box, no rules other than what is needed to get the management interface to work.

It may take me a few days to test things but am happy to try and help.

[dp]

Here is the upgrade output from the command line:

root@OPNsense:~ # opnsense-update -Vur 19.1
+ [ -n '' ]
+ [ '' '=' -T ]
+ [ -z '' ]
+ DO_KERNEL=-k
+ DO_BASE=-b
+ DO_PKGS=-p
+ [ -n '' ]
+ [ -n '' ]
+ [ -n '' ]
+ [ -z '' ]
+ [ -n -k -a -n '' -a -z -u ]
+ [ -n -b -a -n '' -a -z -u ]
+ [ -n '' ]
+ [ -n '' ]
+ [ -n '' ]
+ [ -n '' ]
+ [ -n '' ]
+ [ -z 19.1 ]
+ [ -b '=' -B ]
+ [ -p '=' -P ]
+ [ -n '' ]
+ [ -p '=' -p -a -z -u ]
+ FLAVOUR=Base
+ [ -n '' ]
+ [ -f /usr/local/bin/openssl ]
+ /usr/local/bin/openssl version
+ awk '{ print $1 }'
+ FLAVOUR=OpenSSL
+ PACKAGESSET=packages-19.1-OpenSSL-amd64.tar
+ KERNELSET=kernel-19.1-amd64.txz
+ BASESET=base-19.1-amd64.txz
+ mirror_abi
+ sed -n 's/^[[:space:]]*url:[[:space:]]*\"pkg\+\(.*\/${ABI}\/[^\/]*\)\/.*/\1/p' /usr/local/etc/pkg/repos/OPNsense.conf
+ MIRROR='http://mirrors.nycbug.org/pub/opnsense/${ABI}/18.7'
+ opnsense-verify -a
+ ABI=FreeBSD:11:amd64
+ [ -n '' ]
+ eval 'MIRROR=http://mirrors.nycbug.org/pub/opnsense/${ABI}/18.7'
+ MIRROR=http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7
+ echo http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7
+ MIRROR=http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets
+ [ -n '' ]
+ [ -z '' ]
+ [ 19.1 '=' 18.7.10 -a -n -k ]
+ [ 19.1 '=' 18.7.10 -a -n -b ]
+ [ -z -k-b-p ]
+ [ -p '=' -p ]
+ fetch_set packages-19.1-OpenSSL-amd64.tar
+ STAGE1='opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/packages-19.1-OpenSSL-amd64.tar.sig http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/packages-19.1-OpenSSL-amd64.tar.sig'
+ STAGE2='opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/packages-19.1-OpenSSL-amd64.tar http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/packages-19.1-OpenSSL-amd64.tar'
+ STAGE3='opnsense-verify -q /var/cache/opnsense-update/50494/packages-19.1-OpenSSL-amd64.tar'
+ [ -n '' ]
+ [ -n '' ]
+ echo -n 'Fetching packages-19.1-OpenSSL-amd64.tar: .'
Fetching packages-19.1-OpenSSL-amd64.tar: .+ mkdir -p /var/cache/opnsense-update/50494
+ opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/packages-19.1-OpenSSL-amd64.tar.sig http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/packages-19.1-OpenSSL-amd64.tar.sig
.+ opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/packages-19.1-OpenSSL-amd64.tar http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/packages-19.1-OpenSSL-amd64.tar
...+ [ -n -V ]
+ opnsense-verify -q /var/cache/opnsense-update/50494/packages-19.1-OpenSSL-amd64.tar
+ echo ' done'
done
+ [ -b '=' -b ]
+ [ -z '' -o -n -u ]
+ rm -f /usr/local/opnsense/version/base.lock
+ fetch_set base-19.1-amd64.txz
+ STAGE1='opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/base-19.1-amd64.txz.sig http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/base-19.1-amd64.txz.sig'
+ STAGE2='opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/base-19.1-amd64.txz http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/base-19.1-amd64.txz'
+ STAGE3='opnsense-verify -q /var/cache/opnsense-update/50494/base-19.1-amd64.txz'
+ [ -n '' ]
+ [ -n '' ]
+ echo -n 'Fetching base-19.1-amd64.txz: .'
Fetching base-19.1-amd64.txz: .+ mkdir -p /var/cache/opnsense-update/50494
+ opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/base-19.1-amd64.txz.sig http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/base-19.1-amd64.txz.sig
.+ opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/base-19.1-amd64.txz http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/base-19.1-amd64.txz
................................+ [ -n -V ]
+ opnsense-verify -q /var/cache/opnsense-update/50494/base-19.1-amd64.txz
+ echo ' done'
done
+ [ -k '=' -k ]
+ [ -z '' -o -n -u ]
+ rm -f /usr/local/opnsense/version/kernel.lock
+ fetch_set kernel-19.1-amd64.txz
+ STAGE1='opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/kernel-19.1-amd64.txz.sig http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/kernel-19.1-amd64.txz.sig'
+ STAGE2='opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/kernel-19.1-amd64.txz http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/kernel-19.1-amd64.txz'
+ STAGE3='opnsense-verify -q /var/cache/opnsense-update/50494/kernel-19.1-amd64.txz'
+ [ -n '' ]
+ [ -n '' ]
+ echo -n 'Fetching kernel-19.1-amd64.txz: .'
Fetching kernel-19.1-amd64.txz: .+ mkdir -p /var/cache/opnsense-update/50494
+ opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/kernel-19.1-amd64.txz.sig http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/kernel-19.1-amd64.txz.sig
.+ opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/kernel-19.1-amd64.txz http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/kernel-19.1-amd64.txz
...........+ [ -n -V ]
+ opnsense-verify -q /var/cache/opnsense-update/50494/kernel-19.1-amd64.txz
+ echo ' done'
done
+ [ -k '=' -k ]
+ echo '!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!'
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
+ echo '! A critical upgrade is in progress. !'
! A critical upgrade is in progress. !
+ echo '! Please do not turn off the system. !'
! Please do not turn off the system. !
+ echo !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+ [ -p '=' -p -a -n -u ]
+ echo -n 'Extracting packages-19.1-OpenSSL-amd64.tar...'
Extracting packages-19.1-OpenSSL-amd64.tar...+ rm -rf '/var/cache/opnsense-update/.sets.pending/packages-*'
+ mkdir -p /var/cache/opnsense-update/.sets.pending/packages-19.1
+ pkg-static clean -qya
+ tar -C/var/cache/opnsense-update/.sets.pending/packages-19.1 -xpf /var/cache/opnsense-update/50494/packages-19.1-OpenSSL-amd64.tar
+ echo 19.1
+ [ -n '' ]
+ echo ' done'
done
+ [ -b '=' -b -a -n -u ]
+ echo -n 'Extracting base-19.1-amd64.txz...'
Extracting base-19.1-amd64.txz...+ rm -rf '/var/cache/opnsense-update/.sets.pending/base-*'
+ mkdir -p /var/cache/opnsense-update/.sets.pending
+ mv /var/cache/opnsense-update/50494/base-19.1-amd64.txz /var/cache/opnsense-update/.sets.pending
+ echo 19.1
+ echo ' done'
done
+ [ -k '=' -k ]
+ install_kernel
+ KLDXREF='kldxref /boot/kernel'
+ [ -n -u ]
+ KLDXREF=:
+ echo -n 'Installing kernel-19.1-amd64.txz...'
Installing kernel-19.1-amd64.txz...+ mkdir -p /boot/kernel /boot/kernel.old /usr/lib/debug/boot/kernel
+ rm -r /boot/kernel.old /usr/lib/debug/boot/kernel
+ mv /boot/kernel /boot/kernel.old
+ tar -C / -xpf /var/cache/opnsense-update/50494/kernel-19.1-amd64.txz
+ :
+ echo ' done'
done
+ [ -n -b -a -z -u ]
+ [ -p '=' -P -a -z -u ]
+ [ -z '' ]
+ rm -rf /var/cache/opnsense-update/50494
+ echo 'Please reboot.'
Please reboot.


Did a reboot and still on version 18.

[hbc]

I had the same issue. Just the base files and kernel have been installed. Once rebooted version was still 18.7.10.

I exported configuration, made a fresh install with 19.1 and used the importer during setup to restore configuration. The update from 19.1 to 19.1.1 worked without any problems.

[dp]

hbc,

I am aware of that work around. However I am willing to do this in a effort to find the root cause. At the moment it is only affecting a small number of people but at a future juncture it could be a problem that affects a large group or everyone. Or it may never happen again. Since I have a setup that is exhibiting the symptoms and I am not in production and I am willing to do some testing and digging to find the root cause, I put the offer out there. If there is no interest then so be it.

I look at problems like this: You are using a metal detector. It goes off during a search. No one else had their detector go off. Until you dig it up and find out what it was you have no way of knowing if it was a piece of scrap or a landmine. Where is was at could be on a obscure spot or in what will become the main path. There is no way to know what will happen until you know what it was and where it is.

I am just offering my time and the use of my setup to test. If there is no interest then so be it, I will move on but if the powers to be want to dig in then I am willing. I know just enough to be dangerous. Franco said that he had seen no screen shots and had no feedback so I was looking to provide that. I am new to this platform and have a bit of Linux experience (enough to be dangerous) so I am not a complete idiot (just a partial one, just ask my wife). I did some digging and read the man page for opnsense-update and found the -V option. So I decided to provide that output. Did you see any issues in the output that were significant?

[dp]

After running the command from a couple post back, today I logged back into the firewall and went to the upgrade page. I found the page as shown in the attached picture. It shows the current kernel as 19.1 and the upgrade as 18.7.10. Why would that be? It looks like it is going to roll it back. Could this have something to do with the failed upgrade?

[Bonkerton]

Here is the upgrade output from the command line:

root@OPNsense:~ # opnsense-update -Vur 19.1
------
.+ opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/50494/packages-19.1-OpenSSL-amd64.tar http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/packages-19.1-OpenSSL-amd64.tar
...+ [ -n -V ]

------
Did a reboot and still on version 18.

I re-did a V18 setup in a Virtualbox-VM and updated it from command line like you, then I diff'ed the command line output. The only differences were
- the '/var/cache/opnsense-update/85548' folder.
           I assume the folder-name/-number here is random

and this here:

.+ opnsense-fetch -a -T 30 -q -o /var/cache/opnsense-update/85548/packages-19.1-OpenSSL-amd64.tar http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/packages-19.1-OpenSSL-amd64.tar
........................................................................+ [ -n -V ]

I have a lot more dots. Now I don't know if they signify the size of the file or just the time it took to download.

If I do a
wget http://mirrors.nycbug.org/pub/opnsense/FreeBSD:11:amd64/18.7/sets/packages-19.1-OpenSSL-amd64.tar
at my Linux CLI the file that's downloaded is
'Length: 494723072 (472M)'
How big is yours ?

I rebooted the VM rebooted and it came back up with V19.1.1.

[newsense]

You're one step away from 19.1.2 which is $current. ;-)

[dp]

newsense

He is testing to compare to my output.

Bonkerton

My setup does not have wget but I was able to use fetch and pull down the package. I get the same size as you did. The man page indicated that the dots were time related, they just let you know it is doing something "whistle while you work" and are not related to size. My pipe is around 500Meg and it took 1m32s to download.

Thanks for testing.

[dp]

Continuing to poke around and got this output which agrees with the screenshot I got before:

root@OPNsense:/var/cache/opnsense-update # uname -a
FreeBSD OPNsense.vw.edu 11.2-RELEASE-p8-HBSD FreeBSD 11.2-RELEASE-p8-HBSD  31af16db12b(stable/19.1)  amd64

So it appears the kernel is actually on 19.1 but the OPNSense package did not upgrade? Or am I reading this wrong?

[franco]

The -u switch will do the major upgrade and then it won't update again.

Did you install any ports manually? From FreeBSD packages? Sensei plugin maybe?

On the box do this:

# opnsense-update -fp -n "19.1\/latest"

...and reboot.


Cheers,
Franco