Upgrade from 17.7.12 to 18.* not completing

[mausy5043]

This morning I'm trying to upgrade from OPNsense 17.7.12_1-amd64 to the current 18.* from the WebGUI System > Firmware > Updates  [Check for upgrades] > [Upgrade]

After a reboot I am now on:
OPNsense 17.7.12_1-amd64
FreeBSD 11.1-RELEASE-p6
OpenSSL 1.0.2n 7 Dec 2017

but still no 18.*.

What am I doing wrong?

P.S. Is there an upgrade path using the CLI instead of the GUI?

[mausy5043]

Updating from the CLI rolls me back...

Code Select Expand


% sudo opnsense-update
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking for upgrades (18 candidates): 100%
Processing candidates (18 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking integrity... done (0 conflicting)
Nothing to do.
Nothing to do.
Fetching base-17.7.10-amd64.obsolete: ... done
Fetching base-17.7.10-amd64.txz: ....... done
Fetching kernel-17.7.10-amd64.txz: .... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-17.7.10-amd64.txz... done
Installing base-17.7.10-amd64.txz... done
Installing base-17.7.10-amd64.obsolete... done
Please reboot.


Upgrading via the GUI gave me a hint (see image)

So I logged in:

Code Select Expand


% sudo pkg update
Updating OPNsense repository catalogue...
pkg: Repository OPNsense load error: access repo file(/var/db/pkg/repo-OPNsense.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    1 KiB   1.5kB/s    00:01
Fetching packagesite.txz: 100%  126 KiB 128.6kB/s    00:01
Processing entries: 100%
OPNsense repository update completed. 462 packages processed.
All repositories are up to date.


Which is an interesting output.
After a reboot I tried that again:

Code Select Expand


% sudo pkg update
Updating OPNsense repository catalogue...
pkg: Repository OPNsense has a wrong packagesite, need to re-create database
Fetching meta.txz: 100%    1 KiB   1.5kB/s    00:01
Fetching packagesite.txz: 100%  126 KiB 128.6kB/s    00:01
Processing entries: 100%
OPNsense repository update completed. 462 packages processed.
All repositories are up to date.
admin@gateway:~ % sudo pkg update
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.


[mausy5043]

Continuing:

See image.

After the reboot I still have 17.7

So, I tried using the CLI as follows:

Code Select Expand


% su root
Password:

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Upgrade from console
  6) Reboot system                      13) Restore a backup

Enter an option: 12

Fetching change log information, please wait... done

This will automatically fetch all available updates, apply them,
and reboot if necessary.

A major firmware upgrade is available for this installation: 18.1

Make sure you have read the release notes and migration guide before
attempting this upgrade.  Around 300MB will need to be downloaded and
require 600MB of free space.  Continue with this major upgrade by
typing the major upgrade version number displayed above.

Minor updates may be available, answer 'y' to run them instead.

Proceed with this action? [18.1/y/N]: 18.1

Fetching packages-18.1-OpenSSL-amd64.tar: ................. done
Extracting packages-18.1-OpenSSL-amd64.tar...pkg-static: Repository OPNsense missing. 'pkg update' required
pkg-static: No package database installed.  Nothing to do!
done
Please reboot.
>>> Invoking stop script 'beep'
>>> Invoking stop script 'freebsd'
Performing sanity check on squid configuration.
Configuration for squid passes.
Stopping squid.
Waiting for PIDS: 17289.
Stopping suricata.
Waiting for PIDS: 19457.
>>> Invoking stop script 'backup'
Cannot 'stop' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestop' instead of 'stop'.
Shutdown NOW!
shutdown: [pid 84616]

*** FINAL System shutdown message from admin

Still on 17.7

Looks like I'm running in circles.
HELP!

[mausy5043]

After reboot (see image).

GUI wants to roll back again...

WTF? :o

[franco]

Double-check these first and let us know their output before we continue:

# uname -a
# freebsd-version -u


Thanks,
Franco

[mausy5043]

Double-check these first and let us know their output before we continue:

# uname -a
# freebsd-version -u


Thanks,
Franco

Code Select Expand


% uname -a
FreeBSD gateway.lan 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6  6621d681e(stable/18.1)  amd64
% freebsd-version -u
11.1-RELEASE-p6


[franco]

# opnsense-update -sn "18.1\/latest"
# pkg bootstrap -f

type "y"

# pkg upgrade -f

or if that doesn't work

# pkg install opnsense


Cheers,

[mausy5043]

# opnsense-update -sn "18.1\/latest"
# pkg bootstrap -f

type "y"

# pkg upgrade -f

or if that doesn't work

# pkg install opnsense


Cheers,

Code Select Expand


% sudo opnsense-update -sn "18.1\/latest"
% sudo pkg bootstrap -f
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.opnsense.org/FreeBSD:11:amd64/18.1/latest, please wait...
Verifying signature with trusted certificate pkg.opnsense.org.20171219... done
Installing pkg-1.10.3_1...
package pkg is already installed, forced install
Extracting pkg-1.10.3_1: 100%
% sudo pkg upgrade -f
Updating OPNsense repository catalogue...
pkg: Repository OPNsense has a wrong packagesite, need to re-create database
Fetching meta.txz: 100%    1 KiB   1.5kB/s    00:01
Fetching packagesite.txz: 100%  126 KiB 129.0kB/s    00:01
Processing entries: 100%
OPNsense repository update completed. 468 packages processed.
All repositories are up to date.
Updating database digests format: 100%
Checking for upgrades (149 candidates):  14%
python35 has no direct installation candidates, change it to python27? [Y/n]: Y
Checking for upgrades (149 candidates): 100%
Processing candidates (149 candidates): 100%
pkg: sqlite error while executing UPDATE packages SET name=?1  WHERE name=?2; in file pkg_jobs.c:1731: UNIQUE constraint failed: packages.name
:
[143/144] Fetching py27-dnspython-1.15.0.txz: 100%  167 KiB 170.6kB/s    00:01
Checking integrity...Assertion failed: (strcmp(uid, p->uid) != 0), function pkg_conflicts_check_local_path, file pkg_jobs_conflicts.c, line 386.
Child process pid=54683 terminated abnormally: Abort trap


Retried without trying to replace python35 seemed to work.
Rebooted (may not have been necessary).

Status acc. the GUI:

Code Select Expand


OPNsense 18.1.2_2-amd64
FreeBSD 11.1-RELEASE-p6
OpenSSL 1.0.2n 7 Dec 2017


[franco]

Code Select Expand

python35 has no direct installation candidates, change it to python27? [Y/n]:

Oh geez, I love interactive prompts in scripted command line tools. That's what halted your upgrade. Not a lot we can do other than find and disable it. Until the next prompt comes along.... :(

I guess you are ok now. We don't offer Python 3.5. It will cause further trouble.


Cheers,
Franco

[franco]

For the record, we upgrade using "pkg upgrade -fy" which means force and assume 'yes' assuming it's a forceful operation that does the right thing. The answer to the question pkg asked should have been "n" instead, which in turn would mean the question was asked the wrong way around to avoid an update stall. Superb. oO

EDIT: Currently asking the pkgng folks what they make of this.... :)

[mausy5043]

We don't offer Python 3.5. It will cause further trouble.

:-[

Code Select Expand


% whereis python35
python35: /usr/ports/lang/python35


Hmm ???

[franco]

The ports that we maintain and provide packages for are:

https://github.com/opnsense/tools/blob/master/config/18.1/ports.conf

The ports tree with its 25000 ports mainly FreeBSD / HardenedBSD and can only be operated manually.

We cannot provide safety for more than the ports that we offer as binary packages.


Cheers,
Franco

[mausy5043]

We cannot provide safety for more than the ports that we offer as binary packages.

Clear.
I'll see if I can downgrade my scripts to Python 2.7. Shouldn't be too hard. I just used 3.5 because all my other Python projects use it.

P.S. Please be aware that Python 2.7 is EOL in 2020 (see PEP466: https://www.python.org/dev/peps/pep-0466/).

[franco]

Yes, good point. We have tried to start a migration, but as it looks that's going to have to wait for 2019 as the only thing we could achieve was adding Python 3 in parallel, bloating the install and updates.

When that switch happens, there will be fallout for sure. The Netflow bindings are not running on Python 3. :(


Cheers,
Franco

[franco]

Pushed this to FreeBSD pkg: https://github.com/freebsd/pkg/issues/1687


Cheers,
Franco