Upgrade to 25.1 re-installs 24.7.12_4

[ajr]

This is a HA setup. Master system has been upgraded successfully via GUI.
Upgrading the backup sytem gives a segmentation fault:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.7.12_4 (amd64) at Mon Feb  3 22:29:07 UTC 2025
Fetching changelog information, please wait... done
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 869 packages processed.
All repositories are up to date.
Child process pid=5088 terminated abnormally: Segmentation fault
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (1 candidates): . done
Processing candidates (1 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***
Upgrade to OS version 14.2 was successfull.

I then tried upgrading via console, but without luck.
(typescript appendened).

Any help appreciated,

ajr

PS:

Code Select Expand

*** opn2.in.chaos1.de: OPNsense 24.7.12_4 (amd64) ***

. . .
Proceed with this action? [25.1/y/N]: 25.1

Hi there,

. . .

Fetching packages-25.1-amd64.tar: ........................................................................... done
Fetching base-25.1-amd64.txz: ............... done
Fetching kernel-25.1-amd64.txz: ....... done
Extracting packages-25.1-amd64.tar... done
Extracting base-25.1-amd64.txz... done
Extracting kernel-25.1-amd64.txz... done
Please reboot.
>>> Invoking upgrade script 'sanity.sh'
Passed all upgrade tests.
>>> Invoking upgrade script 'cleanup.sh'
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-25.1-amd64.txz... done
>>> Invoking stop script 'beep'
>>> Invoking stop script 'freebsd'
Stopping flowd.
Waiting for PIDS: 1193 1298.
Stopping flowd_aggregate...done
>>> Invoking stop script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
>>> Invoking backup script 'rrd'
>>> Invoking stop script 'config'
Shutdown NOW!
shutdown: [pid 21274]
                                                                               
*** FINAL System shutdown message from root@opn2.in.chaos1.de ***           

System going down IMMEDIATELY                                                 

                                                                               

System shutdown time has arrived
Connection to opn2.in.chaos1.de closed by remote host.
Connection to opn2.in.chaos1.de closed.
. . .
axels-imac:.../Developing/home2l/home2l_config ajr$ opn2
Last login: Tue Feb  4 11:44:19 2025 from 192.168.220.124
----------------------------------------------
|      Hello, this is OPNsense 24.7          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:     https://opnsense.org/         |         @@@\\\   ///@@@
| Handbook:    https://docs.opnsense.org/    |       ))))))))   ((((((((
| Forums:      https://forum.opnsense.org/   |         @@@///   \\\@@@
| Code:        https://github.com/opnsense   |        @@@@         @@@@
| Reddit:      https://reddit.com/r/opnsense |         @@@@@@@@@@@@@@@
----------------------------------------------

*** opn2.in.chaos1.de: OPNsense 24.7.12_4 (amd64) ***

. . .
Enter an option: 8

root@opn2:~ # freebsd-version
14.2-RELEASE
root@opn2:~ # exit
exit
. . .

Enter an option: 12

Fetching change log information, please wait... done

This will automatically fetch all available updates and apply them.

A major firmware upgrade is available for this installation: 25.1

Make sure you have read the release notes and migration guide before
attempting this upgrade.  Approx. 1000MB will need to be downloaded and
require 2000MB of free space to unpack.  Continue with this major upgrade
by typing the major upgrade version number displayed above.

Minor updates may be available, answer 'y' to run them instead.

Proceed with this action? [25.1/y/N]: 25.1

. . .


Fetching packages-25.1amd64.tar: ......................................................................... done
Extracting packages-25.1-amd64.tar... done
Please reboot.
>>> Invoking upgrade script 'sanity.sh'
Passed all upgrade tests.
>>> Invoking upgrade script 'cleanup.sh'

*** opn2.in.chaos1.de: OPNsense 24.7.12_4 (amd64) ***

. . .

  6) Reboot system                      13) Restore a backup

Enter an option: 6

The system will reboot. Do you want to proceed? [y/N]: y

. . .

axels-imac:.../Developing/home2l/home2l_config ajr$ opn2
Last login: Tue Feb  4 11:55:40 2025 from 192.168.220.124
----------------------------------------------
|      Hello, this is OPNsense 24.7          |         @@@@@@@@@@@@@@@


[franco]

There's no useful information here regarding why it failed. The upgrade log should be in /var/cache/opnsense-update/.upgrade.log


Cheers,
Franco

[ajr]

root@opn2:~ # ls -la /var/cache/opnsense-update/.upgrade.log
ls: /var/cache/opnsense-update/.upgrade.log: No such file or directory

[ajr]

Just created missing directory /var/log/cache and retrying . . .

[ajr]

Nothing appears there.

I did an audit and found issues:

Code Select Expand

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 24.7.12_4 (amd64) at Tue Feb  4 17:44:03 UTC 2025
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 25.1 is incorrect, expected: 24.7.12
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 25.1 is incorrect, expected: 24.7.12
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
No plugins found.
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" at 24.7.12_4 has 69 dependencies to check.
Checking packages: .................................................
pkg-1.21.3 repository mismatch: FreeBSD
pkg-1.21.3 version mismatch, expected 1.19.2_5
Checking packages: ..................... done
***DONE***
Looking at pkg repo config shows:

Code Select Expand

root@opn2:~ # cat /usr/local/etc/pkg/repos/FreeBSD.conf
FreeBSD: { enabled: no }
root@opn2:~ # cat /usr/local/etc/pkg/repos/OPNsense.conf
OPNsense: {
  fingerprints: "/usr/local/etc/pkg/fingerprints/OPNsense",
  url: "https://pkg.opnsense.org/${ABI}/24.7/latest",
  signature_type: "fingerprints",
  priority: 11,
  enabled: yes
}

So it should use opnsense repo, but it used the freebsd one.
Please advice a fix.

[franco]

Ok that makes sense... you broke it:

pkg-1.21.3 repository mismatch: FreeBSD
pkg-1.21.3 version mismatch, expected 1.19.2_5

;)

# opnsense-revert pkg

And try the upgrade again...

[ajr]

The audit is now correct.

Trying the update with the console menu option 12 did not succeed.

The GUI still shows 25.1 as running version on the update page.

Should I use opnsense-update -u -r 25.1  or some other options ?
Which ones ?

ajr

[franco]

The absence of .upgrade.log makes me wonder if it ever starts installing packages, possibly because a sanity check fails, but it does not make much sense. Without that data it's not easy to say what went wrong. Proposing a workaround also not great.

The following may work, but it may not since I don't know what is wrong:

# opnsense-update -up 25.1

And reboot.

Edit: I have an idea what could be the issue.


Cheers,
Franco

[ajr]

I did

Code Select Expand

root@opn2:~ # opnsense-update -up 25.1
Usage: man opnsense-update
root@opn2:~ # opnsense-update -ur 25.1
Fetching packages-25.1-amd64.tar: .......................................................................... done
Extracting packages-25.1-amd64.tar... done
Please reboot.
root@opn2:~ # reboot
. . .
Last login: Wed Feb  5 11:57:49 2025 from 192.168.220.124
----------------------------------------------
|      Hello, this is OPNsense 24.7          |         @@@@@@@@@@@@@@@
. . .
root@opn2:~ # freebsd-version
14.2-RELEASE
root@opn2:~ # ls -l /var/cache/opnsense-update/
total 2
drwxr-xr-x  3 root wheel  3 Feb  5 14:58 .sets.pending
-rw-r--r--  1 root wheel 30 Feb  5 15:08 .upgrade.log
prw-r--r--  1 root wheel  0 Feb  5 15:08 .upgrade.pipe

What shall I do now ?

[franco]

Share the output of /var/cache/opnsense-update/.upgrade.log, you can also use

# opnsense-update -G


Cheers,
Franco

[ajr]

root@opn2:~ #  opnsense-update -G
bash-5.2.37: already unlocked

[newsense]

I believe Franco also needed this file

Code Select Expand

cat /var/cache/opnsense-update/.upgrade.log [code]


Also, any chance you might have deleted /var/cache while troubleshooting - before coming here that is - and that is why you had to manually recreate the missing folder ?

(In many years of using OPNsense on many systems I haven't seen that folder vanish during an update, successful or not.)

[franco]

root@opn2:~ #  opnsense-update -G
bash-5.2.37: already unlocked

Just a single line without an actual error? This is more difficult than I expected.

BTW, "opnsense-update -G" and "cat /var/cache/opnsense-update/.upgrade.log" are the same thing :)


Cheers,
Franco

[ajr]

Also, any chance you might have deleted /var/cache while troubleshooting - before coming here that is - and that is why you had to manually recreate the missing folder ?

I never deleted any log files/directories.

[ajr]

Hi Franco,


The check for updates page shows

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.7.12_4 (amd64) at Thu Feb  6 10:21:00 UTC 2025
Fetching changelog information, please wait... done
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 869 packages processed.
All repositories are up to date.
Child process pid=2663 terminated abnormally: Segmentation fault
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (1 candidates): . done
Processing candidates (1 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.

The next page shows current version 25.1 and new version 24.7.12 for both
base and kernel.

So the update script seems to be confused by the combination of OS 14.2 and
opnsense 24.7.3

Shall I try a opnsense-update -kr 24.7.12 ?


After an upgrade attempt (opnsense-update -ur 25.1), the audit log again shows
    pkg-1.21.3 version mismatch, expected 1.19.2_5

Now, reverting the pkg does no longer work:

root@opn2:~ # pkg info | grep pkg
pkg-1.21.3                     Package manager
root@opn2:~ # opnsense-revert -r 24.7.12 pkg
Fetching pkg.pkg: ...... done
Verifying signature with trusted certificate pkg.opnsense.org.20240611... done
pkg-1.21.3: already unlocked

root@opn2:~ # pkg info | grep pkg
pkg-1.21.3                     Package manager
root@opn2:~ # pkg lock pkg
pkg-1.21.3: lock this package? [y/N]: y
Locking pkg-1.21.3
root@opn2:~ # opnsense-revert -r 24.7.12 pkg
Fetching pkg.pkg: .... done
Verifying signature with trusted certificate pkg.opnsense.org.20240611... done
Unlocking pkg-1.21.3
Installing pkg-1.19.2_5...
package pkg is already installed, forced install
Extracting pkg-1.19.2_5: 100%

But now the audit shows:

pkg-1.19.2_5 repository mismatch: unknown-repository

From the log, it seems, unlocking an already unlocked package seems to be fatal.
bash-5.2.37: already unlocked

So I removed bash earlier.

But now:

root@opn2:~ # opnsense-update -G
beep-1.0_2: already unlocked

Any clue ?
Should I lock all packages and retry ?

Any options left to recover my situation ?

Ajr