503 Service unavailable after installing OPNsense 26.1.8

Started by RutgerDiehard, Today at 12:22:36 PM

Previous topic - Next topic
I installed OPNsense 26.1.8 update last night and didn't reboot as the update didn't require one.

This morning, I attempted to access the UI and got a blank page with 503 Service Unavailable. I SSH into the box and attempted an upgrade to the newly-released OPNsense 26.1.8_5 which hangs. So I reboot the box again, login to the UI and attempt the update. This hangs again with the UI unresponsive.

Are there any logs I can retrieve that would help? The UI is fine unless I attempt another update.


I didn't know there was a health check! Anyway, here's the output:

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 26.1.8_5 (amd64) at Wed May 13 11:38:01 BST 2026
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 26.1.7 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 26.1.7 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
mimugmail (Priority: 5)
SunnyValley (Priority: 7)
OPNsense (Priority: 11)
>>> Check installed plugins
os-acme-client 4.16_1
os-adguardhome-maxit 1.16
os-apcupsd 1.2_3
os-caddy 2.1.0
os-cpu-microcode-intel 1.1
os-crowdsec 1.0.12
os-ddclient 1.31
os-etpro-telemetry 1.8_1
os-gdrive-backup 1.0_1
os-maltrail 1.10_1
os-mdns-repeater 1.2
os-net-snmp 1.6_1
os-netdata 1.2_1
os-q-feeds-connector 1.6
os-sensei 2.5
os-sensei-agent 2.5
os-sensei-updater 2.0
os-sftp-backup 1.1_2
os-smart 2.4
os-sunnyvalley 1.5_2
os-telegraf 1.12.14
os-theme-advanced 1.1
os-theme-cicada 1.41_1
os-theme-dracula 0.7
os-theme-rebellion 1.9.4
os-theme-vicuna 1.51
os-zabbix72-agent 1.18
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" at 26.1.8_5 has 68 dependencies to check.
Checking packages: ........
dnsmasq-2.92_3,1 version mismatch, expected 2.92rel2,1
Checking packages: ........................................
pkg-2.6.2_1 repository mismatch: FreeBSD
pkg-2.6.2_1 version mismatch, expected 2.3.1_1
Checking packages: ..................... done
***DONE***

I'm not 100% sure that's the issue but it could have caused it:

pkg-2.6.2_1 repository mismatch: FreeBSD
pkg-2.6.2_1 version mismatch, expected 2.3.1_1

You need to revert pkg to our version from our repository. Otherwise upgrades will be disruptive.

# opnsense-revert pkg

(if the command doesn't work please post the full output)


Cheers,
Franco
"AI has absolutely reduced the cost of creating technical debt." -- ChatGPT

Thanks Franco,

I'm not sure that's done anything:

root@OPNsense:~ # opnsense-revert pkg
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
Fetching meta.conf:   0%
Fetching data:   0%
SunnyValley repository is up to date.
Updating mimugmail repository catalogue...
Fetching meta.conf: 100%     179 B   0.2 kB/s    00:01   
mimugmail repository is up to date.
All repositories are up to date.
The following packages will be fetched:

New packages to be FETCHED:
        pkg: 2.3.1_1 (6 MiB: 100.00% of the 6 MiB to download)

Number of packages to be fetched: 1

The process will require 6 MiB more space.
6 MiB to be downloaded.
Fetching pkg-2.3.1_1: 100%  6397 KiB   6.6 MB/s    00:01   
pkg-2.6.2_1: already unlocked
root@OPNsense:~ #

Running another healthcheck just gives me the same as before:

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 26.1.8_5 (amd64) at Wed May 13 12:00:50 BST 2026
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 26.1.7 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 26.1.7 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
mimugmail (Priority: 5)
SunnyValley (Priority: 7)
OPNsense (Priority: 11)
>>> Check installed plugins
os-acme-client 4.16_1
os-adguardhome-maxit 1.16
os-apcupsd 1.2_3
os-caddy 2.1.0
os-cpu-microcode-intel 1.1
os-crowdsec 1.0.12
os-ddclient 1.31
os-etpro-telemetry 1.8_1
os-gdrive-backup 1.0_1
os-maltrail 1.10_1
os-mdns-repeater 1.2
os-net-snmp 1.6_1
os-netdata 1.2_1
os-q-feeds-connector 1.6
os-sensei 2.5
os-sensei-agent 2.5
os-sensei-updater 2.0
os-sftp-backup 1.1_2
os-smart 2.4
os-sunnyvalley 1.5_2
os-telegraf 1.12.14
os-theme-advanced 1.1
os-theme-cicada 1.41_1
os-theme-dracula 0.7
os-theme-rebellion 1.9.4
os-theme-vicuna 1.51
os-zabbix72-agent 1.18
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" at 26.1.8_5 has 68 dependencies to check.
Checking packages: ........
dnsmasq-2.92_3,1 version mismatch, expected 2.92rel2,1
Checking packages: ........................................
pkg-2.6.2_1 repository mismatch: FreeBSD
pkg-2.6.2_1 version mismatch, expected 2.3.1_1
Checking packages: ..................... done
***DONE***

This bug is hilarious:

pkg-2.6.2_1: already unlocked

and existed in pkg for almost 2 years now.. only recently fixed as far as I know.[1][2]

# pkg lock -y pkg
# opnsense-revert pkg


Cheers,
Franco

--
[1] https://github.com/opnsense/update/issues/93
[2] https://github.com/freebsd/pkg/issues/2458
"AI has absolutely reduced the cost of creating technical debt." -- ChatGPT

That seems to have completed now:

root@OPNsense:~ # pkg lock -y pkg
Locking pkg-2.6.2_1
root@OPNsense:~ # opnsense-revert pkg
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
Fetching meta.conf:   0%
Fetching data:   0%
SunnyValley repository is up to date.
Updating mimugmail repository catalogue...
Fetching meta.conf: 100%     179 B   0.2 kB/s    00:01   
mimugmail repository is up to date.
All repositories are up to date.

No packages are required to be fetched.
Integrity check was successful.
Unlocking pkg-2.6.2_1
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
Fetching meta.conf:   0%
Fetching data:   0%
SunnyValley repository is up to date.
Updating mimugmail repository catalogue...
Fetching meta.conf: 100%     179 B   0.2 kB/s    00:01   
mimugmail repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be DOWNGRADED:
        pkg: 2.6.2_1 -> 2.3.1_1 [OPNsense]

Number of packages to be downgraded: 1

The process will require 2 MiB more space.
[1/1] Downgrading pkg from 2.6.2_1 to 2.3.1_1...
[1/1] Extracting pkg-2.3.1_1: 100%
pkg-static: warning: database version 38 is newer than libpkg(3) version 36, but still compatible
root@OPNsense:~ #

And another run of the healthcheck:

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 26.1.8_5 (amd64) at Wed May 13 12:58:29 BST 2026
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 26.1.7 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 26.1.7 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
mimugmail (Priority: 5)
SunnyValley (Priority: 7)
OPNsense (Priority: 11)
>>> Check installed plugins
pkg: warning: database version 38 is newer than libpkg(3) version 36, but still compatible
os-acme-client 4.16_1
os-adguardhome-maxit 1.16
os-apcupsd 1.2_3
os-caddy 2.1.0
os-cpu-microcode-intel 1.1
os-crowdsec 1.0.12
os-ddclient 1.31
os-etpro-telemetry 1.8_1
os-gdrive-backup 1.0_1
os-maltrail 1.10_1
os-mdns-repeater 1.2
os-net-snmp 1.6_1
os-netdata 1.2_1
os-q-feeds-connector 1.6
os-sensei 2.5
os-sensei-agent 2.5
os-sensei-updater 2.0
os-sftp-backup 1.1_2
os-smart 2.4
os-sunnyvalley 1.5_2
os-telegraf 1.12.14
os-theme-advanced 1.1
os-theme-cicada 1.41_1
os-theme-dracula 0.7
os-theme-rebellion 1.9.4
os-theme-vicuna 1.51
os-zabbix72-agent 1.18
>>> Check locked packages
pkg: warning: database version 38 is newer than libpkg(3) version 36, but still compatible
>>> Check for missing package dependencies
pkg: warning: database version 38 is newer than libpkg(3) version 36, but still compatible
Checking all packages: .......... done
>>> Check for missing or altered package files
pkg: warning: database version 38 is newer than libpkg(3) version 36, but still compatible
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" at 26.1.8_5 has 68 dependencies to check.
Checking packages: ........
dnsmasq-2.92_3,1 version mismatch, expected 2.92rel2,1
Checking packages: ............................................................. done
***DONE***

I then attempted another update from the UI which updated dnsmasq with no errors or UI crash. A further update shows everything up-to-date!

Thanks Franco

Happy to hear :)


Cheers,
Franco
"AI has absolutely reduced the cost of creating technical debt." -- ChatGPT