Ever since updating to 25.x, every time I log in to the gui and check for updates, I get the message " The upgrade has finished and your device is being rebooted at the moment, please wait..." and I have to hard power cycle the device to clear it. I'm guessing there is an auto update at some point between these checks but it never fully takes or reboots to finish unless I hard power cycle.
Look for hanging processes that prevent the reboot like e.g. crowdsec. Kill these manually.
Do a healthcheck from the Firmware section and post here the full output.
I'm not well versed, so what's the best way to do these commands?
Start with the health check - that can be done from the UI. System > Firmware.
From the web UI, when I navigate to any section regarding firmware it gives me the pop up advising the system is rebooting.
QuoteThe upgrade has finished and your device is being rebooted at the moment, please wait...
Login via ssh as root and use the command
ps awwux
Please report the output.
Like this?
Quoteroot@OPNsense:~ # ps awwux
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 11 199.5 0.0 0 64 - RNL 18Feb25 53493:21.75 [idle]
root 64000 100.0 16.2 1557724 1325780 - Rs 18Feb25 26095:07.01 /usr/local /bin/suricata -D --pcap=igc0 --pidfile /var/run/suricata.pid -c /usr/local/etc/s uricata/suricata.yaml
root 77618 100.0 0.3 34536 21876 - Rs 19Feb25 23783:17.44 /usr/local /bin/python3 /usr/local/opnsense/scripts/etpro_telemetry/send_telemetry.py (pyth on3.11)
root 41031 1.3 1.9 238304 155660 - S 12:19 0:45.44 /usr/local /bin/python3 /usr/local/opnsense/scripts/unbound/logger.py (python3.11)
root 0 0.0 0.0 0 1680 - DLs 18Feb25 84:56.89 [kernel]
root 1 0.0 0.0 12328 1092 - SLs 18Feb25 0:02.73 /sbin/init
root 2 0.0 0.0 0 64 - WL 18Feb25 1:45.82 [clock]
root 3 0.0 0.0 0 80 - DL 18Feb25 0:00.00 [crypto]
root 4 0.0 0.0 0 48 - DL 18Feb25 0:00.00 [cam]
root 5 0.0 0.0 0 16 - DL 18Feb25 0:00.00 [busdma]
root 6 0.0 0.0 0 896 - DL 18Feb25 0:51.90 [zfskern]
root 7 0.0 0.0 0 16 - DL 18Feb25 3:22.23 [pf purge]
root 8 0.0 0.0 0 16 - DL 18Feb25 1:38.99 [rand_harv estq]
root 9 0.0 0.0 0 48 - DL 18Feb25 0:54.12 [pagedaemo n]
root 10 0.0 0.0 0 16 - DL 18Feb25 0:00.00 [audit]
root 12 0.0 0.0 0 256 - WL 18Feb25 0:38.18 [intr]
root 13 0.0 0.0 0 48 - DL 18Feb25 0:00.02 [geom]
root 14 0.0 0.0 0 16 - DL 18Feb25 0:00.00 [sequencer 00]
root 15 0.0 0.0 0 80 - DL 18Feb25 0:11.21 [usb]
root 16 0.0 0.0 0 16 - DL 18Feb25 0:11.76 [acpi_ther mal]
root 17 0.0 0.0 0 16 - DL 18Feb25 0:00.00 [vmdaemon]
root 18 0.0 0.0 0 80 - DL 18Feb25 0:11.22 [bufdaemon ]
root 19 0.0 0.0 0 16 - DL 18Feb25 0:02.66 [vnlru]
root 20 0.0 0.0 0 16 - DL 18Feb25 0:05.16 [syncer]
root 32 0.0 0.0 0 16 - DL 18Feb25 0:00.10 [aiod1]
root 33 0.0 0.0 0 16 - DL 18Feb25 0:00.10 [aiod2]
root 34 0.0 0.0 0 16 - DL 18Feb25 0:00.10 [aiod3]
root 35 0.0 0.0 0 16 - DL 18Feb25 0:00.11 [aiod4]
root 795 0.0 0.0 15320 3980 - Is 18Feb25 0:00.37 /sbin/devd
root 3294 0.0 0.1 20272 8144 - Is 12:19 0:00.00 sshd: /usr /local/sbin/sshd [listener] 0 of 10-100 startups (sshd)
root 3729 0.0 0.1 23820 10708 - S 12:20 0:00.25 /usr/local /sbin/lighttpd -f /usr/local/etc/lighttpd_webgui/lighttpd.conf
root 4285 0.0 0.4 54036 29020 - Is 12:20 0:00.03 /usr/local /bin/php-cgi
root 4530 0.0 0.4 54036 29040 - Is 12:20 0:00.03 /usr/local /bin/php-cgi
root 5205 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 5764 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 6154 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 6316 0.0 0.4 61976 32544 - I 12:20 0:00.25 /usr/local /bin/php-cgi
root 6986 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 7404 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 8036 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 8288 0.0 0.4 61976 32504 - I 12:20 0:00.17 /usr/local /bin/php-cgi
root 8500 0.0 0.4 61976 32684 - I 12:20 0:00.25 /usr/local /bin/php-cgi
root 9007 0.0 0.4 57956 33788 - I 12:20 0:00.40 /usr/local /bin/php-cgi
root 9318 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 9649 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 9712 0.0 0.4 61976 32684 - I 12:20 0:00.21 /usr/local /bin/php-cgi
root 9805 0.0 0.4 57748 32544 - I 12:20 0:00.22 /usr/local /bin/php-cgi
root 9848 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 10371 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 11070 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 11765 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 12450 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 12739 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 13402 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 13441 0.0 0.4 62184 32876 - I 12:20 0:00.16 /usr/local /bin/php-cgi
root 14120 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 14418 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 14911 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 15115 0.0 0.4 57956 33404 - I 12:20 0:00.38 /usr/local /bin/php-cgi
root 15401 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 15420 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 16109 0.0 0.4 62052 35784 - I 12:20 0:00.20 /usr/local /bin/php-cgi
root 16289 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 16466 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 16840 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 17064 0.0 0.4 54036 29032 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 17184 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 17798 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 18383 0.0 0.4 57956 32432 - I 12:20 0:00.15 /usr/local /bin/php-cgi
root 18660 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 19388 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 19733 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 20263 0.0 0.4 54036 29052 - I 12:20 0:00.00 /usr/local /bin/php-cgi
root 21033 0.0 0.1 20808 8776 - Is 13:15 0:00.01 sshd-sessi on: root [priv] (sshd-session)
root 23394 0.0 0.1 24520 10576 - I Sun00 0:00.00 /usr/local /sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
root 23890 0.0 0.2 50008 15624 - Ss Sun00 3:24.51 /usr/local /sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
root 30869 0.0 0.0 13900 2468 - I Sun00 0:00.00 /usr/local /bin/flock -n -o /tmp/pkg_upgrade.progress /usr/local/etc/rc.firmware.subr
root 30946 0.0 0.0 14312 2656 - I Sun00 0:00.00 /bin/sh /u sr/local/etc/rc.firmware.subr
unbound 31176 0.0 0.7 103064 60216 - Ss 12:19 0:03.92 /usr/local /sbin/unbound -c /var/unbound/unbound.conf
root 31195 0.0 0.2 32176 17212 - Is 12:18 0:00.85 /usr/local /bin/python3 /usr/local/opnsense/service/configd.py (python3.11)
root 31978 0.0 0.5 110520 38408 - I 12:18 0:02.30 /usr/local /bin/python3 /usr/local/opnsense/service/configd.py console (python3.11)
root 33344 0.0 0.0 14312 2888 - I 12:16 0:00.00 /bin/sh /u sr/local/etc/rc.d/suricata stop
root 33593 0.0 0.0 13756 2280 - Ss 18Feb25 3:54.46 /usr/sbin/ powerd -b hadp -a hadp -n hadp
root 34845 0.0 0.0 14312 2712 - I Sun00 0:00.00 /bin/sh /u sr/local/opnsense/scripts/firmware/update.sh sync
root 34993 0.0 0.0 14312 2892 - I 12:20 0:00.01 /bin/sh /u sr/local/etc/rc.d/suricata stop
root 36361 0.0 0.0 13648 2088 - I 12:16 0:00.00 pwait 6400 0
root 37748 0.0 0.0 13648 2088 - I 12:20 0:00.00 pwait 6400 0
root 39237 0.0 0.1 20808 9048 - S 13:16 0:00.00 sshd-sessi on: root@pts/0 (sshd-session)
root 40470 0.0 0.0 13760 2208 - Is 12:19 0:00.00 daemon: /u sr/local/opnsense/scripts/unbound/logger.py[41031] (daemon)
root 42972 0.0 0.0 13736 2400 - Ss 18Feb25 1:02.30 /usr/local /sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog
root 44166 0.0 0.0 14312 2580 - I 12:16 0:00.00 /bin/sh /u sr/local/etc/rc.reboot
root 44177 0.0 0.0 14312 2608 - I 12:16 0:00.00 /bin/sh /u sr/local/etc/rc.syshook stop
root 48816 0.0 0.0 14312 2568 - I 12:16 0:00.00 /bin/sh /u sr/local/etc/rc.syshook.d/stop/80-freebsd
root 49401 0.0 0.0 14312 2720 - I 12:16 0:00.01 /bin/sh /u sr/local/etc/rc.freebsd stop
root 50862 0.0 0.2 27332 14660 - S 12:19 0:00.36 /usr/local /bin/python3 /usr/local/sbin/configctl -e -t 0.5 system event config_changed (py thon3.11)
root 51163 0.0 0.0 14312 2768 - I Sun00 0:00.00 /bin/sh /u sr/local/etc/rc.d/suricata stop
root 51173 0.0 0.0 14312 2960 - I 28Feb25 0:00.00 /bin/sh /u sr/local/etc/rc.d/suricata stop
root 51411 0.0 0.2 28356 14896 - S 12:19 0:00.40 /usr/local /bin/python3 /usr/local/opnsense/scripts/syslog/lockout_handler (python3.11)
root 51468 0.0 0.0 14312 2880 - I 12:19 0:00.01 /bin/sh /u sr/local/etc/rc.d/suricata restart
root 52671 0.0 0.0 14312 2636 - I 28Feb25 0:00.00 /bin/sh /u sr/local/etc/rc.reboot
root 53697 0.0 0.0 13648 2020 - I Sun00 0:00.00 pwait 6400 0
root 53715 0.0 0.0 13852 2520 - Is 12:19 0:00.03 /usr/sbin/ cron -s
root 54249 0.0 0.0 14312 2676 - I 28Feb25 0:00.00 /bin/sh /u sr/local/etc/rc.syshook stop
root 54435 0.0 0.0 13648 2152 - I 28Feb25 0:00.00 pwait 6400 0
root 56125 0.0 0.0 14312 2896 - I 12:19 0:00.00 /bin/sh /u sr/local/etc/rc.d/suricata restart
root 57454 0.0 0.0 13852 2528 - I 13:00 0:00.00 cron: runn ing job (cron)
root 58536 0.0 0.0 13648 2088 - I 12:19 0:00.00 pwait 6400 0
root 59713 0.0 0.4 46664 33244 - Is 13:00 0:00.28 /usr/local /bin/python3 /usr/local/opnsense/scripts/etpro_telemetry/send_heartbeat.py (pyth on3.11)
dhcpd 60566 0.0 0.2 28316 12576 - Is 12:19 0:00.02 /usr/local /sbin/dhcpd -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid igc1
root 61233 0.0 0.0 14076 2656 - Is 18Feb25 0:00.03 dhclient: system.syslog (dhclient)
root 64298 0.0 0.0 13648 2076 - SC 13:16 0:00.00 sleep 20
root 66020 0.0 0.0 14312 2648 - I 28Feb25 0:00.00 /bin/sh /u sr/local/etc/rc.syshook.d/stop/80-freebsd
root 66074 0.0 0.0 14312 2428 - I Sun00 0:00.00 /bin/sh /u sr/local/etc/rc.reboot
root 66202 0.0 0.0 14312 2772 - I 28Feb25 0:00.01 /bin/sh /u sr/local/etc/rc.freebsd stop
root 66496 0.0 0.0 14312 2484 - I Sun00 0:00.00 /bin/sh /u sr/local/etc/rc.syshook stop
root 67141 0.0 0.0 14312 2660 - I Sun00 0:00.00 /bin/sh /u sr/local/etc/rc.firmware -r 1500
root 67538 0.0 0.0 14076 2728 - Is 18Feb25 0:00.11 dhclient: igc0 [priv] (dhclient)
root 67567 0.0 0.0 14312 2684 - I Sun00 0:00.00 /bin/sh /u sr/local/opnsense/scripts/firmware/launcher.sh -s /usr/local/etc/rc.firmware.sub r -r 1500
root 69076 0.0 0.0 14312 2444 - I Sun00 0:00.00 /bin/sh /u sr/local/etc/rc.syshook.d/stop/80-freebsd
root 69189 0.0 0.0 14312 2588 - I Sun00 0:00.01 /bin/sh /u sr/local/etc/rc.freebsd stop
dhcpd 70214 0.0 0.1 25372 10372 - Is 12:19 0:00.02 /usr/local /sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igc1
root 70458 0.0 0.0 13760 2200 - Is 12:19 0:00.00 daemon: /u sr/local/opnsense/scripts/dhcp/prefixes.sh[71121] (daemon)
root 70766 0.0 0.0 14312 2896 - I 12:20 0:00.01 /bin/sh /u sr/local/etc/rc.d/suricata stop
root 71121 0.0 0.0 14312 2588 - S 12:19 0:00.02 /bin/sh /u sr/local/opnsense/scripts/dhcp/prefixes.sh
root 75025 0.0 0.0 13648 2084 - I 12:20 0:00.00 pwait 6400 0
root 75639 0.0 0.0 13852 2564 - I 19Feb25 0:00.00 cron: runn ing job (cron)
root 76399 0.0 0.0 14312 2872 - I 12:19 0:00.01 /bin/sh /u sr/local/etc/rc.d/suricata restart
root 81026 0.0 0.0 14312 2888 - I 12:19 0:00.00 /bin/sh /u sr/local/etc/rc.d/suricata restart
root 82667 0.0 0.0 13648 2088 - I 12:19 0:00.00 pwait 6400 0
_dhcp 85883 0.0 0.0 14080 2796 - SCs 18Feb25 0:02.34 dhclient: igc0 (dhclient)
root 87852 0.0 0.0 13924 2512 - SCs 18Feb25 4:08.74 /usr/sbin/ rtsold -aiu -p /var/run/rtsold.pid -A /var/etc/rtsold_script.sh -R /usr/local/op nsense/scripts/interfaces/rtsold_resolvconf.sh
root 89993 0.0 0.0 13924 2464 - Is 18Feb25 0:00.00 rtsold: rt sold.llflags (rtsold)
root 90406 0.0 0.0 13924 2468 - Is 18Feb25 0:00.00 rtsold: rt sold.script (rtsold)
root 91141 0.0 0.0 13924 2460 - Is 18Feb25 0:00.00 rtsold: rt sold.sendmsg (rtsold)
root 91985 0.0 0.0 13924 2472 - Ss 18Feb25 3:50.92 rtsold: sy stem.syslog (rtsold)
root 93842 0.0 0.1 24108 8132 - Ss 12:19 0:00.18 /usr/local /sbin/ntpd -g -c /var/etc/ntpd.conf
root 95919 0.0 0.0 13796 2364 - Is 18Feb25 0:00.02 /usr/local /sbin/dhcp6c -c /var/etc/dhcp6c.conf -p /var/run/dhcp6c.pid
root 97312 0.0 0.0 14044 2772 - Ss 12:19 0:00.96 /usr/local /sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root 62559 0.0 0.0 13780 2324 v0 Is+ 18Feb25 0:00.00 /usr/libex ec/getty Pc ttyv0
root 62928 0.0 0.0 13780 2324 v1 Is+ 18Feb25 0:00.00 /usr/libex ec/getty Pc ttyv1
root 63149 0.0 0.0 13780 2320 v2 Is+ 18Feb25 0:00.00 /usr/libex ec/getty Pc ttyv2
root 63321 0.0 0.0 13780 2324 v3 Is+ 18Feb25 0:00.00 /usr/libex ec/getty Pc ttyv3
root 64012 0.0 0.0 13780 2332 v4 Is+ 18Feb25 0:00.00 /usr/libex ec/getty Pc ttyv4
root 64254 0.0 0.0 13780 2328 v5 Is+ 18Feb25 0:00.00 /usr/libex ec/getty Pc ttyv5
root 64954 0.0 0.0 13780 2324 v6 Is+ 18Feb25 0:00.00 /usr/libex ec/getty Pc ttyv6
root 65258 0.0 0.0 13780 2332 v7 Is+ 18Feb25 0:00.00 /usr/libex ec/getty Pc ttyv7
root 39448 0.0 0.0 14312 2728 0 Ss 13:16 0:00.00 /bin/sh /u sr/local/sbin/opnsense-shell
root 60908 0.0 0.0 14788 3740 0 S 13:16 0:00.01 /bin/csh
root 64545 0.0 0.0 14380 3172 0 R+ 13:16 0:00.00 ps awwux
I also tried to do a reboot and it seems to hang here:
Quote>>> Invoking stop script 'beep'
>>> Invoking stop script 'freebsd'
Stopping suricata.
Waiting for PIDS: 64000
Pulled the plug and forced a reboot, now the health check can be run.
Quote***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 25.1.2 (amd64) at Sat Mar 8 13:38:50 EST 2025
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 25.1.2 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 25.1.2 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
os-etpro-telemetry 1.7_5
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .
pkg: pkg_checksum_hash_sha256_file(read failed): Input/output error
icu-76.1,1: checksum mismatch for /usr/local/share/icu/76.1/icudt76l.dat
Checking all packages.........
pkg: pkg_checksum_hash_sha256_file(read failed): Input/output error
py311-pandas-2.1.4,1: checksum mismatch for /usr/local/lib/python3.11/site-packages/pandas/tests/extension/base/missing.py
pkg: pkg_checksum_hash_sha256_file(read failed): Input/output error
py311-pandas-2.1.4,1: checksum mismatch for /usr/local/lib/python3.11/site-packages/pandas/tests/indexes/categorical/__pycache__/test_reindex.cpython-311.opt-1.pyc
py311-pandas-2.1.4,1: checksum mismatch for /usr/local/lib/python3.11/site-packages/pandas/tests/io/parser/test_c_parser_only.py
Checking all packages.....
pkg: pkg_checksum_hash_sha256_file(read failed): Input/output error
python311-3.11.11: checksum mismatch for /usr/local/lib/python3.11/distutils/command/__pycache__/sdist.cpython-311.pyc
pkg: pkg_checksum_hash_sha256_file(read failed): Input/output error
python311-3.11.11: checksum mismatch for /usr/local/lib/python3.11/distutils/command/__pycache__/upload.cpython-311.pyc
python311-3.11.11: checksum mismatch for /usr/local/lib/python3.11/test/__pycache__/test_eof.cpython-311.opt-1.pyc
pkg: pkg_checksum_hash_sha256_file(read failed): Input/output error
python311-3.11.11: checksum mismatch for /usr/local/lib/python3.11/test/__pycache__/test_tty.cpython-311.opt-1.pyc
pkg: pkg_checksum_hash_sha256_file(read failed): Input/output error
python311-3.11.11: checksum mismatch for /usr/local/lib/python3.11/test/libregrtest/__pycache__/runtests.cpython-311.opt-1.pyc
python311-3.11.11: checksum mismatch for /usr/local/lib/python3.11/test/libregrtest/__pycache__/save_env.cpython-311.opt-2.pyc
Checking all packages.... done
>>> Check for core packages consistency
Core package "opnsense" at 25.1.2 has 69 dependencies to check.
Checking packages: ...................................................................... done
***DONE***
kill -9 64000 would probably have done the trick.
So it's Suricata that is preventing a reboot for some reason. You might also try
- disable Suricata
- perform update
- enable Suricata
But more importantly:
pkg: pkg_checksum_hash_sha256_file(read failed): Input/output error
Your disk or SSD seems to be failing.
Quote from: Patrick M. Hausen on March 08, 2025, 08:10:07 PMkill -9 64000 would probably have done the trick.
So it's Suricata that is preventing a reboot for some reason. You might also try
- disable Suricata
- perform update
- enable Suricata
What commands should be run to fix this? Apologies as I'm not well versed even though I've used opnsense for a while, I rarely had issues until recently.
There is no way to fix this if it is caused by your failing drive. If it is caused by a bug in Suricata they need to fix it.
As a workaround you can always look for that process ID that the system is apperently waiting on and use
kill -9 <ID>to force terminate it and continue the reboot. In your case that was 64000 - different number each time.
Gotcha, I'm just unsure how to disable suricata to see if that's causing the issue next time. But I'll keep this in mind as I'm sure it'll hang again.
From the UI ... Services > Intrusion Detection ...
Or from the dashboard. Klick on the square "stop" symbol.
First and foremost, if your SSD is on its last leg, go to System - Configuration - Backups and click on Download Configuration --- unencrypted.
Then you can try the following:
Verify you're not running out of free space.
df -hT
Then try reinstalling these packages.
pkg install -f pkg icu python311 py311-pandas
When done, do another healthcheck and post here the results you got following these 3 steps.
/usr/local/opnsense/scripts/system/firmware/health.sh
Quoteroot@OPNsense:~ # df -hT
Filesystem Type Size Used Avail Capacity Mounted on
zroot/ROOT/default zfs 222G 1.4G 221G 1% /
devfs devfs 1.0K 0B 1.0K 0% /dev
/dev/gpt/efiboot0 msdosfs 260M 1.3M 259M 1% /boot/efi
zroot/tmp zfs 221G 964K 221G 0% /tmp
zroot/var/log zfs 221G 677M 221G 0% /var/log
zroot zfs 221G 96K 221G 0% /zroot
zroot/var/audit zfs 221G 96K 221G 0% /var/audit
zroot/var/mail zfs 221G 96K 221G 0% /var/mail
zroot/usr/ports zfs 221G 96K 221G 0% /usr/ports
zroot/usr/src zfs 221G 96K 221G 0% /usr/src
zroot/var/crash zfs 221G 96K 221G 0% /var/crash
zroot/home zfs 221G 96K 221G 0% /home
zroot/var/tmp zfs 221G 96K 221G 0% /var/tmp
devfs devfs 1.0K 0B 1.0K 0% /var/dhcpd/dev
devfs devfs 1.0K 0B 1.0K 0% /var/unbound/dev
/usr/local/lib/python3.11 nullfs 222G 1.4G 221G 1% /var/unbound/usr/local/lib/python3.11
/lib nullfs 222G 1.4G 221G 1% /var/unbound/lib
root@OPNsense:~ #
how do I free space, no idea what could be using so much
Quote***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 25.1.2 (amd64) at Sat Mar 8 19:20:29 EST 2025
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 25.1.2 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 25.1.2 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
os-etpro-telemetry 1.7_5
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" at 25.1.2 has 69 dependencies to check.
Checking packages: ...................................................................... done
***DONE***
Noticed intrusion detection service was stopped on the dashboard so I started it and re-ran the health check.
Quote***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 25.1.2 (amd64) at Sat Mar 8 19:28:15 EST 2025
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 25.1.2 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 25.1.2 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
os-etpro-telemetry 1.7_5
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .
isc-dhcp44-server-4.4.3P1_2: checksum mismatch for /usr/local/bin/omshell
Checking all packages.......
php83-google-api-php-client-2.4.0: missing file /usr/local/share/google-api-php-client/ve~dor/monolog/monolog/src/Monolog/Handler/PushoverHandler.php
Child process pid=3838 terminated abnormally: Segmentation fault
>>> Check for core packages consistency
Core package "opnsense" at 25.1.2 has 69 dependencies to check.
Checking packages: ...................................................................... done
***DONE***
Unsure why you have two different health checks there...
Reinstall the google and isc packages and do one more check
pkg install -f php83-google-api-php-client-2.4.0 isc-dhcp44-server-4.4.3P1_2
There were two because the first completed ok, then I looked on the web ui dashboard and saw intrusion detection service had stopped, so I started it and ran the health check again
Quote***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 25.1.2 (amd64) at Sat Mar 8 20:23:49 EST 2025
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 25.1.2 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 25.1.2 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
os-etpro-telemetry 1.7_5
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" at 25.1.2 has 69 dependencies to check.
Checking packages: ...................................................................... done
***DONE***
Looks like you're all set for now.
Thank you !
Back to not rebooting after an update, latest health check:
***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 25.1.5_4 (amd64) at Mon Apr 21 08:00:20 EDT 2025
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 25.1.5 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 25.1.5 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
os-etpro-telemetry 1.7_5
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" at 25.1.5_4 has 70 dependencies to check.
Checking packages: ........................
opnsense-25.1.5_4 version mismatch, expected 25.1.5_5
Checking packages: ............................................... done
***DONE***
Always start a root shell via ssh before an update. When that happens use ps to identify the hanging process preventing the reboot and kill it.
Suricata is the likely culprit there. Stop it before the next couple upgrades - unsure if 25.1.6 will require a reboot - and see if that solves the issue. Make sure Suricata is not running before doing the update.
Quote from: newsense on April 21, 2025, 07:58:53 PMSuricata is the likely culprit there. Stop it before the next couple upgrades - unsure if 25.1.6 will require a reboot - and see if that solves the issue. Make sure Suricata is not running before doing the update.
ok thank you. Sorry for the delayed response, i was traveling for work. Luckily the internet stayed up or the kiddos would have been restless!
Quote from: newsense on April 21, 2025, 07:58:53 PMSuricata is the likely culprit there. Stop it before the next couple upgrades - unsure if 25.1.6 will require a reboot - and see if that solves the issue. Make sure Suricata is not running before doing the update.
How do I stop it? If everything has to be done via ssh or other method I may as well just unplug it. I primarily browse from an iPad so ssh is a pain. Every update seems to revert it to this hang behavior and it's getting really tiring have to do a hard reboot or manual updates over and over.
See reply #14 from Patrick. The easiest is via the services widget on the dashboard IMO.
Only you can decide if Suricata is right for you...
Quote from: gillbot on March 09, 2025, 01:22:03 AMhow do I free space, no idea what could be using so much
Also, if you have not sorted your full disk already:
root@OPNsense:/ # du -shx /* | sort -rh | headthen you can drill down in the biggest offenders...
There was no full disk, I misread the indicator and I thought it was 1% free but it was only 1% full.
Using the widget from the dashboard doesn't stop anything. I get the spinning wheel icon but nothing ever happens. If you refresh the dashboard it's still active and repeated tries do nothing. Is there another way to stop the service? Even if it's cli at least I might be able to update and reboot.
I guess I could have looked at the df output...
With regards to stopping Suricata:
The widget doesn't work even if you try to stop the service BEFORE the update (which was the recommendation)?
I'm not totally surprised it doesn't work when it's blocking shutdown (it's probably deadlocked at that point).
Anyway, from the command line, you need to kill the process that's hanging reboot.
The command (given by Patrick earlier) is
kill -9 pidwhere pid is the [process] id of the hung process.
To figure out the pid, you have several options, mostly coming from the output of ps awwux
If you observe the output of the command in reply #7, it looks like suricata was hogging the CPU. That's a hint.
Then you have several pwait processes featuring the suricata pid on their command line. pwait waits for that process to terminate...
I suspect you had that many because of several attempts to upgrade/reboot.
pwait was probably invoked by the upgrade/reboot procedure to wait for all services to terminate gracefully. Using the id on the pwait command line is probably a safe bet. Careful if the output wrapped...
When you tried to reboot, the pid in question was also featured (reply #8).
If you strongly suspect suricata, I think you will also find the id in /var/run/suricata.pid (just looking at the suricata command line)...
In that instance the command was 'kill -9 64000'