While updating from 25.10.1_2 to the latest release the update process stopped while updating suricata with "/var/log/suricata: filesystem full", or something along this line. Sorry, I forgot again to save the update log file. Anyway, as it turns out, the directory /var/log/suricata didn't even exist. But /var/log/resolver was eating up over 90% of the file system space, leaving no free space. So, I deleted all the crap, restarted the update and everything looks alright now.
I would like to kindly ask that the update procedure gets augmented with sanity checks to prevent out of memory conditions and possibly more stuff. You guys probably have a lot more ideas than me.
The more important bit to me however, is that unbound can fill up the entire /var/log filesystem because there doesn't seem to be proper log file rotation. For now I have deactivated query logging to gain some time before consuming all available disk space again.
And there's one more thing: While investigating I saw that the /var/log filesystem is mounted twice:
root@firewall:~ # df -h
Filesystem Size Used Avail Capacity Mounted on
zroot/ROOT/default 221G 1.6G 219G 1% /
devfs 1.0K 0B 1.0K 0% /dev
/dev/gpt/efifs 256M 864K 255M 0% /boot/efi
zroot 219G 96K 219G 0% /zroot
zroot/tmp 219G 96K 219G 0% /tmp
zroot/var/audit 219G 96K 219G 0% /var/audit
zroot/usr/home 219G 96K 219G 0% /usr/home
zroot/var/log 220G 311M 219G 0% /var/log
zroot/var/crash 219G 96K 219G 0% /var/crash
zroot/var/tmp 219G 96K 219G 0% /var/tmp
zroot/usr/src 219G 96K 219G 0% /usr/src
zroot/var/mail 219G 136K 219G 0% /var/mail
zroot/usr/ports 219G 96K 219G 0% /usr/ports
tmpfs 2.0G 2.6M 2.0G 0% /var/log
tmpfs 1.2G 3.0M 1.2G 0% /tmp
tmpfs 1.2G 152K 1.2G 0% /var/lib/php/tmp
devfs 1.0K 0B 1.0K 0% /var/dhcpd/dev
devfs 1.0K 0B 1.0K 0% /var/unbound/dev
/usr/local/lib/python3.11 221G 1.6G 219G 1% /var/unbound/usr/local/lib/python3.11
/lib 221G 1.6G 219G 1% /var/unbound/lib
root@firewall:~ #
I know what it means and I know that it works fine, but why is /var/log not unmounted before mounting tmpfs in its place?
Do you have the update log?
# opnsense-update -g
Cheers,
Franco
No, unfortunately not. When something fails I'm totally focussed on restoring functionality. I'll try to do better next time. Maybe it'll help to keep a history of logs of opnsense-update?
If you run the command on the shell you can see if it's still stored ;)
Cheers,
Franco