TL;DR: OPNSense crashed due to zero disk space because of /var/log/suricata
OPNSense does not follow linux standard partition mounting.
CONTEXT:I use Linux anywhere I can and the first thing I do is to have the partitions set individually like:
My current /home has been passed through 4 different distros because of the above, just mount it.
PROBLEM: I was having a problem with the company Windows laptop ( urghhhh ) and tried to check if it could be my setup even so my Linux laptop works like a dream.
I was still running the old version since the latest major release was a big release so no normal update among other things, and OPNSense UI was gone.
With some digging, I found the problem and I cannot open those files to see the root cause.
OPNSense team, pet yourselves on the back, 14 days running without disk space and without crashing on its own!!
(https://i.imgur.com/SiN2B5x.png)
With that being said, this is the new box.
Everything is under "/ " and that is not good.
The box that crashed had -18GB for / and /var/hdcpd/dev was 100% full
I did delete all the logs for /var/log/suricata but based on those logs date, the system has been running still for 14 days and once I rebooted it after the clean-up, oh boy, all hell broke loose.
I can imagine all the processes had no idea in which world they were in lmao
(https://i.imgur.com/ohcPjsE.png)
SOLUTIONThe installation should ask about /var the same way it asks about the swap.
It is better and wiser to have no logs than have no system.
HAPPY ENDING
This happened while WFH so hotspot for a few hours.
At least the installation is as easy as it gets and restoring a backup I had made things smooth BUT it didn't load the Firewall NAT rules for some reason.
I have Pi-Hole + Unbound Recursive DNS and I use the firewall to force anything name resolution via them only and block everything else ( DoT/DoH ), I got luck the 2020 post still exists so I could recreate those rules.
I was postponing this major release because why not?!
The new box I had around waiting for this is an i7, 32GB, 512GB NVMe.
I have been exploring the firewall a lot and Elasticsearch uses memory like a motherf.
Got a PCIe 4x Intel and can finally go past 1G (1-1.3G ) via IPoE instead of PPPoE
Last but not least, the box that crashed and it is a backup now running this latest version ( i5, 16GB, 256GB, RTK NIC ), this new widget UI was somewhat lag-ish.
After the reboot, they would be broken and no data to display while on this i7 box everything runs a lot smoother.
(https://i.imgur.com/Y23CtYd.png)
Quote from: hakuna on October 25, 2024, 02:11:10 PM
OPNSense does not follow linux standard partition mounting.
Why should it? It's not built on Linux.
And at least all my OPNsense installations do create separate datasets for the standard Unix directories:
root@opnsense:~ # df
Filesystem 1K-blocks Used Avail Capacity Mounted on
zroot/ROOT/24.7 233496696 1937352 231559344 1% /
devfs 1 0 1 0% /dev
zroot/var/mail 231559480 136 231559344 0% /var/mail
zroot/tmp 231560452 1108 231559344 0% /tmp
zroot/var/tmp 231559440 96 231559344 0% /var/tmp
zroot/var/log 231699192 139848 231559344 0% /var/log
zroot/var/crash 231559440 96 231559344 0% /var/crash
zroot/usr/home 231559440 96 231559344 0% /usr/home
zroot/var/audit 231559440 96 231559344 0% /var/audit
zroot 231559440 96 231559344 0% /zroot
devfs 1 0 1 0% /var/dhcpd/dev
devfs 1 0 1 0% /var/unbound/dev
/usr/local/lib/python3.11 233496696 1937352 231559344 1% /var/unbound/usr/local/lib/python3.11
/lib 233496696 1937352 231559344 1% /var/unbound/lib
You can set a quota on the /var/log dataset if you so desire.
Quote from: Patrick M. Hausen on October 25, 2024, 02:15:05 PM
Why should it? It's not built on Linux.
I mean, you understood what I meant :)
Quote
And at least all my OPNsense installations do create separate datasets for the standard Unix directories:
root@opnsense:~ # df
Filesystem 1K-blocks Used Avail Capacity Mounted on
zroot/ROOT/24.7 233496696 1937352 231559344 1% /
devfs 1 0 1 0% /dev
zroot/var/mail 231559480 136 231559344 0% /var/mail
zroot/tmp 231560452 1108 231559344 0% /tmp
zroot/var/tmp 231559440 96 231559344 0% /var/tmp
zroot/var/log 231699192 139848 231559344 0% /var/log
zroot/var/crash 231559440 96 231559344 0% /var/crash
zroot/usr/home 231559440 96 231559344 0% /usr/home
zroot/var/audit 231559440 96 231559344 0% /var/audit
zroot 231559440 96 231559344 0% /zroot
devfs 1 0 1 0% /var/dhcpd/dev
devfs 1 0 1 0% /var/unbound/dev
/usr/local/lib/python3.11 233496696 1937352 231559344 1% /var/unbound/usr/local/lib/python3.11
/lib 233496696 1937352 231559344 1% /var/unbound/lib
You can set a quota on the /var/log dataset if you so desire.
I was gonna say that is because you have it installed with ZFS but so was mine even so I had a single disk.
This new box I installed it with UFS instead.
I am running the latest 24.7.7 and mine does not look like that unless I need to install it again from scratch and manually set those partitions OR set a CRON since everything is running and I don't wanna do another full install, to run a script to check and delete big log files.