SSD drive died - I disabled much from logging - what (still) fills my drive?

[oc]

Hello,

After about six months of having deployed my first "production" OPNsense, during a scheduled reboot (replacing the UPS with a larger one) the computer's BIOS indicated that S.M.A.R.T is deeming the drive as "about to fail". I have good backups, and was able to still access the old drive even after I had restored the setup onto the new one (I was initially missing a backup, that of ntopng, which together with zenarmor has its own configuration settings file) - so I can write if off as a good exercise of restoring OPNsense (albeit onto the same computer).

However after having gone through that experience (I never had any of my SSDs die on me, and I own some that I have used for well over ten years - including the one in the laptop I'm writing this message from), and after having more experience with OPNsense than in the beginning, and knowing that I won't spend much time with historical records (for instance I had set up Zenarmor to keep logs for 90 days, I now reduced that to three days), to extend the life of my next SSD I have decided to lighten logging considerably (I had already enabled /var/log RAM disk and /tmp RAM disk). I have disabled netflow, reduced zenarmor logging, ntopng logging, re-enabled the cron for trimming zroot - and started paying close attention to free space and S.M.A.R.T.

So here's my question: what continues to fill up my drive? The dashboard widget reports 0.7GB more used space in less than 24h. Some pointers on what to look for would be really helpful. My (new) mindset is to keep logging disabled as much as possible, and to only enable something if I need it for troubleshooting and/or forensics. But at the moment I'd like to know what's going on, what is writing to the drive - I would like this one to last MUCH longer. Note: I am not a FreeBSD file system expert, but am able to SSH/SCP in the filesystem as necessary.

Thank you :)

[Patrick M. Hausen]

Besorg dir eine Shell als root.

Code Select Expand

cd /
du -sk * | sort -rn | head
# größtes Directory steht oben
cd <directory>
# zurück zu Schritt 2 "du ..."


[oc]

Vielen Dank, Patrick...aber:

root@OC-OPNsense01:~ # du -sk * | sort -rn | head
du: No match.

[oc]

And...according to the dashboard widget, another 100MB has been added to disk since the last message I've posted.

[Monviech (Cedrik)]

ncdu

thats my tool of choice for finding large folders.

[Patrick M. Hausen]

@oc Du hast das

Code Select Expand

cd /

vorher vergessen. Nach deinem Prompt zu urteilen bist du immer noch im Home-Directory /root.

[oc]

@oc Du hast das

Code Select Expand

cd /

vorher vergessen. Nach deinem Prompt zu urteilen bist du immer noch im Home-Directory /root.

Danke Patrick!

root@OC-OPNsense01:/ # du -sk * | sort -rn | head
2866972 usr
1231306 var
174862  boot
11405   tmp
10852   lib
5799    conf
3372    sbin
2489    etc
929     bin
146     libexec

[Patrick M. Hausen]

Wie ich geschrieben hatte ...

Code Select Expand

cd usr
du ... # wieder von vorne
So findest du, was deine SSD füllt.

[oc]

Patrick: na endlich hat es geklappt - danke :)

The disk is filled by zenarmor/elastic. I'll have to further tweak logging. I definitely want the ZA "protection", and would like to be able to take a look at what's happening, but ideally it would store the fresh logs in RAM before flushing them (and I've bumped up space in Reporting Data/Memory Disk Size to 500MB, the maximum permitted).

Update: I have also temporarily stopped ntopng - that one also appears to fill the disk with the timeseries...