SSD drive died - I disabled much from logging - what (still) fills my drive?

[oc]

Hello,

After about six months of having deployed my first "production" OPNsense, during a scheduled reboot (replacing the UPS with a larger one) the computer's BIOS indicated that S.M.A.R.T is deeming the drive as "about to fail". I have good backups, and was able to still access the old drive even after I had restored the setup onto the new one (I was initially missing a backup, that of ntopng, which together with zenarmor has its own configuration settings file) - so I can write if off as a good exercise of restoring OPNsense (albeit onto the same computer).

However after having gone through that experience (I never had any of my SSDs die on me, and I own some that I have used for well over ten years - including the one in the laptop I'm writing this message from), and after having more experience with OPNsense than in the beginning, and knowing that I won't spend much time with historical records (for instance I had set up Zenarmor to keep logs for 90 days, I now reduced that to three days), to extend the life of my next SSD I have decided to lighten logging considerably (I had already enabled /var/log RAM disk and /tmp RAM disk). I have disabled netflow, reduced zenarmor logging, ntopng logging, re-enabled the cron for trimming zroot - and started paying close attention to free space and S.M.A.R.T.

So here's my question: what continues to fill up my drive? The dashboard widget reports 0.7GB more used space in less than 24h. Some pointers on what to look for would be really helpful. My (new) mindset is to keep logging disabled as much as possible, and to only enable something if I need it for troubleshooting and/or forensics. But at the moment I'd like to know what's going on, what is writing to the drive - I would like this one to last MUCH longer. Note: I am not a FreeBSD file system expert, but am able to SSH/SCP in the filesystem as necessary.

Thank you :)

[Patrick M. Hausen]

Besorg dir eine Shell als root.

Code Select Expand

cd /
du -sk * | sort -rn | head
# größtes Directory steht oben
cd <directory>
# zurück zu Schritt 2 "du ..."


[oc]

Vielen Dank, Patrick...aber:

root@OC-OPNsense01:~ # du -sk * | sort -rn | head
du: No match.

[oc]

And...according to the dashboard widget, another 100MB has been added to disk since the last message I've posted.

[Monviech (Cedrik)]

ncdu

thats my tool of choice for finding large folders.

[Patrick M. Hausen]

@oc Du hast das

Code Select Expand

cd /

vorher vergessen. Nach deinem Prompt zu urteilen bist du immer noch im Home-Directory /root.

[oc]

@oc Du hast das

Code Select Expand

cd /

vorher vergessen. Nach deinem Prompt zu urteilen bist du immer noch im Home-Directory /root.

Danke Patrick!

root@OC-OPNsense01:/ # du -sk * | sort -rn | head
2866972 usr
1231306 var
174862  boot
11405   tmp
10852   lib
5799    conf
3372    sbin
2489    etc
929     bin
146     libexec

[Patrick M. Hausen]

Wie ich geschrieben hatte ...

Code Select Expand

cd usr
du ... # wieder von vorne
So findest du, was deine SSD füllt.

[oc]

Patrick: na endlich hat es geklappt - danke :)

The disk is filled by zenarmor/elastic. I'll have to further tweak logging. I definitely want the ZA "protection", and would like to be able to take a look at what's happening, but ideally it would store the fresh logs in RAM before flushing them (and I've bumped up space in Reporting Data/Memory Disk Size to 500MB, the maximum permitted).

Update: I have also temporarily stopped ntopng - that one also appears to fill the disk with the timeseries...

[RamSense]

I have still an ungoing ticked with zenarmour about the rootcause of my extreme SSD wear and having to replace the SSD. Since I have replaced the SSD and I have removed Zenarmour (at least until a decent sollution from Zenarmour), no more problems here.

[Patrick M. Hausen]

Apologies for answering in German. I got confused, somehow.

One possible "solution" would be to simply buy a large (1T or so) high write endurance SSD. Yeah, easier said then done in the current state of the market. Back when those used to cost a hundred or so, I personally would have thrown money at the problem. Maybe try to get one used. Samsung 970, WD Black, ... that kind of devices.

[pfry]

[...]Maybe try to get one used. Samsung 970, WD Black, ... that kind of devices.

I can't say about used, and I didn't see form factor or interface mentioned (explicitly) offhand, but older low-end enterprise-type devices may currently be cheaper than high-end consumer (and have higher endurance - 1-3DWPD for 5 years). Unfortunately the price (in the U.S.) starts at around $250 for 2.5" NVME devices in the 1TB range. M.2 devices were never common, and seem to be made of unobtanium now. And of course there is the power consumption - 12-25W for 2.5", 8-11W for M.2 - which must be managed. (I'd recommend avoiding the real weenie roasters.)

(Of course, if I was to buy in this market, there'd be a surplus dump the next week.)

[nero355]

Maybe try to get one used. Samsung 970, WD Black, ... that kind of devices.

NOFI, but that's consumer crap and will probably have the same outcome over time as his old SSD that died... :'(

"This is the way" =>

older low-end enterprise-type devices

Or whatever Workstation/Enterprise SSD you can get for a decent price :)

This example :

(I never had any of my SSDs die on me, and I own some that I have used for well over ten years - including the one in the laptop I'm writing this message from)

Is probably one of those too : Good old Intel 320 Series 160/180 GB with MLC NAND or so ??

Ohh and by the way :
You could disable Hostwatch to reduce logging too! ;)

[Patrick M. Hausen]

I think a TBW of 600 for a 1T device is quite good and will easily last a couple of years. I'm using both the Samsung and the WD in my two NAS systems.

[nero355]

I think a TBW of 600 for a 1T device is quite good and will easily last a couple of years. I'm using both the Samsung and the WD in my two NAS systems.

For what exactly : Just the OS or something else ?

Don't get me wrong :
I am sure they will last a while, but I wouldn't be surprised if they would suddenly fail for whatever reason...
They all also have very basic power loss protection because of the lack of onboard capacitors that protect the RAM Cache (Which they luckily have unlike some other SSD models IIRC) and that can have a very bad effect on the integrity of your data !!