SSD drive died - I disabled much from logging - what (still) fills my drive?

[Patrick M. Hausen]

Just the OS or something else?

VM and jail storage in a mirror vdev/pool. Constantly written. One of the VMs is Elastiflow on Ubuntu.

Monitoring health with Scrutiny and Grafana. No issues with 50% of "Percentage used" in SMART output. Replaced with new ones shortly after going over 50%

For an OPNsense installation even with Elastic 1 TB is plenty. And at the end of the day you need to store the data *somewhere*. So even if you could use an external Elastic instance with Zenarmor in the free or home tier, that one would probably run on an SSD, too - as does my Elastiflow.

I'd categorise the two mentioned products as "prosumer". I have dozens of the Samsung 970 at home, friends' homes, at work, ... never lost one so far.

[oc]

Thank you ALL for your answers, especially on a Sunday. This is a wonderful community, which is commendable!

1. the almost-dead drive is a Samsung 256GB. I unfortunately didn't do a S.M.A.R.T on it prior to deploying it in my "first" opnsense attempt, but I think it came from a desktop or laptop (a friend gave it to me years ago, he used to work in a company that scrapped all hardware after 3 years, and I had the disk in storage for a few years). So conceivably it might have been "tired" when I deployed it, but I highly doubt that it was written onto at high volumes - those companies lock down workstations and all files are hosted on servers (or the cloud, nowadays). However it's likely that I broke it by letting it getting constantly hammered by opnsense and it's add-ons :)

2. I have an 128GB Intel 520 Series, which appears to have enjoyed Industry-leading 5-year warranty and historically low failure/return rates when released (in 2012, so it's also not a spring chicken). But I did run a S.M.A.R.T on it, and it reports 96% health and ~10,000 hours of uptime (which is a lot, but perhaps no big deal). I did install opnsense on it temporarily, but decided to go with a bigger drive (512GB, brandnew, but a crappy brand - I am monitoring it, and have all backups to date). But I am considering deploying that one, if I see that the "crappy 512GB" starts going downhill fast. I will also keep my eyes open for an enterprise-grade SSD...it's getting harder now, with everyone going to the cloud...in the past I literally had access to mountains of untouched never-to-be-deployed enterprise-grade equipment headed for the bin.

3. at work we log about 25TB/day (not a typo), which we're now in the process of consolidating to elastic - so I'm quite aware of the challenges when ingesting such high log volumes - and which was also one of the reasons I might have gone a bit overboard when enabling too much logging on my opnsense - it was regularly logging over 1mil entries/day. I have now redone the ZA configuration, and am now protecting/logging individual VLANs(about 8, for segmentation)/NICs(I have 5 of them, a quad and the onboard one) instead of latching ZA onto the trunk port. This allowed me to avoid assessing&logging entries from the most verbose host on my network - which due to the very high volume would have been very hard to qualitatively analyze anyway. I can however re-enable it with one click if necessary.

4. getting back to the elastic architecture, IMO it would have been great if I could have assigned some GBs as a ram disk, and for elastic to log to its heart's content, entries to be pruned when they exceed the assigned RAM quantity. I could have lived (privately, unlike at work where it's highly regulated) with potentially missing out on older entries - but it would have been great for the longevity of my OS drive. Or another decent option (perhaps that is possible) would be if I could add another drive, and to redirect all elastic logs to it - it's much less bad if I lose my logs, but the FW continues with all its tasks (again, at home that would be acceptable, at work we have everything set to stop forwarding if it can be logged). I know I can log to an external host, but I would like to keep the "firewall service" to one host if possible. I used to rely on one of the major enterprise firewalls (licensed, promotionally) prior to deploying opnsense, so I am mindful of the form factor. But I really love opnsense so I want to carry on <3

5. is disabling hostwatch still a thing? I noticed it was a considered an issue in the past, but Frank suggested it's been fixed in Jan'26 - I am using the most recent version. I have about 30-40 devices altogether, and aside from the mobile devices which tend to use random MACs, all else is supposed to be very stable over time. How can I check how much hostwatch has been logging so far?

6. in conclusion, I consider this experience a great learning opportunity. I encountered a couple of challenges when restoring opnsense (external to opnsense itself), which I have found solutions for - so right now it would take me under 1h to restore, so no big deal. But I am concerned about the FW dying suddenly because I'm sometimes away from home for longer (e.g., during holidays), and I am using it for many reasons, including as a VPN gateway to keep all my travel traffic curated - and I offer friends the same services, so there are several people relying on this 'safe perimeter security', and I don't want them to be disappointed. Also, it is my way to monitor a lot of things at home. It feels like a very no-enterprise way to introduce such a frail SPOF into the equation...my2c

[harshw]

Apologies for answering in German. I got confused, somehow.

One possible "solution" would be to simply buy a large (1T or so) high write endurance SSD. Yeah, easier said then done in the current state of the market. Back when those used to cost a hundred or so, I personally would have thrown money at the problem. Maybe try to get one used. Samsung 970, WD Black, ... that kind of devices.

Another solution is to get a Micron MX300 Crucial MX500 (and specifically that model) drive - even a 512GB one will do. Then use Micron's Executive software to over-provision the drive by 50% i.e. it's a 256GB drive for all intents and purposes. The drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM. You should check if TRIM is enabled for your filesystem - are you using ZFS or UFS?

[oc]

Another solution is to get a Micron MX300 (and specifically that model) drive - even a 512GB one will do. Then use Micron's Executive software to over-provision the drive by 50% i.e. it's a 256GB drive for all intents and purposes. The drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM. You should check if TRIM is enabled for your filesystem - are you using ZFS or UFS?

Thank you - that one is indeed more reasonably priced.

"drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM" - does that mean that it can take 3-4 times its provisioned capacity (3-4x256GB)? Or am I misunderstanding?

"You should check if TRIM is enabled for your filesystem - are you using ZFS or UFS?" - I use ZFS (for snapshotting), and have enabled trim in cron:

Is that what you meant? Thanks :)

[harshw]

Another solution is to get a Micron MX300 (and specifically that model) drive - even a 512GB one will do. Then use Micron's Executive software to over-provision the drive by 50% i.e. it's a 256GB drive for all intents and purposes. The drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM. You should check if TRIM is enabled for your filesystem - are you using ZFS or UFS?

Thank you - that one is indeed more reasonably priced.

"drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM" - does that mean that it can take 3-4 times its provisioned capacity (3-4x256GB)? Or am I misunderstanding?

"You should check if TRIM is enabled for your filesystem - are you using ZFS or UFS?" - I use ZFS (for snapshotting), and have enabled trim in cron:

Is that what you meant? Thanks :)

My apologies for the last post, I meant the Crucial MX500 and not the MX300 

You have TRIM enabled so unless you had a drive that didn't support TRIM, maybe it died because it was crappy?

Good discussion here: https://forums.servethehome.com/index.php?threads/can-a-1-dwpd-ssd-become-a-3-dwpd-just-with-overprovisioning.45641/

You can calculate Write Amplification Factor (WAF) from the MX500 SMART data. I would encourage you to get one of these, over-provision by 33% ~ 50% and watch the calculated WAF with your specific workload. As noted in the discussion linked to above, I run either Micron 5200/5300MAX or Crucial MX500 OP'ed by at least 33% for all my OPNsense, Ubiquiti, ESXi system drives etc. Even when the underlying OS didn't support TRIM (ubiquiti crap for example) I've seen these drives keep the WAF super low. You can usually get HPE branded Micron 5300 MAX drives for cheap. I use them for my OPNsense although it is tough to update HPE SSDs or even obtain firmware for them

[pfry]

[...]
3. at work we log about 25TB/day (not a typo), which we're now in the process of consolidating to elastic - so I'm quite aware of the challenges when ingesting such high log volumes - and which was also one of the reasons I might have gone a bit overboard when enabling too much logging on my opnsense - it was regularly logging over 1mil entries/day. [...]

I was going to ask about the environment. I do not use an IPS; I have most logging enabled on pf (all inbound, NAT and firewall only outbound). I have static IPs and a couple lightly-used servers, so ~500 sessions active at any given time (with some spikes if the data scrapers go nuts). OPNsense has written 2.44TB over ~600 days active, or about a day's worth on the SSD (according to SMART). Apparently ZFS compression works really well on OPNsense.

25TB/d kinda sounds like write-only storage territory. I like to keep logs for a fairly long time (currently "Maximum preserved files" = 200), but it would take some effort to read anything useful from a 5PB store. Is that 25TB distributed?

[...]You have TRIM enabled so unless you had a drive that didn't support TRIM[...]

TRIM is most useful when the device is full. Hmm... Given the write level, it probably was...

You generally don't need to do anything fancy to get (at least some of) the endurance benefits of overprovisioning - just don't fill your device. Explicit device-level overprovisioning will generally help write performance, though. I partitioned the SSDs in my old (current) primary server for 180GB on 256GB devices, just to be paranoid. This was in 2014, by the way. (This is why I generally look at endurance over 10 years, not 3 or 5.)

I wonder how consumer-type PSLC caches behave in a write-heavy environment (I haven't looked into it, as I don't have any).

[nero355]

I'd categorise the two mentioned products as "prosumer".

I have dozens of the Samsung 970 at home, friends' homes, at work, ... never lost one so far.

- Samsung 840 Pro/850 Pro/970 Pro/980 Pro
- WD "Black" Series
=
All "Gaming SSDs" in my book, sorry! :)

I have an 128GB Intel 520 Series, which appears to have enjoyed Industry-leading 5-year warranty and historically low failure/return rates when released (in 2012, so it's also not a spring chicken).

But I did run a S.M.A.R.T on it, and it reports 96% health and ~10,000 hours of uptime (which is a lot, but perhaps no big deal).

Despite not being "Real Intel SSDs" in my book they were seriously BULLETPROOF compared to the Samsung 850 Pro Series in the past so IMHO you can trust that SSD with your life :)

Is disabling hostwatch still a thing?

I think I have seen a topic or two where it was, but a very recent one made it look like the service is well behaving now...

Whatever the status is : If you don't need it, DISABLE IT! ;)

the mobile devices which tend to use random MACs

Force your Clients to Disable that nonsense...

Another solution is to get a Micron MX300 Crucial MX500 (and specifically that model) drive - even a 512GB one will do.

Then use Micron's Executive software to over-provision the drive by 50% i.e. it's a 256GB drive for all intents and purposes.
The drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM.
You should check if TRIM is enabled for your filesystem - are you using ZFS or UFS?

The Crucial MX500 Series is one of my favorites too :)

I also have the original and even better M500 960 GB model in use!
There was also the M500 DC variant which was even better Power Loss Protection-wise but I am not sure if there is a MX500 DC too...

"drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM" - does that mean that it can take 3-4 times its provisioned capacity (3-4x256GB)? Or am I misunderstanding?

I am guessing it can, especially if there is some kind of Garbage Collection feature in it's firmware : Those Intel 520 models were really good at it!

_{I can't remember if the MX500 has this feature... Too long ago for my memory at the moment... d0h!}

Good discussion here: https://forums.servethehome.com/index.php?threads/can-a-1-dwpd-ssd-become-a-3-dwpd-just-with-overprovisioning.45641/

I run either Micron 5200/5300MAX or Crucial MX500 OP'ed by at least 33% for all my OPNsense, Ubiquiti, ESXi system drives etc.

NICE! :)

Even when the underlying OS didn't support TRIM (ubiquiti crap for example)

Isn't that a matter of editing fstab correctly ?!

TRIM is most useful when the device is full.

I disagree : It's ALWAYS useful!

It's applied to keep the performance of the SSD at optimal level after all :)