Any way to reduce drive wear?

[drosophila]

I've set up OPNsense-nano in two ways and both have issues regarding drive wear:

1) using an external USB enclosure.
- the issue with this is that while disk writes don't matter, this being the root fs doesn't allow the drive to spin down, even on an idle system. I'd have expected that, it being nano, all access is to the ramdrives, while OS data / programs will essentially be cached. The system has barely any configuration and no traffic whatsoever, but still the drive never powers down, even after a day of just sitting there. When the drive is merely attached to the system, but not mounted, it does power dosn after 30 minutes or so, which would be fine.
Problem: this being a mechanical drive (laptop drive), actual spinning time will likely increase wear and increase power usage, both of which I wish to avoid by it spinning down once booted.

My preferred resolution would be to just cache everything required during normal operation and not touch the drive anymore, allowing it to sleep until something unusual happens.

2) using a USB attached thumbdrive.
- this configuration is efficient until there is an upgrade. The way the upgrades are done, the drive keeps being hit with small bits of writes, which, besides dragging performance down to abysmal levels, forces the drive to repeatedly erase a section of flash, write a single byte, erase it again, write the next byte, for hours on end (literally). Combined with the nonexistent wear levelling and trim support of these things, I expect this to wear down the device in no time. The drives have no internal cache whatsoever that could be enabled by camcontrol or smartctl (they don't even speak SMART).

My preferred solution would be to have a write cache of, say, 100MB, to and from which all sequential modifications are done, and only flush that to the drive about once per minute, give or take.

I haven't found any way to do option 2 on BSD (maybe with ZFS), and for 1) I don't know enough about both BSD and OPNsense to evaluate at all. Thus far, the only option would be to use a recent SATA SSD, but the external / USB solution is so elegant and flexible in all other respects that it would be a pity to ditch it, plus the SSD would be horrendous overkill.

[meyergru]

The problem with 2) is probably that over USB, writes are always small chunks, even if you try to write larger blocks. If there is no RAM cache on the internal SSD, you will get read-modify-write cycles even when you write in larger blocks. Having decent "pro" types of SSDs is essential for ZFS applications.

The default ZFS flush delay was changed earlier on in OpnSense, such that when you install it now, it will already be at reasonable values to avoid excessive wear on SSDs. You could modify the values beyond the defaults, however, I doubt that would actually change much. The discussion about this was here: https://forum.opnsense.org/index.php?topic=37901.15

[drosophila]

Thanks for the consideration! I get about 14MB/s writing to the thumbdrive with dd, whereas the update process reports around 0.1MB/s. I tried the dd with bs=64k, but thanks to your hint I looked it up and the chunk size can be as low as 8, 16, 32, or 64 bytes for "full speed" drives, and up to 512 with "high-speed" devices. Even if read-modify-write cannot be avoided, it would perhaps suffice if the writes were bunched, so small chunks might still be OK if it were aggregated. Of course, this only works if the write cache also serves as read cache so that pending changes would not have to be flushed before a read can commence. IDK if this would be normal behavior if such a cache were to actually exist.
I only found mention of the "logging/nologging" parameter for ufs mounts, but that looks like it'll just add more writes by first writing what it plans to write, and then actually writing that, and then erasing that from the log. Sounds similar to the journalling of ext4 or the ZIL in ZFS, but with the latter I could possibly trick it into using a dedicated ZIL drive that itself is a RAM disk. Obviously this would be a really bad idea for a file server, but as a drive cache might be acceptable, but still wouldn't mean the writes actually do get bunched. Plus, using ZFS on a puny 4/8GB USB thumb also seems like a bad idea, and if I went for a proper internal SSD, I might just as well forego all these considerations because that does caching and has tons of write cycles plus heaps of spare sectors, TRIM, etc., anyway. So currently I'm using UFS, but there isn't even a choice anyway with -nano. :)

[meyergru]

First, I would not use anything other than ZFS with OpnSense - and also, whatever caching you do on the host side does not help when the resulting writes are chunked and not being cached in the SSD (because of the small USB writes).

Therefore, it does not matter if you use an additional ZFS write cache or just change the VFS parameters to use more RAM and flush less often.

When you read the thread I linked, there was an important hint there: ZFS never rewrites any blocks - it always writes new ones (after all, it is a COW filesystem). Thus, read cache does not help you in any meaningful way.

No matter how you look at it: What matters most is a drive that has write cache RAM in order to be able to group small writes together such that they can be written as one large erase block. Thus, the price would be that of a "pro" drive anyway and if you can connect this directly without USB, it would definitely be preferable. I know of no small "pro" type SSD drives, anyway.

[drosophila]

I've just finished reading that thread, and ZFS writing metatata constantly indeed was new to me. However, (I edited it in but it overlapped), with the -nano images you're always on UFS, which I find a reasonable choice given the type and size of these drives, and especially if ZFS won't even help in that department, anyway.
Right, so as option 2) is out unless I delay updates to once per year which obviously is a bad idea, I'd need to see if option 1) can somehow be made to work, still -nano and thus UFS.

In the linked thread there were lovely graphs that were made with "CheckMK", but sadly markus.tobatz didn't say how to install that on OPNsense, but in the end they're not much different from what iostat does.