[Solved] High disk writes

[franco]

PS: On my current install it's about 1.2 MB per sync, but I know we've seen more before.

[Patrick M. Hausen]

On my OPNsense with the transaction group timeout restored to 5 seconds it's 600-800k every flush. So that's matching with the 120k/s some of the other posters observe.

I wasn't aware that it's just metadata, so thanks. Will be a point to raise at the next ZFS production users call.

I think my argument about not using ZFS for virtual disks still holds.

Kind regards,
Patrick

[franco]

I'm unsure how the disk size plays into this. Bigger disk may mean more metadata.

20GB writes per day was observable with users...


Cheers,
Franco

[tverweij]

What needs to be flushed to disk will be flushed to disk. And ZFS never does in-place overwrites. So if you write every 5 seconds or 18 times the transaction groups every 90 seconds ...

If @tverweij running virtualised instances is indeed using ZFS that specifically is a bad idea. Due to its copy on write nature you cannot thin provision virtual disks (well you can, but it doesn't make sense) because ZFS will eventually write every single disk block and so blow up the disk to its configured maximum size.

For virtual disks it's much better to use UFS and manage snapshots and backups at the hypervisor host level.

That is a good word of advice.
Maybe a good idea to add this advice to the installation docs?

Because I chose ZFS because the docs say:
"Install (UFS|ZFS) - Choose UFS or ZFS filesystem. ZFS is in most cases the best option as it is the most reliable option, but it does require enough capacity (a couple of gigabytes at least)."

[Mars79]

Hi Franco,

according to my understanding of ZFS it won't make much of a difference if you flush transaction groups every 5 or every 90 seconds. What needs to be flushed to disk will be flushed to disk. And ZFS never does in-place overwrites. So if you write every 5 seconds or 18 times the transaction groups every 90 seconds ...

If @tverweij running virtualised instances is indeed using ZFS that specifically is a bad idea. Due to its copy on write nature you cannot thin provision virtual disks (well you can, but it doesn't make sense) because ZFS will eventually write every single disk block and so blow up the disk to its configured maximum size.

For virtual disks it's much better to use UFS and manage snapshots and backups at the hypervisor host level.

HTH,
Patrick

When running a virtualized OPNsense on ZFS, the issue of blowing up disks can be solved by turning on "discard" in the Proxmox VM settings and setting up a daily ZFS trim cron job within OPNsense.