ZFS trim and scrub

[kevindd992002]

I have OPNsense 24.7.11_2-amd64 installed in a Proxmox VM and am wondering if it has ZFS trim and scrub scheduled by default? Or do I still need to add both tasks as a cron job?

If they're not enabled by default, why? Is there a reason we don't want these two?

[mooh]

Unless you have attached a ZFS pool to the VM, the machine hosting the zpool should do the trimming and scrubbing.

[kevindd992002]

Unless you have attached a ZFS pool to the VM, the machine hosting the zpool should do the trimming and scrubbing.

I do have another pool inside the VM. So does that mean I have to create a cron job for trimming and scrubbing? What is the difference between the trim cron job and the autotrim property?

[mooh]

Yes, I you have a ZFS pool inside your VM, the VM needs to do the trimming and scrubbing. If this pools happens to be mapped to a file in a ZFS pool of the host, the scrubbing is not needed since it is done by the host. Actually, scrubbing a virtual ZFS pool doesn't make sense.
Furthermore, if you have a zfs inside a zpool on the guest inside a zfs of a zpool on the host, performance will be suboptimal. Other than for training purposes, you'll be better off to mount a host ZFS to your VM directly.

Setting autotrim of a zpool to true causes ZFS to do trimming automatically, as part of the normal operation of the file system, which may happen to be at inconvenient times. If that setting is off, you may run "zpool trim ..." at your discretion.

[kevindd992002]

Well, initially I had a non-ZFS FS on my opnsense VM in the ZFS pve host. Then my opnsense VM got corrupted. Did some research and a couple of people suggested that the best way to go is to reinstall opnsense on the VM with a ZFS filesystem as well, and that's what I did. Why is this suboptimal? How do I go about "mounting a host ZFS to the opnsense VM directly"?

Here's that past topic if you're interested on why I did what I did: https://forum.opnsense.org/index.php?topic=42099.msg207468#msg207468

[meyergru]

Yes, I you have a ZFS pool inside your VM, the VM needs to do the trimming and scrubbing. If this pools happens to be mapped to a file in a ZFS pool of the host, the scrubbing is not needed since it is done by the host. Actually, scrubbing a virtual ZFS pool doesn't make sense.

Sorry, but this is plain wrong or at least ambiguous.. The trimming and scrubbing taking place on the VM host works on its own ZFS pools, not on the OpnSense pools.

Within those pools, you have virtual disks, which are usually ZFS volumes. What the VM OS - in this case OpnSense - does with that, is its own business and the VM host does not look into it. OpnSense could as well be installed on UFS, which Proxmox would not even understand. On the other hand, you could have LVM filesystems on Proxmox and ZFS on OpnSense. You see: Even if both OpnSense and Proxmox use ZFS pools, they are independent of one another. If this wasn't clear already, just look at the names: Proxmox names its root zpool "rpool", whereas OpnSense names its pool "zroot".

Thus taking care of filesystems (or zpools) in an OpnSense VM is just as much its business as without virtualisation. Think of logical errors in the VM itself, which are not manifested on the underlying hardware.

For the default zroot pool, autotrim is on. While you could scrub, it would only be useful if you had multiple disks. If OpnSense is operated as VM, it would be more useful if the underlying Proxmox storage was redundant.

[kevindd992002]

Yes, I you have a ZFS pool inside your VM, the VM needs to do the trimming and scrubbing. If this pools happens to be mapped to a file in a ZFS pool of the host, the scrubbing is not needed since it is done by the host. Actually, scrubbing a virtual ZFS pool doesn't make sense.

Sorry, but this is plain wrong or at least ambiguous.. The trimming and scrubbing taking place on the VM host works on its own ZFS pools, not on the OpnSense pools.

Within those pools, you have virtual disks, which are usually ZFS volumes. What the VM OS - in this case OpnSense - does with that, is its own business and the VM host does not look into it. OpnSense could as well be installed on UFS, which Proxmox would not even understand. On the other hand, you could have LVM filesystems on Proxmox and ZFS on OpnSense. You see: Even if both OpnSense and Proxmox use ZFS pools, they are independent of one another. If this wasn't clear already, just look at the names: Proxmox names its root zpool "rpool", whereas OpnSense names its pool "zroot".

Thus taking care of filesystems (or zpools) in an OpnSense VM is just as much its business as without virtualisation. Think of logical errors in the VM itself, which are not manifested on the underlying hardware.

For the default zroot pool, autotrim is on. While you could scrub, it would only be useful if you had multiple disks. If OpnSense is operated as VM, it would be more useful if the underlying Proxmox storage was redundant.

There you go. Thanks for the detailed explanation.

Only thing I'm wondering about is why autotrim in my opnsense zpool isn't enabled?

[meyergru]

I do not know if autotrim has always been on with OpnSense. Maybe your initial installation did have it disabled?

Did you check with "zpool get all | fgrep autotrim"? All my instances have it enabled.

[mooh]

I guess I should have written something like "Only the machine hosting the physical devices should do the trimming and scrubbing" to be more clear than "scrubbing a virtual ZFS pool doesn't make sense".

And for completeness: It's only a waste of resources to do integrity checks on data that lives inside a file system with integrity checks. If data integrity is of utmost importance and one intends to move the VM to host machines that don't have such file systems, it is still advisable to do the scrubbing inside the VM.

[kevindd992002]

I do not know if autotrim has always been on with OpnSense. Maybe your initial installation did have it disabled?

Did you check with "zpool get all | fgrep autotrim"? All my instances have it enabled.

Yes.

Code Select Expand

root@OPNsense:~ # zpool get all | fgrep autotrim
zroot  autotrim                       off                            default

I forgot the exact version of my initial install but it's pretty recent, like maybe June 2024 version.

[dinguz]

I recently reinstalled a system with the 24.7 image, and it was still set to disabled in that release.
IIRC, the historical reason for having autotrim disabled by default was that it allowed system administrators to schedule it more effectively, avoiding potential random performance drops caused by automatic triggering.

[meyergru]

Maybe I remember that wrong and have set autotrim anywhere myself...

[EricPerl]

It's in your HOWTO ;-)

That is, if the ZFS "autotrim" feature is not set manually. You can either set this via the OpnSense CLI with "zpool set autotrim=on zroot" or, better, add a daily cron job to to this (System: Settings: Cron) with "zroot" as parameter.
You can trim your zpool once via CLI with "zpool trim zroot".

[meyergru]

Ah, there you go, I am getting old. Nice HowTo, BTW...

I do this because the ZFS volume will fill up to 100% after a while and thus VM backups take very long.

[Taomyn]

Was looking at this thread with interest as also having OPNsense running under Proxmox, so I went to check the auto-trim option - it's disabled. So I ran it manually for good measure.

I then ran a status check on the pool and it came back with

Code Select Expand

root@bart:~ # zpool status zroot
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:

NAME        STATE    READ WRITE CKSUM
zroot      ONLINE      0    0    0
  da0p4    ONLINE      0    0    0

errors: No known data errors


Should I perform the upgrade? I've seen this on Proxmox a couple of times and did the upgrade without issues.