I have OPNsense 24.7.11_2-amd64 installed in a Proxmox VM and am wondering if it has ZFS trim and scrub scheduled by default? Or do I still need to add both tasks as a cron job?
If they're not enabled by default, why? Is there a reason we don't want these two?
Unless you have attached a ZFS pool to the VM, the machine hosting the zpool should do the trimming and scrubbing.
Quote from: mooh on January 02, 2025, 04:12:00 PMUnless you have attached a ZFS pool to the VM, the machine hosting the zpool should do the trimming and scrubbing.
I do have another pool inside the VM. So does that mean I have to create a cron job for trimming and scrubbing? What is the difference between the trim cron job and the autotrim property?
Yes, I you have a ZFS pool inside your VM, the VM needs to do the trimming and scrubbing. If this pools happens to be mapped to a file in a ZFS pool of the host, the scrubbing is not needed since it is done by the host. Actually, scrubbing a virtual ZFS pool doesn't make sense.
Furthermore, if you have a zfs inside a zpool on the guest inside a zfs of a zpool on the host, performance will be suboptimal. Other than for training purposes, you'll be better off to mount a host ZFS to your VM directly.
Setting autotrim of a zpool to true causes ZFS to do trimming automatically, as part of the normal operation of the file system, which may happen to be at inconvenient times. If that setting is off, you may run "zpool trim ..." at your discretion.
Well, initially I had a non-ZFS FS on my opnsense VM in the ZFS pve host. Then my opnsense VM got corrupted. Did some research and a couple of people suggested that the best way to go is to reinstall opnsense on the VM with a ZFS filesystem as well, and that's what I did. Why is this suboptimal? How do I go about "mounting a host ZFS to the opnsense VM directly"?
Here's that past topic if you're interested on why I did what I did: https://forum.opnsense.org/index.php?topic=42099.msg207468#msg207468
Quote from: mooh on January 02, 2025, 06:31:49 PMYes, I you have a ZFS pool inside your VM, the VM needs to do the trimming and scrubbing. If this pools happens to be mapped to a file in a ZFS pool of the host, the scrubbing is not needed since it is done by the host. Actually, scrubbing a virtual ZFS pool doesn't make sense.
Sorry, but this is plain wrong or at least ambiguous.. The trimming and scrubbing taking place on the VM host works on its own ZFS pools, not on the OpnSense pools.
Within those pools, you have virtual disks, which are usually ZFS volumes. What the VM OS - in this case OpnSense - does with that, is its own business and the VM host does not look into it. OpnSense could as well be installed on UFS, which Proxmox would not even understand. On the other hand, you could have LVM filesystems on Proxmox and ZFS on OpnSense. You see: Even if both OpnSense and Proxmox use ZFS pools, they are independent of one another. If this wasn't clear already, just look at the names: Proxmox names its root zpool "rpool", whereas OpnSense names its pool "zroot".
Thus taking care of filesystems (or zpools) in an OpnSense VM is just as much its business as without virtualisation. Think of logical errors in the VM itself, which are not manifested on the underlying hardware.
For the default zroot pool, autotrim is on. While you could scrub, it would only be useful if you had multiple disks. If OpnSense is operated as VM, it would be more useful if the underlying Proxmox storage was redundant.
Quote from: meyergru on January 19, 2025, 04:37:14 PMQuote from: mooh on January 02, 2025, 06:31:49 PMYes, I you have a ZFS pool inside your VM, the VM needs to do the trimming and scrubbing. If this pools happens to be mapped to a file in a ZFS pool of the host, the scrubbing is not needed since it is done by the host. Actually, scrubbing a virtual ZFS pool doesn't make sense.
Sorry, but this is plain wrong or at least ambiguous.. The trimming and scrubbing taking place on the VM host works on its own ZFS pools, not on the OpnSense pools.
Within those pools, you have virtual disks, which are usually ZFS volumes. What the VM OS - in this case OpnSense - does with that, is its own business and the VM host does not look into it. OpnSense could as well be installed on UFS, which Proxmox would not even understand. On the other hand, you could have LVM filesystems on Proxmox and ZFS on OpnSense. You see: Even if both OpnSense and Proxmox use ZFS pools, they are independent of one another. If this wasn't clear already, just look at the names: Proxmox names its root zpool "rpool", whereas OpnSense names its pool "zroot".
Thus taking care of filesystems (or zpools) in an OpnSense VM is just as much its business as without virtualisation. Think of logical errors in the VM itself, which are not manifested on the underlying hardware.
For the default zroot pool, autotrim is on. While you could scrub, it would only be useful if you had multiple disks. If OpnSense is operated as VM, it would be more useful if the underlying Proxmox storage was redundant.
There you go. Thanks for the detailed explanation.
Only thing I'm wondering about is why autotrim in my opnsense zpool isn't enabled?
I do not know if autotrim has always been on with OpnSense. Maybe your initial installation did have it disabled?
Did you check with "zpool get all | fgrep autotrim"? All my instances have it enabled.
I guess I should have written something like "Only the machine hosting the physical devices should do the trimming and scrubbing" to be more clear than "scrubbing a virtual ZFS pool doesn't make sense".
And for completeness: It's only a waste of resources to do integrity checks on data that lives inside a file system with integrity checks. If data integrity is of utmost importance and one intends to move the VM to host machines that don't have such file systems, it is still advisable to do the scrubbing inside the VM.
Quote from: meyergru on January 20, 2025, 03:42:46 PMI do not know if autotrim has always been on with OpnSense. Maybe your initial installation did have it disabled?
Did you check with "zpool get all | fgrep autotrim"? All my instances have it enabled.
Yes.
root@OPNsense:~ # zpool get all | fgrep autotrim
zroot autotrim off default
I forgot the exact version of my initial install but it's pretty recent, like maybe June 2024 version.
I recently reinstalled a system with the 24.7 image, and it was still set to disabled in that release.
IIRC, the historical reason for having autotrim disabled by default was that it allowed system administrators to schedule it more effectively, avoiding potential random performance drops caused by automatic triggering.
Maybe I remember that wrong and have set autotrim anywhere myself...
It's in your HOWTO (https://forum.opnsense.org/index.php?topic=44159.0) ;-)
QuoteThat is, if the ZFS "autotrim" feature is not set manually. You can either set this via the OpnSense CLI with "zpool set autotrim=on zroot" or, better, add a daily cron job to to this (System: Settings: Cron) with "zroot" as parameter.
You can trim your zpool once via CLI with "zpool trim zroot".
Ah, there you go, I am getting old. Nice HowTo, BTW...
I do this because the ZFS volume will fill up to 100% after a while and thus VM backups take very long.
Was looking at this thread with interest as also having OPNsense running under Proxmox, so I went to check the auto-trim option - it's disabled. So I ran it manually for good measure.
I then ran a status check on the pool and it came back with
root@bart:~ # zpool status zroot
pool: zroot
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
da0p4 ONLINE 0 0 0
errors: No known data errors
Should I perform the upgrade? I've seen this on Proxmox a couple of times and did the upgrade without issues.
The short answer to that is no. The long answer you will find here (https://forum.opnsense.org/index.php?topic=29304.msg206737#msg206737).
Quote from: meyergru on January 19, 2025, 04:37:14 PMFor the default zroot pool, autotrim is on. While you could scrub, it would only be useful if you had multiple disks.
I tend to disagree: a scrub will check the checksums and therefore
show if the file system is corrupted.
It won't be able to self-heal with just one disk, but it will be able to tell you that you need to pay attention to that pool (e.g. replace the disk and restore from backup).
Quote from: EricPerl on January 20, 2025, 09:53:30 PMIt's in your HOWTO (https://forum.opnsense.org/index.php?topic=44159.0) ;-)
QuoteThat is, if the ZFS "autotrim" feature is not set manually. You can either set this via the OpnSense CLI with "zpool set autotrim=on zroot" or, better, add a daily cron job to to this (System: Settings: Cron) with "zroot" as parameter.
You can trim your zpool once via CLI with "zpool trim zroot".
Oh nice, I didn't know meyergru had a How To guide. Good to use for my environment then.
Quote from: meyergru on January 21, 2025, 12:54:00 PMThe short answer to that is no. The long answer you will find here (https://forum.opnsense.org/index.php?topic=29304.msg206737#msg206737).
Thanks for letting me know and I'm glad I asked - I saw your feature request and will wait to see if something official gets added.