Last week I setup a new ESXi VM template to move from UFS to ZFS and upgrade to 26.1.
I ran several ZFS unmap tests inflating thin VMDK with large ISO and deleting.
zpool set autotrim=on zroot
System didn't appear to auto trim / unmap within ~30 min.
But running - zpool trim zroot - manual trim worked. VMDK shrunk to very close to exact used space So was happy and proceeded to swap over to new ZFS VM template.
Only been weekend since deployed and now reviewing today VMDK is 47GB yet Opnsense reports only 5GB used?
I've since ran manual trim again but it only shrunk VMDK ~1GB. There is no way this much data has ever been written other than some internal ZFS function.
Is ZFS scrub or compression screwing with thin provisioning unmap / zero space?
I'm at a loss what FreeBSD + ZFS + VMFS thin disk is doing - Any suggestions appreciated.
Thin disk is 80GB
# df -h
Filesystem Size Used Avail Capacity Mounted on
zroot/ROOT/default 69G 5.0G 64G 7% /
devfs 1.0K 0B 1.0K 0% /dev
/dev/gpt/efiboot0 260M 1.3M 259M 1% /boot/efi
zroot/var/mail 64G 160K 64G 0% /var/mail
zroot/var/log 64G 31M 64G 0% /var/log
zroot/usr/src 64G 96K 64G 0% /usr/src
zroot/tmp 64G 206M 64G 0% /tmp
zroot 64G 96K 64G 0% /zroot
zroot/usr/ports 64G 96K 64G 0% /usr/ports
zroot/var/audit 64G 96K 64G 0% /var/audit
zroot/home 64G 96K 64G 0% /home
zroot/var/crash 64G 96K 64G 0% /var/crash
zroot/var/tmp 64G 388K 64G 0% /var/tmp
devfs 1.0K 0B 1.0K 0% /var/unbound/dev
/usr/local/lib/python3.13 69G 5.0G 64G 7% /var/unbound/usr/local/lib/python3.13
/lib 69G 5.0G 64G 7% /var/unbound/lib
/dev/md43 484M 48K 445M 0% /usr/local/zenarmor/output/active/temp
fdescfs 1.0K 0B 1.0K 0% /dev/fd
procfs 8.0K 0B 8.0K 0% /proc
tmpfs 100M 24K 100M 0% /usr/local/zenarmor/run/tracefs
# zpool status
pool: zroot
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
da0p4 ONLINE 0 0 0
errors: No known data errors
Edit: Reviewing backup logs the fresh VM template VMDK was 14GB prior to few GB of Zenarmor / NTOPNG data accumulated over weekend.
VMDK has grown around ~29GB beyond what it should be in around 3 days.
Anybody using ZFS + ESXi thin disks?
I ran:
dd if=/dev/zero of=/zerofile bs=1M count=62000
rm zerofile
zpool trim zroot
This shrunk vmdk down from ~47GB to ~30GB. Still larger than the ~20GB it should be.
I think ZFS compression is a problem for ESXi unmap. With Zerofile created it doesn't actually consume space.
I may try temporarily turning off compression on pool zroot/ROOT/default and trying zerofile again.
I'm still unclear why VMDK size exploded in just few days.
I'm not familiar with the intricacies of ZFS in a VM, but regarding autotrim: this does only trim recently freed up disk space, locally so to speak. It doesn't do full disk passes periodically; you would need zpool trim for that, as you have already found out.
Quote from: dinguz on April 21, 2026, 10:11:41 PMI'm not familiar with the intricacies of ZFS in a VM, but regarding autotrim: this does only trim recently freed up disk space, locally so to speak. It doesn't do full disk passes periodically; you would need zpool trim for that, as you have already found out.
I dunno either and have minimal experience with ZFS. And even less with ZFS in a VM. But their is obviously a massive write amplification happening. VMDK is back to 46GB today from 30GB yesterday after pool trim. Yet VM itself reports 5.5GB used on /
Storage policy of VM is x4 so VM its actually taking 182GB total datastore space for a measly 5.5GB of data inside the VM. No bueno.
I'm going to run some unmap tests on fresh VM template with UFS and likely return to UFS. I don't see ZFS compatible with thin disk...at least not its default pool / dataset parameters.
ZFS is a copy-on-write filesystem so as things stood last I checked, thin provisioning was futile. Any virtual disk will eventually have every single block written to.
I do not know if the introduction of virtual TRIM support in the hypervisor changes that in a signiifcant way. if things still are as they used to be, use thick provisioning with ZFS, problem solved. A firewall does not need more than a handful of dozens of GB.
Quote from: Patrick M. Hausen on April 22, 2026, 09:19:48 PMZFS is a copy-on-write filesystem so as things stood last I checked, thin provisioning was futile. Any virtual disk will eventually have every single block written to.
I do not know if the introduction of virtual TRIM support in the hypervisor changes that in a signiifcant way. if things still are as they used to be, use thick provisioning with ZFS, problem solved. A firewall does not need more than a handful of dozens of GB.
Well like dinguz mentioned ZFS is not doing auto TRIM on the full disk. So yeah it seems to defeat purpose. But want thin provision as their is 4 copies of VMDK overhead.
My ~5 year old UFS VM had VMDK grown years back with bad NTOPNG settings. And attempting to shrink VMDK it hasn't been working. Either multi-user or single user mode with # fsck_ffs -Ey /dev/da0p3
(Which is why I decided to test out ZFS with fresh 26.1 install) Only thing I didn't try on old UFS VM was a zero file and an offline #vmkfstools -K on ESXi host. I didn't feel that was necessary since Opnsense config restore works so well. Better to just build new VM.
I just deployed a new UFS VM. Ran some ISO copy + deletes. Auto TRIM/UNMAP in online multi-user mode is working by running #tunefs -t enable /.
# tunefs -p /.
tunefs: trim: (-t) enabled
So at this point going to swap back UFS.
To shrink an UFS VMDK
- fill with zeroes inside the guest, e.g. "dd if=/dev/zero of=/deleteme; rm /deleteme"
- shutdown VM
- in Workstation or Fusion simply compact disk from the UI
- in ESXi use vmkfstools to create a new thin copy - this one should then be significantly smaller
HTH,
Patrick
Quote from: Patrick M. Hausen on April 22, 2026, 11:06:51 PMTo shrink an UFS VMDK
- fill with zeroes inside the guest, e.g. "dd if=/dev/zero of=/deleteme; rm /deleteme"
- shutdown VM
- in Workstation or Fusion simply compact disk from the UI
- in ESXi use vmkfstools to create a new thin copy - this one should then be significantly smaller
HTH,
Patrick
Thanks. Yes I've done process several times on Debian VMs + Win VMs (sdelete). But never FreeBSD.
Do have one question since you mentioned zerofile.
Is there any consideration with Opnsense services that should be shutdown prior to zerofile as the entire disk for a moment has zero disk space? Would it be safest to do it in single user mode?
This is why I've never been fond of this manual shrink method when auto unmap doesn't work. Problems can happen when 0 free space, if even for a moment. And of course vmfstools also requires the VM downtime.
Single user would of course be safest. Since you need to shutdown for the thin copy, anyway ...
Sorry. I've come to "50 G is plenty for a firewall and 50 G is nothing, so just use thick" conclusion.
Quote from: ThyOnlySandman on April 22, 2026, 08:56:14 PMI don't see ZFS compatible with thin disk...
To be honest I think it will never be compatible in the future because ZFS "formats the storage and writes data at the same time" so to speak when new data is written.
But I could be wrong ofcourse! :)
Quote from: Patrick M. Hausen on April 23, 2026, 12:06:06 AM"50 G is plenty for a firewall and 50 G is nothing, so just use thick provisioning"
OK, but OPNsense recommends 120 GB so let's do that then :)
Quote from: nero355 on April 23, 2026, 12:36:39 AMTo be honest I think it will never be compatible in the future because ZFS "formats the storage and writes data at the same time" so to speak when new data is written.
Well it was learning experience. The only other environment I've ever played with VMware and ZFS inside is VM is my nested 4 node Proxmox cluster for learning Proxmox. (And ZFS). But that lab cluster is using a dedicated non redundant SSD and the many virtual disks were all setup as thick. So this was first time seeing ZFS + thin. Just threw me off as I initially saw the manual ZFS unmap / trim work really well. But didn't give test VM time to run to see VMDK balloon.
All is well - I'm back to a fresh UFS VM.
Cheers
I have found ZFS to be more reliable. With UFS, I experienced disk corruption after power outages and other unclean shutdowns on a few occasions, but never with ZFS.
Quote from: dinguz on April 25, 2026, 07:09:21 PMI have found ZFS to be more reliable. With UFS, I experienced disk corruption after power outages and other unclean shutdowns on a few occasions, but never with ZFS.
Correct. Thick provisioning + ZFS is the way, IMHO.