Thin disk / ZFS / Unmap?

[ThyOnlySandman]

Last week I setup a new ESXi VM template to move from UFS to ZFS and upgrade to 26.1.
I ran several ZFS unmap tests inflating thin VMDK with large ISO and deleting.

zpool set autotrim=on zroot

System didn't appear to auto trim / unmap within ~30 min.
But running - zpool trim zroot - manual trim worked.  VMDK shrunk to very close to exact used space  So was happy and proceeded to swap over to new ZFS VM template.

Only been weekend since deployed and now reviewing today VMDK is 47GB yet Opnsense reports only 5GB used?
I've since ran manual trim again but it only shrunk VMDK ~1GB.  There is no way this much data has ever been written other than some internal ZFS function.

Is ZFS scrub or compression screwing with thin provisioning unmap / zero space?
I'm at a loss what FreeBSD + ZFS + VMFS thin disk is doing - Any suggestions appreciated.

Thin disk is 80GB

# df -h
             Filesystem                   Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default            69G    5.0G     64G     7%    /
devfs                        1.0K      0B    1.0K     0%    /dev
/dev/gpt/efiboot0            260M    1.3M    259M     1%    /boot/efi
zroot/var/mail                64G    160K     64G     0%    /var/mail
zroot/var/log                 64G     31M     64G     0%    /var/log
zroot/usr/src                 64G     96K     64G     0%    /usr/src
zroot/tmp                     64G    206M     64G     0%    /tmp
zroot                         64G     96K     64G     0%    /zroot
zroot/usr/ports               64G     96K     64G     0%    /usr/ports
zroot/var/audit               64G     96K     64G     0%    /var/audit
zroot/home                    64G     96K     64G     0%    /home
zroot/var/crash               64G     96K     64G     0%    /var/crash
zroot/var/tmp                 64G    388K     64G     0%    /var/tmp
devfs                        1.0K      0B    1.0K     0%    /var/unbound/dev
/usr/local/lib/python3.13     69G    5.0G     64G     7%    /var/unbound/usr/local/lib/python3.13
/lib                          69G    5.0G     64G     7%    /var/unbound/lib
/dev/md43                    484M     48K    445M     0%    /usr/local/zenarmor/output/active/temp
fdescfs                      1.0K      0B    1.0K     0%    /dev/fd
procfs                       8.0K      0B    8.0K     0%    /proc
tmpfs                        100M     24K    100M     0%    /usr/local/zenarmor/run/tracefs

# zpool status
  pool: zroot
 state: ONLINE
config:

   NAME        STATE     READ WRITE CKSUM
   zroot       ONLINE       0     0     0
     da0p4     ONLINE       0     0     0

errors: No known data errors

Edit:  Reviewing backup logs the fresh VM template VMDK was 14GB prior to few GB of Zenarmor / NTOPNG data accumulated over weekend.
VMDK has grown around ~29GB beyond what it should be in around 3 days.

[ThyOnlySandman]

Anybody using ZFS + ESXi thin disks?

I ran:
dd if=/dev/zero of=/zerofile bs=1M count=62000
rm zerofile
zpool trim zroot

This shrunk vmdk down from ~47GB to ~30GB.  Still larger than the ~20GB it should be.
I think ZFS compression is a problem for ESXi unmap.  With Zerofile created it doesn't actually consume space.
I may try temporarily turning off compression on pool zroot/ROOT/default and trying zerofile again.
I'm still unclear why VMDK size exploded in just few days.

[dinguz]

I'm not familiar with the intricacies of ZFS in a VM, but regarding autotrim: this does only trim recently freed up disk space, locally so to speak. It doesn't do full disk passes periodically; you would need zpool trim for that, as you have already found out.

[ThyOnlySandman]

I'm not familiar with the intricacies of ZFS in a VM, but regarding autotrim: this does only trim recently freed up disk space, locally so to speak. It doesn't do full disk passes periodically; you would need zpool trim for that, as you have already found out.

I dunno either and have minimal experience with ZFS.  And even less with ZFS in a VM.  But their is obviously a massive write amplification happening.  VMDK is back to 46GB today from 30GB yesterday after pool trim.  Yet VM itself reports 5.5GB used on /
Storage policy of VM is x4 so VM its actually taking 182GB total datastore space for a measly 5.5GB of data inside the VM.  No bueno.


I'm going to run some unmap tests on fresh VM template with UFS and likely return to UFS.  I don't see ZFS compatible with thin disk...at least not its default pool / dataset parameters.

[Patrick M. Hausen]

ZFS is a copy-on-write filesystem so as things stood last I checked, thin provisioning was futile. Any virtual disk will eventually have every single block written to.

I do not know if the introduction of virtual TRIM support in the hypervisor changes that in a signiifcant way. if things still are as they used to be, use thick provisioning with ZFS, problem solved. A firewall does not need more than a handful of dozens of GB.

[ThyOnlySandman]

ZFS is a copy-on-write filesystem so as things stood last I checked, thin provisioning was futile. Any virtual disk will eventually have every single block written to.

I do not know if the introduction of virtual TRIM support in the hypervisor changes that in a signiifcant way. if things still are as they used to be, use thick provisioning with ZFS, problem solved. A firewall does not need more than a handful of dozens of GB.

Well like dinguz mentioned ZFS is not doing auto TRIM on the full disk.  So yeah it seems to defeat purpose.  But want thin provision as their is 4 copies of VMDK overhead.

My ~5 year old UFS VM had VMDK grown years back with bad NTOPNG settings.  And attempting to shrink VMDK it hasn't been working.  Either multi-user or single user mode with # fsck_ffs -Ey /dev/da0p3

(Which is why I decided to test out ZFS with fresh 26.1 install)  Only thing I didn't try on old UFS VM was a zero file and an offline #vmkfstools -K on ESXi host.  I didn't feel that was necessary since Opnsense config restore works so well.  Better to just build new VM.

I just deployed a new UFS VM.  Ran some ISO copy + deletes.  Auto TRIM/UNMAP in online multi-user mode is working by running #tunefs -t enable /.

# tunefs -p /.
tunefs: trim: (-t)  enabled

So at this point going to swap back UFS.

[Patrick M. Hausen]

To shrink an UFS VMDK

- fill with zeroes inside the guest, e.g. "dd if=/dev/zero of=/deleteme; rm /deleteme"
- shutdown VM
- in Workstation or Fusion simply compact disk from the UI
- in ESXi use vmkfstools to create a new thin copy - this one should then be significantly smaller

HTH,
Patrick

[ThyOnlySandman]

To shrink an UFS VMDK

- fill with zeroes inside the guest, e.g. "dd if=/dev/zero of=/deleteme; rm /deleteme"
- shutdown VM
- in Workstation or Fusion simply compact disk from the UI
- in ESXi use vmkfstools to create a new thin copy - this one should then be significantly smaller

HTH,
Patrick

Thanks.  Yes I've done process several times on Debian VMs + Win VMs (sdelete).  But never FreeBSD.

Do have one question since you mentioned zerofile.
Is there any consideration with Opnsense services that should be shutdown prior to zerofile as the entire disk for a moment has zero disk space?  Would it be safest to do it in single user mode?

This is why I've never been fond of this manual shrink method when auto unmap doesn't work.  Problems can happen when 0 free space, if even for a moment.  And of course vmfstools also requires the VM downtime.

[Patrick M. Hausen]

Single user would of course be safest. Since you need to shutdown for the thin copy, anyway ...

Sorry. I've come to "50 G is plenty for a firewall and 50 G is nothing, so just use thick" conclusion.

[nero355]

I don't see ZFS compatible with thin disk...

To be honest I think it will never be compatible in the future because ZFS "formats the storage and writes data at the same time" so to speak when new data is written.

But I could be wrong ofcourse! :)

"50 G is plenty for a firewall and 50 G is nothing, so just use thick provisioning"

OK, but OPNsense recommends 120 GB so let's do that then :)

[ThyOnlySandman]

To be honest I think it will never be compatible in the future because ZFS "formats the storage and writes data at the same time" so to speak when new data is written.

Well it was learning experience.  The only other environment I've ever played with VMware and ZFS inside is VM is my nested 4 node Proxmox cluster for learning Proxmox.  (And ZFS).  But that lab cluster is using a dedicated non redundant SSD and the many virtual disks were all setup as thick.  So this was first time seeing ZFS + thin.  Just threw me off as I initially saw the manual ZFS unmap / trim work really well.  But didn't give test VM time to run to see VMDK balloon.

All is well - I'm back to a fresh UFS VM. 
Cheers