Virtulized OPNsense, VHDX, ZFS, reclaim space

[iorx]

Hi,

Very little experience on zfs, just toyed around with it in a FreeNAS solution for many years ago. In my ongoing project to teach myself OPNsense I've installed a OPNsense with ZFS as i guest in Hyper-V (Server 2022).
Everything works great and fast firewall wise. I just noticed though that the VHDX on the host is ballooning in size. Now at around 75GB.

So, I reach out here for some BSD and ZFS knowledge. I've googled around on the subject but nothing seams to work. zfs; turn of dedup and compression. Filling the disk with 0 (dd), sync, removing file, sync.
I can't even see that the storage free changes. So, not knowing what's I'm doing here is a good summary.

Here is some info I think is relevant:

Code Select Expand

root@fwo:/ # du -shc /*
4.5K    /COPYRIGHT
875K    /bin
174M    /boot
6.8M    /conf
3.0K    /dev
4.5K    /entropy
2.1M    /etc
512B    /home
8.6M    /lib
113K    /libexec
512B    /media
512B    /mnt
512B    /net
512B    /proc
512B    /rescue
41K    /root
3.4M    /sbin
512B    /sys
93K    /tmp
797M    /usr
88M    /var
512B    /zroot
1.1G    total
root@fwo:/ # df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default    114G    1.0G    113G     1%    /
devfs                 1.0K    1.0K      0B   100%    /dev
/dev/da0p1            260M    1.8M    258M     1%    /boot/efi
zroot/var/mail        113G    120K    113G     0%    /var/mail
zroot/var/tmp         113G     96K    113G     0%    /var/tmp
zroot/var/log         113G     42M    113G     0%    /var/log
zroot/tmp             113G    196K    113G     0%    /tmp
zroot/var/audit       113G     96K    113G     0%    /var/audit
zroot/usr/src         113G     96K    113G     0%    /usr/src
zroot                 113G     96K    113G     0%    /zroot
zroot/var/crash       113G     96K    113G     0%    /var/crash
zroot/usr/home        113G     96K    113G     0%    /usr/home
zroot/usr/ports       113G     96K    113G     0%    /usr/ports
devfs                 1.0K    1.0K      0B   100%    /var/dhcpd/dev
devfs                 1.0K    1.0K      0B   100%    /var/unbound/dev
devfs                 1.0K    1.0K      0B   100%    /var/captiveportal/zone0/dev


Nowhere I can see that OPNsense is using 75GB of storage. My inexperience with finding stuff in BSD maybe?

Any hint on what's going on here? :D

Brgs,

[Patrick M. Hausen]

ZFS is copy on write. So every change in the filesystem leads to more space being used in a virtual disk. I would advise not to use ZFS if you intend to overprovision storage (what VMware calls "thin" provisioning). Use UFS instead. Or create the disk in the final desired size ("thick" provisioning).

[iorx]

So I've read, Copy on Write. Nice feature but not in this case I conclude.

I found this:
(Source link: https://forums.FreeBSD.org/threads/reclaim-unused-space-on-a-virtual-machine-on-a-sparse-vm.67880/post-405459)
UPDATE: For the record, I had success trimming the fat of my VM.
My VMs were not started with virtio-scsi driver. This driver supports unmap. So, as they were running without this. They were using the blocks (but not returning to the hypervisor, in this case KVM).

So, in order to reduce the size of a ZFS vm in KVM I followed this steps
1 I changed the disk driver to virtio-scsi
2 In the vm xml conf file. Added the option in driver section discard=unmap <driver name='qemu'discard='unmap' type='qcow2'/>
3 If enabled, disable compression and dedup. zfs set compression=off and zfs set dedup=off
4 Fill the disk with data dd bs=10M if=/dev/zero of=archivaso
5 Remove the big file, rm archivaso
6 Stop the vm
7 sparse (again) the vm. virt-sparsify FAT_VM.qcow2 SLIM_VM.qcow2
8 Change the image in your KVM conf, from FAT_VM to SLIM_VM.
9 Ensure you repeat step 2.
10 Start the vm
11 If needed, enable compression and dedup. zfs set compression=lz4 and zfs set dedup=on

Hope this helps to anyone looking to "shrink" their ZFS vms.

This guy/girl obviously managed to get the size down on a VM with ZFS.
I tried the outlined procedure but didn't gain much. The dd took forever. And I missed to turn off compression and dedup beforehand ... Dhoo :o! Not giving it a second try right now.

ZFS is good on handling unexpected shutdowns as I understand, which is a ince to have feature.
I've got UPS so maybe shouldn't be to worried about disk corruption.

But OK. Back to tackling my basic knowledge in BSD (if you got better things to do don't spend your time teaching me). I think I got this right but a confirmation would be good:

Code Select Expand

root@fwo:/ # du -shc /*
4.5K    /COPYRIGHT
875K    /bin
174M    /boot
6.8M    /conf
3.0K    /dev
4.5K    /entropy
2.1M    /etc
512B    /home
8.6M    /lib
113K    /libexec
512B    /media
512B    /mnt
512B    /net
512B    /proc
512B    /rescue
41K    /root
3.4M    /sbin
512B    /sys
93K    /tmp
797M    /usr
88M    /var
512B    /zroot
1.1G    total


So, my current install of OPNsense allocates only 1.1G of disk space, is that right?
This the right way: du -shc /*

Just learned that ZFS partition can't be shrunken. So I'll probably take a backup of OPNsense and start over on a new VM, this time with UFS and a smaller disk.

[y2kw]

Hi

You said at the first post that this is hyper v and here the solution is for KVM.

What I'm missing? I have this issue on hyper v and I want to know how I can sole this?
Mu vhdx get very big and but inside the OS it's barely use the space

[Patrick M. Hausen]

Two possibilities:

- do not overprovision, i.e. use "thick" virtual disk images. 32 GB is enough to run OPNsense

or

- do not use ZFS for "thin" virtual disks unless your hypervisor supports TRIM/unmap

HTH,
Patrick

[y2kw]

Thanks for answering

When it will be really full it will crush?

[Patrick M. Hausen]

No, but ZFS will always fill a thin virtual disk to the real configured capacity. If there is enough space on your hypervisor's storage, then nothing bad will happen.

E.g. configure 40 GB thin virtual disk, install OPNsense, it will take up 2-3 GB or some such. Eventually it will take up 40 GB. That's all. So for ZFS provision 40 GB thick from the start. Or use UFS, you can snapshot and backup from your hypervisor, right?

[y2kw]

OK, I think I'll do that but my concerns is that I will lost the option to store logs for long period of time(Month).

yes, but it almost blew my hard disk, i moved other VMs to free some space.

[Patrick M. Hausen]

Throw a 1 TB SSD in your hypervisor machine @50 bucks? I mean, storage is dirt cheap these days ...