Messages

To be honest I think it will never be compatible in the future because ZFS "formats the storage and writes data at the same time" so to speak when new data is written.

Well it was learning experience.  The only other environment I've ever played with VMware and ZFS inside is VM is my nested 4 node Proxmox cluster for learning Proxmox.  (And ZFS).  But that lab cluster is using a dedicated non redundant SSD and the many virtual disks were all setup as thick.  So this was first time seeing ZFS + thin.  Just threw me off as I initially saw the manual ZFS unmap / trim work really well.  But didn't give test VM time to run to see VMDK balloon.

All is well - I'm back to a fresh UFS VM. 
Cheers

To shrink an UFS VMDK

- fill with zeroes inside the guest, e.g. "dd if=/dev/zero of=/deleteme; rm /deleteme"
- shutdown VM
- in Workstation or Fusion simply compact disk from the UI
- in ESXi use vmkfstools to create a new thin copy - this one should then be significantly smaller

HTH,
Patrick

Thanks.  Yes I've done process several times on Debian VMs + Win VMs (sdelete).  But never FreeBSD.

Do have one question since you mentioned zerofile.
Is there any consideration with Opnsense services that should be shutdown prior to zerofile as the entire disk for a moment has zero disk space?  Would it be safest to do it in single user mode?

This is why I've never been fond of this manual shrink method when auto unmap doesn't work.  Problems can happen when 0 free space, if even for a moment.  And of course vmfstools also requires the VM downtime.

ZFS is a copy-on-write filesystem so as things stood last I checked, thin provisioning was futile. Any virtual disk will eventually have every single block written to.

I do not know if the introduction of virtual TRIM support in the hypervisor changes that in a signiifcant way. if things still are as they used to be, use thick provisioning with ZFS, problem solved. A firewall does not need more than a handful of dozens of GB.

Well like dinguz mentioned ZFS is not doing auto TRIM on the full disk.  So yeah it seems to defeat purpose.  But want thin provision as their is 4 copies of VMDK overhead.

My ~5 year old UFS VM had VMDK grown years back with bad NTOPNG settings.  And attempting to shrink VMDK it hasn't been working.  Either multi-user or single user mode with # fsck_ffs -Ey /dev/da0p3

(Which is why I decided to test out ZFS with fresh 26.1 install)  Only thing I didn't try on old UFS VM was a zero file and an offline #vmkfstools -K on ESXi host.  I didn't feel that was necessary since Opnsense config restore works so well.  Better to just build new VM.

I just deployed a new UFS VM.  Ran some ISO copy + deletes.  Auto TRIM/UNMAP in online multi-user mode is working by running #tunefs -t enable /.

# tunefs -p /.
tunefs: trim: (-t)  enabled

So at this point going to swap back UFS.

I'm not familiar with the intricacies of ZFS in a VM, but regarding autotrim: this does only trim recently freed up disk space, locally so to speak. It doesn't do full disk passes periodically; you would need zpool trim for that, as you have already found out.

I dunno either and have minimal experience with ZFS.  And even less with ZFS in a VM.  But their is obviously a massive write amplification happening.  VMDK is back to 46GB today from 30GB yesterday after pool trim.  Yet VM itself reports 5.5GB used on /
Storage policy of VM is x4 so VM its actually taking 182GB total datastore space for a measly 5.5GB of data inside the VM.  No bueno.


I'm going to run some unmap tests on fresh VM template with UFS and likely return to UFS.  I don't see ZFS compatible with thin disk...at least not its default pool / dataset parameters.

Anybody using ZFS + ESXi thin disks?

I ran:
dd if=/dev/zero of=/zerofile bs=1M count=62000
rm zerofile
zpool trim zroot

This shrunk vmdk down from ~47GB to ~30GB.  Still larger than the ~20GB it should be.
I think ZFS compression is a problem for ESXi unmap.  With Zerofile created it doesn't actually consume space.
I may try temporarily turning off compression on pool zroot/ROOT/default and trying zerofile again.
I'm still unclear why VMDK size exploded in just few days.

Last week I setup a new ESXi VM template to move from UFS to ZFS and upgrade to 26.1.
I ran several ZFS unmap tests inflating thin VMDK with large ISO and deleting.

zpool set autotrim=on zroot

System didn't appear to auto trim / unmap within ~30 min.
But running - zpool trim zroot - manual trim worked.  VMDK shrunk to very close to exact used space  So was happy and proceeded to swap over to new ZFS VM template.

Only been weekend since deployed and now reviewing today VMDK is 47GB yet Opnsense reports only 5GB used?
I've since ran manual trim again but it only shrunk VMDK ~1GB.  There is no way this much data has ever been written other than some internal ZFS function.

Is ZFS scrub or compression screwing with thin provisioning unmap / zero space?
I'm at a loss what FreeBSD + ZFS + VMFS thin disk is doing - Any suggestions appreciated.

Thin disk is 80GB

# df -h
             Filesystem                   Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default            69G    5.0G     64G     7%    /
devfs                        1.0K      0B    1.0K     0%    /dev
/dev/gpt/efiboot0            260M    1.3M    259M     1%    /boot/efi
zroot/var/mail                64G    160K     64G     0%    /var/mail
zroot/var/log                 64G     31M     64G     0%    /var/log
zroot/usr/src                 64G     96K     64G     0%    /usr/src
zroot/tmp                     64G    206M     64G     0%    /tmp
zroot                         64G     96K     64G     0%    /zroot
zroot/usr/ports               64G     96K     64G     0%    /usr/ports
zroot/var/audit               64G     96K     64G     0%    /var/audit
zroot/home                    64G     96K     64G     0%    /home
zroot/var/crash               64G     96K     64G     0%    /var/crash
zroot/var/tmp                 64G    388K     64G     0%    /var/tmp
devfs                        1.0K      0B    1.0K     0%    /var/unbound/dev
/usr/local/lib/python3.13     69G    5.0G     64G     7%    /var/unbound/usr/local/lib/python3.13
/lib                          69G    5.0G     64G     7%    /var/unbound/lib
/dev/md43                    484M     48K    445M     0%    /usr/local/zenarmor/output/active/temp
fdescfs                      1.0K      0B    1.0K     0%    /dev/fd
procfs                       8.0K      0B    8.0K     0%    /proc
tmpfs                        100M     24K    100M     0%    /usr/local/zenarmor/run/tracefs

# zpool status
  pool: zroot
 state: ONLINE
config:

   NAME        STATE     READ WRITE CKSUM
   zroot       ONLINE       0     0     0
     da0p4     ONLINE       0     0     0

errors: No known data errors

Edit:  Reviewing backup logs the fresh VM template VMDK was 14GB prior to few GB of Zenarmor / NTOPNG data accumulated over weekend.
VMDK has grown around ~29GB beyond what it should be in around 3 days.

Attempting to clean up (delete) old Sub CA cert and it's issued certs.
The cert won't delete because its in use by legacy OpenVPN which has since been removed from WebUI.
So apparently the legacy OpenVPN configuration has remained.  When I switched long time ago I believe I had just disabled the legacy OpenVPN servers.

Does this now require some manual edits of XML config and import?
What's simplest way to purge legacy OpenVPN config to free up certs for deletion?

It depends on a lot of factors.

Hi.  In the past I've read about state violation rule but like you say, lots of factors, so I've never really felt that I understand when issue surfaces.  Which is not often.

In regards to last time with IPSEC.  I restarted IPSEC service multiple times on both firewalls and Phase 2s were good and established fresh.  Traffic wouldn't flow yet I made no further config changes, rebooted remote Opnsense and VPN came backup up and traffic flowed through tunnel.  So I'm not sure.

I know there's implicit deny all rule on ACLs, but on all interfaces I have a manual deny all rule.  So when I see "Default deny / State violation" I know its Opnsense's internal rule and something with TCP states is wrong.  Otherwise if it was regular flow - it would be logged denied against my manual rule.

Both locations use Opnsense LAN INT on /29 subnet connected to L3 switch VLANs.  I have Zenarmor set on Opnsense LAN INT.  Perhaps Zenarmor be culprit for your mentioned reasons states get messed up - duplicated packets back, packet reordering, etc?

Bump.

Hi.

Can anyone shed some light to help me better understand the Default deny / State violations rule?  The causes and fix?
In the past when traffic isn't flowing that should be and is logging blocked by state violation I just reboot.
But I'm curious about resolving without reboot. 

For example
Yesterday my Spectrum home connection did a IP change on me for first time in 5+ years.
I re-configured a IPSEC VPN on both sides for new IP but traffic wasn't flowing despite phase 2s online.
Then I saw all VPN tunnel traffic being denied by state violation rule.
Rebooted the remote Opnsense and VPN began flowing again.

Would have reset states table under Firewall --> Diag --> States --> Actions fix issue?

Probably unrelated as my unbound wasn't crashing the service but I was having constant unreliable DNS resolution back on 24.1.x.  Both internal 53 + external DOT 853.  DNS resolutions would randomly fail and then try second later and work.  Constant SERVFAILS.

The fix I found finally was interfacing binding.  For years I had unbound both "Network Interfaces" + "Outgoing Network Interfaces" on specific interfaces.  Upon setting both to ALL interfaces (0.0.0.0) all my unbound problems went away.  Unbound ACLs to control access.  Finally returned to being reliable.

Probably not your situation but worth a check if your not binding unbound all interfaces.

(Also had numerous other binding issues.  Like Webui not running at boot due to specific MGT interfaces and redis also not starting when not set on LAN interfaces.  Believe either changes to freebsd itself or IPv6 work being done on Opnsense - Specific interface binding became quite problematic if not set on ALL within last year.)

opnsense-patch -c plugins 1e23572

Patch is the fix, however my setup did require a full opnsense reboot after installing patch.

Probably best to open an issue on Github

Created github issue - https://github.com/opnsense/plugins/issues/4358

I reviewed it a bit more and attempting to edit any config results in error.  Hunch is a writable permission issue since nothing of my nginx config can be changed.

Believe main config directory is /usr/local/etc/nginx  , directory root / wheel , root has rwx , wheel only has r-x, other r-x.

Also - never used but looked into opnsense-revert tool.  But nginx has been on 1.34 since opnsense 24.7.x
I also don't understand the _2 version of os-nginx.  Per release notes its just 1.34 yet opnsense plugin is 1.34_2
https://github.com/opnsense/plugins/blob/stable/24.7/www/nginx/pkg-descr

Tried the patch - opnsense-patch -c plugins 1e23572

Unfortunately did not fix it.  Tried Nginx restart.  Reverted patch.

Still cannot edit existing Nginx ACL or change active ACL on HTTP server.

Edit: actually just editing HTTP server with zero changes and attempting to save results in error.

Needing to adjust some NGINX ACLs today and ACL issue.

Upon editing ACL to add new IP I am getting "Unexpected error, check log for details"
Then tried creating new ACL and it worked.  Then tried editing same ACL.  Same error.

So then created brand new ACL with all needed IPs and saved ok.
Then went to HTTP server and attempted to change ACL to new ACL.  Same error. 
Cannot change HTTP server ACL.  :-/

Also tried all same above with NGINX service stopped.  Same error.

Reviewed NGINX log, Opnsense general + backend logs. Don't see anything mentioned about NGINX config / ACL.

Anyone aware of NGINX ACL issues or suggestions?

OPNsense 24.7.8
os-nginx 1.34_2