Started getting monit warnings of space utilization on my OPNsense instance. If I'm looking at this correctly it looks like it's related to the /var/unbound areas for whatever reason. Not sure how to properly diag this.
Filesystem Type Size Used Avail Capacity Mounted on
zroot/ROOT/default zfs 80G 60G 20G 75% /
devfs devfs 1.0K 0B 1.0K 0% /dev
/dev/gpt/efiboot0 msdosfs 260M 1.8M 258M 1% /boot/efi
fdescfs fdescfs 1.0K 0B 1.0K 0% /dev/fd
procfs procfs 8.0K 0B 8.0K 0% /proc
zroot/var/mail zfs 20G 128K 20G 0% /var/mail
zroot zfs 20G 96K 20G 0% /zroot
zroot/usr/src zfs 20G 96K 20G 0% /usr/src
zroot/usr/home zfs 20G 96K 20G 0% /usr/home
zroot/var/tmp zfs 20G 192K 20G 0% /var/tmp
zroot/var/crash zfs 20G 96K 20G 0% /var/crash
zroot/tmp zfs 20G 940K 20G 0% /tmp
zroot/usr/ports zfs 20G 96K 20G 0% /usr/ports
zroot/var/log zfs 21G 773M 20G 4% /var/log
zroot/var/audit zfs 20G 96K 20G 0% /var/audit
tmpfs tmpfs 1.6G 130M 1.5G 8% /var/log
tmpfs tmpfs 487M 163M 324M 33% /tmp
devfs devfs 1.0K 0B 1.0K 0% /var/dhcpd/dev
/dev/md43 ufs 242M 1.1M 221M 0% /usr/local/zenarmor/output/active/temp
tmpfs tmpfs 166M 3.4M 163M 2% /usr/local/zenarmor/run/tracefs
devfs devfs 1.0K 0B 1.0K 0% /var/unbound/dev
/usr/local/lib/python3.11 nullfs 80G 60G 20G 75% /var/unbound/usr/local/lib/python3.11
/lib nullfs 80G 60G 20G 75% /var/unbound/lib
No, it's not. The lines with "nullfs" in them are loopback mounts meaning the same directory is mounted at another location. So chrooted Unbound can find its libraries for example.
/zroot/ROOT/default which is the boot environment currently mounted at / is using 60G of your 80G zpool. To find what is using all this space do this as root:
cd /
du -skx * | sort -rn | head
On my system the result is
root@opnsense:/ # du -skx * | sort -rn | head
1283475 usr
193620 boot
94226 var
11823 conf
9975 lib
[...]
So the largest directory is usr. To continue:
cd usr
du -skx * | sort -rn | head
cd into the new largest one. Rinse and repeat until you found the culprit.
HTH,
Patrick
Quote from: Patrick M. Hausen on April 08, 2025, 08:51:24 PMNo, it's not. The lines with "nullfs" in them are loopback mounts meaning the same directory is mounted at another location. So chrooted Unbound can find its libraries for example.
/zroot/ROOT/default which is the boot environment currently mounted at / is using 60G of your 80G zpool. To find what is using all this space do this as root:
cd /
du -skx * | sort -rn | head
On my system the result is
root@opnsense:/ # du -skx * | sort -rn | head
1283475 usr
193620 boot
94226 var
11823 conf
9975 lib
[...]
So the largest directory is usr. To continue:
cd usr
du -skx * | sort -rn | head
cd into the new largest one. Rinse and repeat until you found the culprit.
HTH,
Patrick
This is absolutely amazing, wasn't familiar with that process. That worked perfectly, looks like zenarmor logs. Let me go investigate that now.
@Patrick M. Hausen's query helped me locate a ton of Zenarmor logs, apparently it was in debug logging mode... guess it was left there from working with zenarmor support some time ago and finally filled up the drive. All seems well now.
Variant using human readable format (instead kB per -k):
root@OPNsense:/ # du -shx /* | sort -rh | head
1.7G /usr
661M /var
188M /boot
...
TIL sort knows a "-h" flag. I've been using this since the late 80s. Newfangled nonsense ... 😂
😂
What's nonsense is 'du | sort' without matching options...
Fortunately, search engines come to the rescue sometimes. But you replied much faster than I did.
One day I'll try AI for this stuff.
P.S. The -x is important depending on the exact situation.