Thoughts on disk space utilization. Appears to be unbound related?

[FullyBorked]

Started getting monit warnings of space utilization on my OPNsense instance.  If I'm looking at this correctly it looks like it's related to the /var/unbound areas for whatever reason.  Not sure how to properly diag this. 

Code Select Expand

Filesystem                 Type       Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default         zfs         80G     60G     20G    75%    /
devfs                      devfs      1.0K      0B    1.0K     0%    /dev
/dev/gpt/efiboot0          msdosfs    260M    1.8M    258M     1%    /boot/efi
fdescfs                    fdescfs    1.0K      0B    1.0K     0%    /dev/fd
procfs                     procfs     8.0K      0B    8.0K     0%    /proc
zroot/var/mail             zfs         20G    128K     20G     0%    /var/mail
zroot                      zfs         20G     96K     20G     0%    /zroot
zroot/usr/src              zfs         20G     96K     20G     0%    /usr/src
zroot/usr/home             zfs         20G     96K     20G     0%    /usr/home
zroot/var/tmp              zfs         20G    192K     20G     0%    /var/tmp
zroot/var/crash            zfs         20G     96K     20G     0%    /var/crash
zroot/tmp                  zfs         20G    940K     20G     0%    /tmp
zroot/usr/ports            zfs         20G     96K     20G     0%    /usr/ports
zroot/var/log              zfs         21G    773M     20G     4%    /var/log
zroot/var/audit            zfs         20G     96K     20G     0%    /var/audit
tmpfs                      tmpfs      1.6G    130M    1.5G     8%    /var/log
tmpfs                      tmpfs      487M    163M    324M    33%    /tmp
devfs                      devfs      1.0K      0B    1.0K     0%    /var/dhcpd/dev
/dev/md43                  ufs        242M    1.1M    221M     0%    /usr/local/zenarmor/output/active/temp
tmpfs                      tmpfs      166M    3.4M    163M     2%    /usr/local/zenarmor/run/tracefs
devfs                      devfs      1.0K      0B    1.0K     0%    /var/unbound/dev
/usr/local/lib/python3.11  nullfs      80G     60G     20G    75%    /var/unbound/usr/local/lib/python3.11
/lib                       nullfs      80G     60G     20G    75%    /var/unbound/lib


[Patrick M. Hausen]

No, it's not. The lines with "nullfs" in them are loopback mounts meaning the same directory is mounted at another location. So chrooted Unbound can find its libraries for example.

/zroot/ROOT/default which is the boot environment currently mounted at / is using 60G of your 80G zpool. To find what is using all this space do this as root:

Code Select Expand

cd /
du -skx * | sort -rn | head

On my system the result is

Code Select Expand

root@opnsense:/ # du -skx * | sort -rn | head
1283475 usr
193620 boot
94226 var
11823 conf
9975 lib
[...]

So the largest directory is usr. To continue:

Code Select Expand

cd usr
du -skx * | sort -rn | head

cd into the new largest one. Rinse and repeat until you found the culprit.

HTH,
Patrick

[FullyBorked]

No, it's not. The lines with "nullfs" in them are loopback mounts meaning the same directory is mounted at another location. So chrooted Unbound can find its libraries for example.

/zroot/ROOT/default which is the boot environment currently mounted at / is using 60G of your 80G zpool. To find what is using all this space do this as root:

Code Select Expand

cd /
du -skx * | sort -rn | head

On my system the result is

Code Select Expand

root@opnsense:/ # du -skx * | sort -rn | head
1283475 usr
193620 boot
94226 var
11823 conf
9975 lib
[...]

So the largest directory is usr. To continue:

Code Select Expand

cd usr
du -skx * | sort -rn | head

cd into the new largest one. Rinse and repeat until you found the culprit.

HTH,
Patrick

This is absolutely amazing, wasn't familiar with that process.  That worked perfectly, looks like zenarmor logs.  Let me go investigate that now. 

[FullyBorked]

@Patrick M. Hausen's query helped me locate a ton of Zenarmor logs, apparently it was in debug logging mode... guess it was left there from working with zenarmor support some time ago and finally filled up the drive.  All seems well now. 

[EricPerl]

Variant using human readable format (instead kB per -k):

Code Select Expand

root@OPNsense:/ # du -shx /* | sort -rh | head
1.7G    /usr
661M    /var
188M    /boot
...

[Patrick M. Hausen]

TIL sort knows a "-h" flag. I've been using this since the late 80s. Newfangled nonsense ... 😂

[EricPerl]

😂
What's nonsense is 'du | sort' without matching options...
Fortunately, search engines come to the rescue sometimes. But you replied much faster than I did.
One day I'll try AI for this stuff.

[Patrick M. Hausen]

P.S. The -x is important depending on the exact situation.