high disk usage, du and df differ

[e97]

Hello,
I've been running into issues with diskspace - 32GB.

Code: [Select]

# df -h
Filesystem                  Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs              31G     16G     13G    56%    /
devfs                       1.0K    1.0K      0B   100%    /dev
/dev/gpt/efifs              256M    1.7M    254M     1%    /boot/efi
tmpfs                       8.0G    1.2G    6.8G    15%    /var/log
tmpfs                       8.0G    1.5M    8.0G     0%    /tmp
devfs                       1.0K    1.0K      0B   100%    /var/dhcpd/dev
devfs                       1.0K    1.0K      0B   100%    /var/unbound/dev
/usr/local/lib/python3.9     31G     16G     13G    56%    /var/unbound/usr/local/lib/python3.9

Code: [Select]

root@OPNsense:/ # du -chs *
8.0K COPYRIGHT
1.4M bin
340M boot
7.6M conf
4.0K dev
4.0K entropy
3.5M etc
4.0K home
 14M lib
160K libexec
4.0K media
4.0K mnt
4.0K net
4.0K proc
4.0K rescue
 44K root
5.8M sbin
  0B sys
1.5M tmp
1.4G usr
1.7G var
3.5G total
GUI shows 16G used.

16G vs 3.5G what am I missing?

[meyergru]

Swap.

[knebb]

Hi,

have you changed anything on the disk layout or filesystems?

If not I'd recommend not to worry. With filesystems like ZFS (or BTRFS) the du/df tools are not really reliable any more. Because of compression and data deduplication. They might, but but they might not as well.

Anyways, I'd say you system looks fine. But yes, the 16G looks strange. For me, this directory in df says 1.4G used, but in `/usr/local/lib/python3.9` or in `/var/unbound/usr/local/lib/python3.9` I have only 482M according to `du`.

I guess it is something related to a chroot environment of unbound.  There might be some files in `/var/unbound/usr/local/lib/python3.9` where the `/usr...`is bind-mount over it. Sorry, just a guess.
If you urgently need to figure out, I'd stop unbound service, try to unmount the oython3.9 dir and see what's underneath.
As @meyergru suggested you might use `swapinfo`but I doubt it is related.

/KNEBB

[meyergru]

Swap is configured per default, but it usually deducts from the physical disk space, so does not explain this difference.

Compression would work out in the other direction, namely more used space is shown in du than in df.

Deduplication is off per default and would work the other direction as well.

The "du -csh *" skips all files and directories with dots, but I doubt that this is the culprit.

It could also be ZFS snapshots / boot environments. You can see those with "zfs list" / "bectl list".

[e97]

I'm using UFS - maybe I should re-do this using ZFS?

mount

Code: [Select]

# mount
/dev/gpt/rootfs on / (ufs, local, soft-updates, journaled soft-updates)
devfs on /dev (devfs)
/dev/gpt/efifs on /boot/efi (msdosfs, local)
tmpfs on /var/log (tmpfs, local)
tmpfs on /tmp (tmpfs, local)
devfs on /var/dhcpd/dev (devfs)
devfs on /var/unbound/dev (devfs)
/usr/local/lib/python3.9 on /var/unbound/usr/local/lib/python3.9 (nullfs, local, read-only, soft-updates, journaled soft-updates)
Not swap -- see attached screenshot - 0% of 8GB used.

I had issues with flowd / netflow logs becoming full. flowd_aggregator crashed and I tried to run it manually but it hung so I reset the netflow data using the Web UI (Reporting > Settings > "Reset Netflow data")

I found a thread (https://www.reddit.com/r/OPNsenseFirewall/comments/qmsk9e/disk_usage_growing_cant_find_it/) that said it might be due to open file handles from the crashed flowd_aggregator and I can check with lsof | grep '(deleted)' but opnsense doesnt have lsof (at least not in plugins or packages) and I'm not sure how to install it.

I previously had the opnsense lock up due to out of disk space. Recent kept logs is at 7 days. I only want logs for debugging any recent issues -- not long term so I have ram disk 16GB (can bump to 32GB if necessary)

I sym-linked /var/netflow to ramdisk on /var/log to avoid this and it seems to be working but filesize keeps growing

Code: [Select]

2023-12-18T03:40:16-05:00 Notice flowd_aggregate.py vacuum done
2023-12-18T03:40:15-05:00 Notice flowd_aggregate.py vacuum interface_086400.sqlite
2023-12-18T03:40:15-05:00 Notice flowd_aggregate.py vacuum interface_003600.sqlite
2023-12-18T03:40:15-05:00 Notice flowd_aggregate.py vacuum interface_000300.sqlite
2023-12-18T03:40:15-05:00 Notice flowd_aggregate.py vacuum interface_000030.sqlite
2023-12-18T03:40:15-05:00 Notice flowd_aggregate.py vacuum dst_port_086400.sqlite
2023-12-18T03:40:13-05:00 Notice flowd_aggregate.py vacuum dst_port_003600.sqlite
2023-12-18T03:40:09-05:00 Notice flowd_aggregate.py vacuum dst_port_000300.sqlite
2023-12-18T03:40:09-05:00 Notice flowd_aggregate.py vacuum src_addr_086400.sqlite
2023-12-18T03:40:09-05:00 Notice flowd_aggregate.py vacuum src_addr_003600.sqlite
2023-12-18T03:40:08-05:00 Notice flowd_aggregate.py vacuum src_addr_000300.sqlite
2023-12-18T03:39:45-05:00 Notice flowd_aggregate.py vacuum src_addr_details_086400.sqlite
2023-12-17T19:37:41-05:00 Notice flowd_aggregate.py vacuum done
2023-12-17T19:37:41-05:00 Notice flowd_aggregate.py start watching flowd
2023-12-17T19:37:41-05:00 Notice flowd_aggregate.py startup, check database.
latest df

Code: [Select]

# df -h
Filesystem                  Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs              31G     16G     13G    56%    /
devfs                       1.0K    1.0K      0B   100%    /dev
/dev/gpt/efifs              256M    1.7M    254M     1%    /boot/efi
tmpfs                       8.0G    4.4G    3.6G    55%    /var/log
tmpfs                       8.0G    1.9M    8.0G     0%    /tmp
devfs                       1.0K    1.0K      0B   100%    /var/dhcpd/dev
devfs                       1.0K    1.0K      0B   100%    /var/unbound/dev
/usr/local/lib/python3.9     31G     16G     13G    56%    /var/unbound/usr/local/lib/python3.9
latest du

Code: [Select]

/ # du -chs *
8.0K COPYRIGHT
1.4M bin
340M boot
7.6M conf
4.0K dev
4.0K entropy
3.5M etc
4.0K home
 14M lib
160K libexec
4.0K media
4.0K mnt
4.0K net
4.0K proc
4.0K rescue
 48K root
5.8M sbin
  0B sys
1.9M tmp
1.4G usr
4.8G var
6.6G total
find largest files

Code: [Select]

/ # find /var/ -type f -exec du -Ah {} + | sort -h | tail -30
2.3M /var/unbound/usr/local/lib/python3.9/site-packages/netaddr/eui/iab.txt
2.7M /var/unbound/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
3.8M /var/log/flowd.log
3.9M /var/log/configd/configd_20231218.log
5.3M /var/unbound/usr/local/lib/python3.9/site-packages/netaddr/eui/oui.txt
5.7M /var/unbound/usr/local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39.so
6.1M /var/log/netflow/src_addr_086400.sqlite
6.1M /var/unbound/usr/local/lib/python3.9/config-3.9/libpython3.9.a
7.5M /var/db/pkg/vuln.xml
8.3M /var/unbound/data/unbound.duckdb
 12M /var/log/flowd.log.000002
 12M /var/log/flowd.log.000005
 12M /var/log/flowd.log.000008
 12M /var/log/flowd.log.000010
 13M /var/log/flowd.log.000003
 13M /var/log/flowd.log.000004
 13M /var/log/flowd.log.000006
 13M /var/log/flowd.log.000007
 13M /var/log/flowd.log.000009
 14M /var/db/pkg/local.sqlite
 15M /var/log/flowd.log.000001
 25M /var/log/netflow/dst_port_086400.sqlite
 29M /var/log/netflow/src_addr_003600.sqlite
 43M /var/unbound/usr/local/lib/python3.9/site-packages/duckdb.cpython-39.so
 64M /var/log/netflow/src_addr_000300.sqlite
226M /var/log/netflow/dst_port_003600.sqlite
254M /var/log/netflow/dst_port_000300.sqlite
301M /var/log/filter/filter_20231217.log
684M /var/log/filter/filter_20231218.log
2.7G /var/log/netflow/src_addr_details_086400.sqlite

[doktornotor]

I'm using UFS - maybe I should re-do this using ZFS?

Sounds like an excellent idea. At least it doesn't crash after every power failure. Also the filesystem is lz4-compressed by default, if you are space-constrained.

[e97]

Sounds like an excellent idea. At least it doesn't crash after every power failure. Also the filesystem is lz4-compressed by default, if you are space-constrained.

Most of the guides say either ufs or zfs is fine. The installer even defaults to ufs. Outside of the lz4-compression, I dont think the filesystem choice should make much of a difference.

Re: power loss corruption, I recently had a few power outages due to storms and none of my systems had any issues. Including this opnsense system with ufs. The drives in the systems dont have Power Loss Protection (PLP) but I use SSDs with well reviewed controllers.

The only power loss filesystem corruption I've had in recent years, was a low quality sd card in a raspberry pi which consistently wouldnt boot after 2-3 power loss events. After changing the sd card to a name brand one, I havent had an issue with power loss corruption.

Another data point is at the same time pi #1 with the cheap sd card got corrupted and re-flashed multiple times, another raspberry pi system with a photography grade sd card has been running for multiple years without issue.

[doktornotor]

Your choice. I can assure you that I have seen UFS irreparable (the fsck utility is also severely broken) crashes after hard reboot so many times that I wouldn't touch that filesystem with a 10ft pole on any remotely managed box.