OPNsense runs out of space

[m4rtin]

Hi,

my OPNsense runs out of space since about September. I don't know what exactly I changed but since then the free space declines about 16 G in 2 weeks (see screenshot).

I one removed some log files but it didn't fully solve the problem.

Code Select Expand

du -sh doesn't seem to show all files (see screnshots).

Do you have any idea?

[Patrick M. Hausen]

https://forum.opnsense.org/index.php?topic=34791.msg168539#msg168539

[CJ]

https://forum.opnsense.org/index.php?topic=34791.msg168539#msg168539

Why that set of options?  I would think these would be easier.

Code Select Expand

du -h / | sort -rh | head

Or you could just skip the -r and head completely and look at the bottom.

[Patrick M. Hausen]

-s sum up the directories
-k report in kbytes
-x don't cross file system mount points

If you use -h, sort -n will sort 1G before 2M before 3k ...

[m4rtin]

Thanks for the hint! I found an unues mongodb and some logfiles. In total about 20 GB. Now there are still 60 GB used if I look on the webinterface or 20 GB if I look in the shell, but I can live with that as long as it doesn't get more.

[cookiemonster]

Don't forget to check System > Settings > Logging. Tooltips for the fields are useful. The most important in my opinion is the number of days to preserve, as it ensures the log file rotation keeps storage on reasonable levels, depending on the need/want for history.

[CJ]

-s sum up the directories
-k report in kbytes
-x don't cross file system mount points

If you use -h, sort -n will sort 1G before 2M before 3k ...

Not if you add -h to the sort. :)

[Patrick M. Hausen]

I honestly did not know sort can do that. I am using my variant for over 30 years, now and did not see a reason to change it.  :)

[CJ]

I honestly did not know sort can do that. I am using my variant for over 30 years, now and did not see a reason to change it.  :)

;D

[m4rtin]

I now changed the preserve days to 7, but still 83 GB seemes to be in use (dashboard).

But the command only outputs 29 GB in /:

Code Select Expand

root@OPNsense:/var/log # du -ah / | sort -rh | head -n 20
29G    /
25G    /var
23G    /var/log
13G    /var/log/flowd.log
9.3G    /var/log/filter
4.2G    /usr
2.0G    /usr/swap0
1.7G    /usr/local
1.1G    /var/netflow
666M    /var/log/suricata
607M    /usr/local/lib
547M    /var/log/filter/filter_20231129.log
540M    /var/log/filter/filter_20231127.log
534M    /var/log/filter/filter_20231206.log
532M    /var/log/filter/filter_20231128.log
531M    /var/log/filter/filter_20231205.log
529M    /var/log/filter/filter_20231130.log
522M    /var/log/filter/filter_20231207.log
512M    /var/log/filter/filter_20231204.log
509M    /var/log/filter/filter_20231123.log


Do you know where the other 54 GB may hide?   :D

[dinguz]

try as root/admin user:

Code Select Expand

find / -type f -exec du -h {} + | sort -h

[m4rtin]

That's basically the same:

Code Select Expand

40M    /usr/local/etc/suricata/rules/rules.sqlite
42M    /usr/local/datastore/mongodb/mongod.log
43M    /usr/local/lib/python3.9/site-packages/duckdb.cpython-39.so
43M    /var/log/suricata/eve.json
43M    /var/unbound/usr/local/lib/python3.9/site-packages/duckdb.cpython-39.so
47M    /usr/local/bin/mongod
55M    /usr/bin/ld.lld
56M    /usr/local/zenarmor/bin/ipdrstreamer
58M    /var/netflow/interface_000030.sqlite
64M    /usr/local/zenarmor/db/GeoIP/GeoLite2-City.mmdb
79M    /var/netflow/dst_port_003600.sqlite
81M    /usr/bin/c++
83M    /usr/bin/lldb
92M    /var/log/suricata/eve.json.2
100M    /usr/local/datastore/mongodb/journal/WiredTigerLog.0000000001
100M    /usr/local/datastore/mongodb/journal/WiredTigerPreplog.0000000001
100M    /usr/local/datastore/mongodb/journal/WiredTigerPreplog.0000000002
102M    /var/log/suricata/eve.json.3
103M    /var/log/suricata/eve.json.1
108M    /var/log/suricata/eve.json.0
112M    /var/netflow/dst_port_086400.sqlite
130M    /var/netflow/dst_port_000300.sqlite
164M    /var/log/filter/filter_20231213.log
264M    /var/netflow/src_addr_086400.sqlite
417M    /var/log/filter/filter_20231210.log
418M    /var/log/filter/filter_20231209.log
447M    /var/netflow/src_addr_details_086400.sqlite
479M    /var/log/filter/filter_20231208.log
493M    /var/log/filter/filter_20231211.log
498M    /var/log/filter/filter_20231212.log
522M    /var/log/filter/filter_20231207.log
2.0G    /usr/swap0
13G    /var/log/flowd.log


[doktornotor]

At the risk of stating the obvious, did you use some reliable method to check the disk space usage first? Cannot even make sense of where does the graph come from in the original post.

Code Select Expand


# zpool list
# df -h

Some more notes:

- Those netflow DBs and logs can eat entire disk space easily. Get some decent storage before enabling it. If unable, disable and reset netflow data.
- You seem to be running the (absolutely horrlble) MongoDB thing on your firewall. For what? Yuck.
- Collecting half gig of firewall logs a day - what's the log retention set to.

Finally: have you ever rebooted the box after deleting those mongdb and whatnot files you mentioned earlier?

[doktornotor]

I mean, with ZFS in place with compression enabled, we are not even getting meaningful figures here. Consider:

# man du

Code Select Expand


     -A      Display the apparent size instead of the disk usage.  This can be
             helpful when operating on compressed volumes or sparse files.


Code Select Expand


# find /var/log/filter -type f -exec du -Ah {} + | sort -h
9.2M    /var/log/filter/filter_20231213.log
17M    /var/log/filter/filter_20231210.log
23M    /var/log/filter/filter_20231204.log
24M    /var/log/filter/filter_20231211.log
30M    /var/log/filter/filter_20231212.log
58M    /var/log/filter/filter_20231206.log
59M    /var/log/filter/filter_20231209.log
75M    /var/log/filter/filter_20231207.log
92M    /var/log/filter/filter_20231208.log
95M    /var/log/filter/filter_20231205.log


vs.

Code Select Expand


# find /var/log/filter -type f -exec du -h {} + | sort -h
1.4M    /var/log/filter/filter_20231213.log
1.9M    /var/log/filter/filter_20231210.log
3.1M    /var/log/filter/filter_20231204.log
3.1M    /var/log/filter/filter_20231211.log
4.2M    /var/log/filter/filter_20231212.log
8.1M    /var/log/filter/filter_20231209.log
8.2M    /var/log/filter/filter_20231206.log
11M    /var/log/filter/filter_20231207.log
13M    /var/log/filter/filter_20231205.log
13M    /var/log/filter/filter_20231208.log


So, e.g. those firewall log files here you listed, they are actually not half gig, but ~5G per day. :o

Code Select Expand


547M    /var/log/filter/filter_20231129.log
540M    /var/log/filter/filter_20231127.log
534M    /var/log/filter/filter_20231206.log
532M    /var/log/filter/filter_20231128.log
531M    /var/log/filter/filter_20231205.log
529M    /var/log/filter/filter_20231130.log
522M    /var/log/filter/filter_20231207.log
512M    /var/log/filter/filter_20231204.log
509M    /var/log/filter/filter_20231123.log


[m4rtin]

At the risk of stating the obvious, did you use some reliable method to check the disk space usage first? Cannot even make sense of where does the graph come from in the original post.

Code Select Expand


# zpool list
# df -h

Some more notes:

- Those netflow DBs and logs can eat entire disk space easily. Get some decent storage before enabling it. If unable, disable and reset netflow data.
- You seem to be running the (absolutely horrlble) MongoDB thing on your firewall. For what? Yuck.
- Collecting half gig of firewall logs a day - what's the log retention set to.

Finally: have you ever rebooted the box after deleting those mongdb and whatnot files you mentioned earlier?

Hi!

The graph is from zabbix monitoring. I installed the plugin in the opnsense, so I can monitor the space. I attached the used space.

Code Select Expand

root@OPNsense:~ # zpool list
no pools available
root@OPNsense:~ # df -h
Filesystem                  Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs             115G     80G     26G    76%    /
devfs                       1.0K    1.0K      0B   100%    /dev
/dev/gpt/efifs              256M    1.7M    254M     1%    /boot/efi
devfs                       1.0K    1.0K      0B   100%    /var/dhcpd/dev
/dev/md43                    48M     24K     44M     0%    /usr/local/zenarmor/output/active/temp
devfs                       1.0K    1.0K      0B   100%    /var/unbound/dev
/usr/local/lib/python3.9    115G     80G     26G    76%    /var/unbound/usr/local/lib/python3.9


The MongoDB is from the sensei plugin. In the meantime I deleted sensei and reinstalled it. I keep its data for 2 days.

Edit: retention days are set to: 7