OPNsense runs out of space

[m4rtin]

Hi,

my OPNsense runs out of space since about September. I don't know what exactly I changed but since then the free space declines about 16 G in 2 weeks (see screenshot).

I one removed some log files but it didn't fully solve the problem.

Code: [Select]

du -sh doesn't seem to show all files (see screnshots).

Do you have any idea?

[Patrick M. Hausen]

https://forum.opnsense.org/index.php?topic=34791.msg168539#msg168539

[CJ]

https://forum.opnsense.org/index.php?topic=34791.msg168539#msg168539

Why that set of options?  I would think these would be easier.

Code: [Select]

du -h / | sort -rh | head
Or you could just skip the -r and head completely and look at the bottom.

[Patrick M. Hausen]

-s sum up the directories
-k report in kbytes
-x don't cross file system mount points

If you use -h, sort -n will sort 1G before 2M before 3k ...

[m4rtin]

Thanks for the hint! I found an unues mongodb and some logfiles. In total about 20 GB. Now there are still 60 GB used if I look on the webinterface or 20 GB if I look in the shell, but I can live with that as long as it doesn't get more.

[cookiemonster]

Don't forget to check System > Settings > Logging. Tooltips for the fields are useful. The most important in my opinion is the number of days to preserve, as it ensures the log file rotation keeps storage on reasonable levels, depending on the need/want for history.

[CJ]

-s sum up the directories
-k report in kbytes
-x don't cross file system mount points

If you use -h, sort -n will sort 1G before 2M before 3k ...

Not if you add -h to the sort.

[Patrick M. Hausen]

I honestly did not know sort can do that. I am using my variant for over 30 years, now and did not see a reason to change it. 

[CJ]

I honestly did not know sort can do that. I am using my variant for over 30 years, now and did not see a reason to change it. 

 

[m4rtin]

I now changed the preserve days to 7, but still 83 GB seemes to be in use (dashboard).

But the command only outputs 29 GB in /:

Code: [Select]

root@OPNsense:/var/log # du -ah / | sort -rh | head -n 20
 29G    /
 25G    /var
 23G    /var/log
 13G    /var/log/flowd.log
9.3G    /var/log/filter
4.2G    /usr
2.0G    /usr/swap0
1.7G    /usr/local
1.1G    /var/netflow
666M    /var/log/suricata
607M    /usr/local/lib
547M    /var/log/filter/filter_20231129.log
540M    /var/log/filter/filter_20231127.log
534M    /var/log/filter/filter_20231206.log
532M    /var/log/filter/filter_20231128.log
531M    /var/log/filter/filter_20231205.log
529M    /var/log/filter/filter_20231130.log
522M    /var/log/filter/filter_20231207.log
512M    /var/log/filter/filter_20231204.log
509M    /var/log/filter/filter_20231123.log

Do you know where the other 54 GB may hide?   

[dinguz]

try as root/admin user:

Code: [Select]

find / -type f -exec du -h {} + | sort -h

[m4rtin]

That's basically the same:

Code: [Select]

40M    /usr/local/etc/suricata/rules/rules.sqlite
 42M    /usr/local/datastore/mongodb/mongod.log
 43M    /usr/local/lib/python3.9/site-packages/duckdb.cpython-39.so
 43M    /var/log/suricata/eve.json
 43M    /var/unbound/usr/local/lib/python3.9/site-packages/duckdb.cpython-39.so
 47M    /usr/local/bin/mongod
 55M    /usr/bin/ld.lld
 56M    /usr/local/zenarmor/bin/ipdrstreamer
 58M    /var/netflow/interface_000030.sqlite
 64M    /usr/local/zenarmor/db/GeoIP/GeoLite2-City.mmdb
 79M    /var/netflow/dst_port_003600.sqlite
 81M    /usr/bin/c++
 83M    /usr/bin/lldb
 92M    /var/log/suricata/eve.json.2
100M    /usr/local/datastore/mongodb/journal/WiredTigerLog.0000000001
100M    /usr/local/datastore/mongodb/journal/WiredTigerPreplog.0000000001
100M    /usr/local/datastore/mongodb/journal/WiredTigerPreplog.0000000002
102M    /var/log/suricata/eve.json.3
103M    /var/log/suricata/eve.json.1
108M    /var/log/suricata/eve.json.0
112M    /var/netflow/dst_port_086400.sqlite
130M    /var/netflow/dst_port_000300.sqlite
164M    /var/log/filter/filter_20231213.log
264M    /var/netflow/src_addr_086400.sqlite
417M    /var/log/filter/filter_20231210.log
418M    /var/log/filter/filter_20231209.log
447M    /var/netflow/src_addr_details_086400.sqlite
479M    /var/log/filter/filter_20231208.log
493M    /var/log/filter/filter_20231211.log
498M    /var/log/filter/filter_20231212.log
522M    /var/log/filter/filter_20231207.log
2.0G    /usr/swap0
 13G    /var/log/flowd.log


[doktornotor]

At the risk of stating the obvious, did you use some reliable method to check the disk space usage first? Cannot even make sense of where does the graph come from in the original post.

Code: [Select]

# zpool list
# df -h
Some more notes:

- Those netflow DBs and logs can eat entire disk space easily. Get some decent storage before enabling it. If unable, disable and reset netflow data.
- You seem to be running the (absolutely horrlble) MongoDB thing on your firewall. For what? Yuck.
- Collecting half gig of firewall logs a day - what's the log retention set to.

Finally: have you ever rebooted the box after deleting those mongdb and whatnot files you mentioned earlier?

[doktornotor]

I mean, with ZFS in place with compression enabled, we are not even getting meaningful figures here. Consider:

# man du

Code: [Select]

     -A      Display the apparent size instead of the disk usage.  This can be
             helpful when operating on compressed volumes or sparse files.


Code: [Select]

# find /var/log/filter -type f -exec du -Ah {} + | sort -h
9.2M    /var/log/filter/filter_20231213.log
 17M    /var/log/filter/filter_20231210.log
 23M    /var/log/filter/filter_20231204.log
 24M    /var/log/filter/filter_20231211.log
 30M    /var/log/filter/filter_20231212.log
 58M    /var/log/filter/filter_20231206.log
 59M    /var/log/filter/filter_20231209.log
 75M    /var/log/filter/filter_20231207.log
 92M    /var/log/filter/filter_20231208.log
 95M    /var/log/filter/filter_20231205.log

vs.

Code: [Select]

# find /var/log/filter -type f -exec du -h {} + | sort -h
1.4M    /var/log/filter/filter_20231213.log
1.9M    /var/log/filter/filter_20231210.log
3.1M    /var/log/filter/filter_20231204.log
3.1M    /var/log/filter/filter_20231211.log
4.2M    /var/log/filter/filter_20231212.log
8.1M    /var/log/filter/filter_20231209.log
8.2M    /var/log/filter/filter_20231206.log
 11M    /var/log/filter/filter_20231207.log
 13M    /var/log/filter/filter_20231205.log
 13M    /var/log/filter/filter_20231208.log

So, e.g. those firewall log files here you listed, they are actually not half gig, but ~5G per day.

Code: [Select]

547M    /var/log/filter/filter_20231129.log
540M    /var/log/filter/filter_20231127.log
534M    /var/log/filter/filter_20231206.log
532M    /var/log/filter/filter_20231128.log
531M    /var/log/filter/filter_20231205.log
529M    /var/log/filter/filter_20231130.log
522M    /var/log/filter/filter_20231207.log
512M    /var/log/filter/filter_20231204.log
509M    /var/log/filter/filter_20231123.log


[m4rtin]

At the risk of stating the obvious, did you use some reliable method to check the disk space usage first? Cannot even make sense of where does the graph come from in the original post.

Code: [Select]

# zpool list
# df -h
Some more notes:

- Those netflow DBs and logs can eat entire disk space easily. Get some decent storage before enabling it. If unable, disable and reset netflow data.
- You seem to be running the (absolutely horrlble) MongoDB thing on your firewall. For what? Yuck.
- Collecting half gig of firewall logs a day - what's the log retention set to.

Finally: have you ever rebooted the box after deleting those mongdb and whatnot files you mentioned earlier?

Hi!

The graph is from zabbix monitoring. I installed the plugin in the opnsense, so I can monitor the space. I attached the used space.

Code: [Select]

root@OPNsense:~ # zpool list
no pools available
root@OPNsense:~ # df -h
Filesystem                  Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs             115G     80G     26G    76%    /
devfs                       1.0K    1.0K      0B   100%    /dev
/dev/gpt/efifs              256M    1.7M    254M     1%    /boot/efi
devfs                       1.0K    1.0K      0B   100%    /var/dhcpd/dev
/dev/md43                    48M     24K     44M     0%    /usr/local/zenarmor/output/active/temp
devfs                       1.0K    1.0K      0B   100%    /var/unbound/dev
/usr/local/lib/python3.9    115G     80G     26G    76%    /var/unbound/usr/local/lib/python3.9

The MongoDB is from the sensei plugin. In the meantime I deleted sensei and reinstalled it. I keep its data for 2 days.

Edit: retention days are set to: 7