62GB of query.csv in /var/cache/unbound.duckdb

[anicoletti]

We received notification via Zabbix that one of our OPNsense firewalls was at 10% disk space free. We attempted to connect to it but the WebGUI was failing to load. We were able to access it via SSH, and upon running df we noticed the system was completely full. I manually deleted a few log files and restarted the WebGUI to get logged in. We had this issue about 2 months ago with this location but we actually rebuilt the firewall completely on new hardware, just restoring the original configuration.

After reviewing this issue further today, I went ahead and purged the rest of the logs, including RRD and Netflow data, but there was still 62GB of unaccounted for space used.

Ended up hopping back onto the shell and ran the following command:

Code Select Expand

du -h / | grep '[0-9\.]\+G'

The results showed that 62G was under /var/cache/unbound.duckdb. On checking that folder, I found these files.

Code Select Expand

-rw-r--r--  1 root     unbound         2888 May 21 08:45 client.csv
-rw-r--r--  1 unbound  unbound          214 May 20 08:44 load.sql
-rw-r--r--  1 unbound  unbound  33267605145 May 20 08:44 query.csv
-rw-r--r--  1 unbound  unbound         1503 May 20 08:44 schema.sql
-rw-r--r--  1 root     unbound  33726476128 May 21 08:45 tmp_query.csv

Two query.csv files totalling 62GB seems a bit off to me. Any ideas on why these got so bad and how to prevent this issue in the future?

[Patrick M. Hausen]

Did you enable query logging in Unbound?

[anicoletti]

Yes, but I would assume (and that might be my problem) it followed log retention policies like the other logs.

[Patrick M. Hausen]

I don't know but logging every single DNS request is a heck of a lot of data.

[franco]

/var/cache/unbound.duckdb is just the database dump that's required for DuckDB going from one database-incompatible version to the next.

The folder should have been dropped, but might as easily be dropped manually.


Cheers,
Franco

[anicoletti]

Thanks for the clarification. I'll go and remove this file from our firewalls if they exist.