How to prevent filter logs from filling up disk space

[geotek]

Under heavy load, the log files in /var/log/filter are sometimes filling up all my disk space.

I have already configured the "maximum file size" and the "maximum preserved files" under System / Settings / Local, but the filter logs are getting much larger than they should, so this setting does not seem to have any effect on the number and size of the filter logs.

Is this behavior intended? If so, is there another way to rotate filter logs so that there is no risk of running out of disk space?

Shutting off the logs completely is no option for me.

I am using OPNsense 25.1.5_5-amd64

[pfry]

How heavy is your load vs. disk space?

For context, I have all filter logging enabled with a 200 file limit on a 1.6TB SSD, and over ~5 months it's written ~367GB according to SMART, equating to (apparently, via df) ~1.9GB in logs. My firewall filters for 13 static IPs on a 500Mb Internet link and varies between 150 to 4000 sessions, with pfrate varying between 5 and 40 (from Reporting: Health / System / States) (instantaneous stats will vary more). This seems like a very light logging load to me, as I designed the system to handle... well, a lot more.

[geotek]

On average the filter logs are only 2 GBytes per day, but I have already observed 5 GBytes per hour on some occasions. When I get a disk space warning from my network management it is usually already too late and the GUI is inoperable because of low disk space.

I would expect that local logging is intelligent enough to just delete the oldest log files if the disk is in danger of running out of space, but apparently this is not the case.

[pfry]

[...]
I would expect that local logging is intelligent enough to just delete the oldest log files if the disk is in danger of running out of space, but apparently this is not the case.

I agree with this statement. (I have not tested the behavior on my own system, of course.)

From a practical standpoint, though, I don't see how you utilize your logs. It looks like you're dealing with 2-3 orders of magnitude more logs than I am, and I find my logs to be borderline unsearchable (~30s response time for complete "no limit"). Capturing logs at that rate is not a problem, but with the current on-device logging system you end up with largely write-only storage. How do you deal with that?

[cookiemonster]

> I would expect that local logging is intelligent enough to just delete the oldest log files if the disk is in danger of running out of space, but apparently this is not the case.
Since is an expectation, do you have an example of a system that does that?

[geotek]

do you have an example of a system that does that?

Systemd does it with the SystemKeepFree Option.

But this is not the point. My original question was, why "maximum file size" and the "maximum preserved files" don't seem to work as expected. My expectation is that if I specify a maximum log file size that no log files exceeding this size limit are created.

[_tribal_]

I'm also interested in this question, I've already disabled the log of everything I can, but it continues to grow by gigabytes per day.

[Patrick M. Hausen]

But this is not the point. My original question was, why "maximum file size" and the "maximum preserved files" don't seem to work as expected. My expectation is that if I specify a maximum log file size that no log files exceeding this size limit are created.

The logfiles are rotated by a cron job every hour. The "maximum file size" tells the system to rotate and compress any file that's over that particular size. I you really manage to fill the disk within a single hour ... no chance.

Are you running ZFS? Compression is the default for ZFS datasets which helps with log files a lot.

[_tribal_]

The "maximum file size" tells the system to rotate and compress any file that's over that particular size.

nope. Neither the size of the log files with the settings from the menu does not match, and none of them are placed in the archive as you write.
File system is of course ZFS

Code Select Expand

root@OPNsense:~ # ls -lh /var/log/filter
total 234881
-rw-------  1 root wheel   47M Jun 12 17:01 filter_20250612.0017.log
-rw-------  1 root wheel   48M Jun 12 18:01 filter_20250612.0018.log
-rw-------  1 root wheel   45M Jun 12 19:01 filter_20250612.0019.log
-rw-------  1 root wheel   32M Jun 12 20:00 filter_20250612.0020.log
-rw-------  1 root wheel   47M Jun 12 21:00 filter_20250612.0021.log
-rw-------  1 root wheel   33M Jun 12 22:00 filter_20250612.0022.log
-rw-------  1 root wheel   41M Jun 12 23:01 filter_20250612.0023.log
-rw-------  1 root wheel   46M Jun 12 23:59 filter_20250612.log
-rw-------  1 root wheel   45M Jun 13 01:01 filter_20250613.0001.log
-rw-------  1 root wheel   42M Jun 13 02:01 filter_20250613.0002.log
-rw-------  1 root wheel   32M Jun 13 03:01 filter_20250613.0003.log
-rw-------  1 root wheel   21M Jun 13 04:00 filter_20250613.0004.log
-rw-------  1 root wheel   38M Jun 13 05:01 filter_20250613.0005.log
-rw-------  1 root wheel   36M Jun 13 06:00 filter_20250613.0006.log
-rw-------  1 root wheel   25M Jun 13 07:01 filter_20250613.0007.log
-rw-------  1 root wheel   36M Jun 13 08:01 filter_20250613.0008.log
-rw-------  1 root wheel   60M Jun 13 10:00 filter_20250613.0009.log
-rw-------  1 root wheel   44M Jun 13 11:01 filter_20250613.0010.log
-rw-------  1 root wheel   43M Jun 13 12:01 filter_20250613.0011.log
-rw-------  1 root wheel   47M Jun 13 13:00 filter_20250613.0012.log
-rw-------  1 root wheel   47M Jun 13 14:00 filter_20250613.0013.log
-rw-------  1 root wheel   46M Jun 13 15:00 filter_20250613.0014.log
-rw-------  1 root wheel   48M Jun 13 16:01 filter_20250613.0015.log
-rw-------  1 root wheel   47M Jun 13 17:01 filter_20250613.0016.log
-rw-------  1 root wheel   47M Jun 13 18:01 filter_20250613.0017.log
-rw-------  1 root wheel   48M Jun 13 19:00 filter_20250613.0018.log
-rw-------  1 root wheel   33M Jun 13 20:01 filter_20250613.0019.log
-rw-------  1 root wheel   40M Jun 13 21:00 filter_20250613.0020.log
-rw-------  1 root wheel   50M Jun 13 22:01 filter_20250613.0021.log
-rw-------  1 root wheel   48M Jun 13 23:01 filter_20250613.0022.log
-rw-------  1 root wheel   44M Jun 13 23:59 filter_20250613.log
-rw-------  1 root wheel  660K Jun 14 00:00 filter_20250614.log


[Patrick M. Hausen]

Look at th3 time stamps - they are rotated every hour if they are above the configured size.

[cookiemonster]

I think he's asking why the file is not rotated when reached the specified size without waiting for the turn of the hour. He would expect to see a number of files of circa 20 Mb within the hour, any hour but no more than 31.

[Patrick M. Hausen]

Because files are only ever rotated at the full hour. Additionally they must be above the configured size threshold.

[cookiemonster]

Yes that is clear. I don't know how to change the behaviour to what he wants. I've seen it done but it wasn't newsyslog but a custom logging library on another type of system. I used to manage a team that looked after these systems and we changed the config file to make it do that.
Essentially the defined file size triggered the rotation, not the time.
Edit: logrotate can do it. that's what I use in ubuntu linux, just remembered.

[_tribal_]

they are rotated every hour if they are above the configured size.

You wrote above that it is compressed to reduce the size, I don't see any archives in the log directory (except for nginx logs and a few other applications). These are just text strings and archiving would significantly reduce the size of the files, and freebsd logging settings allow you to do this, why this functionality is not used is another question.

[Patrick M. Hausen]

You wrote above that it is compressed to reduce the size, I don't see any archives in the log directory

They used to be compressed - when you set up with ZFS that is not necessary, anymore, because ZFS compresses transparently and it is much easier to e.g. search for things with grep/awk and friends.