Under heavy load, the log files in /var/log/filter are sometimes filling up all my disk space.
I have already configured the "maximum file size" and the "maximum preserved files" under System / Settings / Local, but the filter logs are getting much larger than they should, so this setting does not seem to have any effect on the number and size of the filter logs.
Is this behavior intended? If so, is there another way to rotate filter logs so that there is no risk of running out of disk space?
Shutting off the logs completely is no option for me.
I am using OPNsense 25.1.5_5-amd64
How heavy is your load vs. disk space?
For context, I have all filter logging enabled with a 200 file limit on a 1.6TB SSD, and over ~5 months it's written ~367GB according to SMART, equating to (apparently, via df) ~1.9GB in logs. My firewall filters for 13 static IPs on a 500Mb Internet link and varies between 150 to 4000 sessions, with pfrate varying between 5 and 40 (from Reporting: Health / System / States) (instantaneous stats will vary more). This seems like a very light logging load to me, as I designed the system to handle... well, a lot more.
On average the filter logs are only 2 GBytes per day, but I have already observed 5 GBytes per hour on some occasions. When I get a disk space warning from my network management it is usually already too late and the GUI is inoperable because of low disk space.
I would expect that local logging is intelligent enough to just delete the oldest log files if the disk is in danger of running out of space, but apparently this is not the case.
Quote from: geotek on April 25, 2025, 10:43:28 AM[...]
I would expect that local logging is intelligent enough to just delete the oldest log files if the disk is in danger of running out of space, but apparently this is not the case.
I agree with this statement. (I have not tested the behavior on my own system, of course.)
From a practical standpoint, though, I don't see how you utilize your logs. It looks like you're dealing with 2-3 orders of magnitude more logs than I am, and I find my logs to be borderline unsearchable (~30s response time for complete "no limit"). Capturing logs at that rate is not a problem, but with the current on-device logging system you end up with largely write-only storage. How do you deal with that?
> I would expect that local logging is intelligent enough to just delete the oldest log files if the disk is in danger of running out of space, but apparently this is not the case.
Since is an expectation, do you have an example of a system that does that?
Quote from: cookiemonster on April 25, 2025, 05:57:19 PMdo you have an example of a system that does that?
Systemd does it with the SystemKeepFree Option.
But this is not the point. My original question was, why "maximum file size" and the "maximum preserved files" don't seem to work as expected. My expectation is that if I specify a maximum log file size that no log files exceeding this size limit are created.
I'm also interested in this question, I've already disabled the log of everything I can, but it continues to grow by gigabytes per day.
Quote from: geotek on June 10, 2025, 06:53:11 PMBut this is not the point. My original question was, why "maximum file size" and the "maximum preserved files" don't seem to work as expected. My expectation is that if I specify a maximum log file size that no log files exceeding this size limit are created.
The logfiles are rotated by a cron job every hour. The "maximum file size" tells the system to rotate and compress any file that's over that particular size. I you really manage to fill the disk within a single hour ... no chance.
Are you running ZFS? Compression is the default for ZFS datasets which helps with log files a lot.
Quote from: Patrick M. Hausen on June 10, 2025, 10:01:39 PMThe "maximum file size" tells the system to rotate and compress any file that's over that particular size.
nope. Neither the size of the log files with the settings from the menu does not match, and none of them are placed in the archive as you write.
File system is of course ZFS
root@OPNsense:~ # ls -lh /var/log/filter
total 234881
-rw------- 1 root wheel 47M Jun 12 17:01 filter_20250612.0017.log
-rw------- 1 root wheel 48M Jun 12 18:01 filter_20250612.0018.log
-rw------- 1 root wheel 45M Jun 12 19:01 filter_20250612.0019.log
-rw------- 1 root wheel 32M Jun 12 20:00 filter_20250612.0020.log
-rw------- 1 root wheel 47M Jun 12 21:00 filter_20250612.0021.log
-rw------- 1 root wheel 33M Jun 12 22:00 filter_20250612.0022.log
-rw------- 1 root wheel 41M Jun 12 23:01 filter_20250612.0023.log
-rw------- 1 root wheel 46M Jun 12 23:59 filter_20250612.log
-rw------- 1 root wheel 45M Jun 13 01:01 filter_20250613.0001.log
-rw------- 1 root wheel 42M Jun 13 02:01 filter_20250613.0002.log
-rw------- 1 root wheel 32M Jun 13 03:01 filter_20250613.0003.log
-rw------- 1 root wheel 21M Jun 13 04:00 filter_20250613.0004.log
-rw------- 1 root wheel 38M Jun 13 05:01 filter_20250613.0005.log
-rw------- 1 root wheel 36M Jun 13 06:00 filter_20250613.0006.log
-rw------- 1 root wheel 25M Jun 13 07:01 filter_20250613.0007.log
-rw------- 1 root wheel 36M Jun 13 08:01 filter_20250613.0008.log
-rw------- 1 root wheel 60M Jun 13 10:00 filter_20250613.0009.log
-rw------- 1 root wheel 44M Jun 13 11:01 filter_20250613.0010.log
-rw------- 1 root wheel 43M Jun 13 12:01 filter_20250613.0011.log
-rw------- 1 root wheel 47M Jun 13 13:00 filter_20250613.0012.log
-rw------- 1 root wheel 47M Jun 13 14:00 filter_20250613.0013.log
-rw------- 1 root wheel 46M Jun 13 15:00 filter_20250613.0014.log
-rw------- 1 root wheel 48M Jun 13 16:01 filter_20250613.0015.log
-rw------- 1 root wheel 47M Jun 13 17:01 filter_20250613.0016.log
-rw------- 1 root wheel 47M Jun 13 18:01 filter_20250613.0017.log
-rw------- 1 root wheel 48M Jun 13 19:00 filter_20250613.0018.log
-rw------- 1 root wheel 33M Jun 13 20:01 filter_20250613.0019.log
-rw------- 1 root wheel 40M Jun 13 21:00 filter_20250613.0020.log
-rw------- 1 root wheel 50M Jun 13 22:01 filter_20250613.0021.log
-rw------- 1 root wheel 48M Jun 13 23:01 filter_20250613.0022.log
-rw------- 1 root wheel 44M Jun 13 23:59 filter_20250613.log
-rw------- 1 root wheel 660K Jun 14 00:00 filter_20250614.log
Look at th3 time stamps - they are rotated every hour if they are above the configured size.
I think he's asking why the file is not rotated when reached the specified size without waiting for the turn of the hour. He would expect to see a number of files of circa 20 Mb within the hour, any hour but no more than 31.
Because files are only ever rotated at the full hour. Additionally they must be above the configured size threshold.
Yes that is clear. I don't know how to change the behaviour to what he wants. I've seen it done but it wasn't newsyslog but a custom logging library on another type of system. I used to manage a team that looked after these systems and we changed the config file to make it do that.
Essentially the defined file size triggered the rotation, not the time.
Edit: logrotate can do it. that's what I use in ubuntu linux, just remembered.
Quote from: Patrick M. Hausen on June 13, 2025, 11:31:13 PMthey are rotated every hour if they are above the configured size.
You wrote above that it is compressed to reduce the size, I don't see any archives in the log directory (except for nginx logs and a few other applications). These are just text strings and archiving would significantly reduce the size of the files, and freebsd logging settings allow you to do this, why this functionality is not used is another question.
Quote from: _tribal_ on June 17, 2025, 02:46:10 PMYou wrote above that it is compressed to reduce the size, I don't see any archives in the log directory
They used to be compressed - when you set up with ZFS that is not necessary, anymore, because ZFS compresses transparently and it is much easier to e.g. search for things with grep/awk and friends.
Quote from: Patrick M. Hausen on June 17, 2025, 02:58:44 PMThey used to be compressed - when you set up with ZFS
Thanks for the explanation, I didn't take note of the specifics of how ZFS works.