Is this a bug? var/log filling up 4gb of logs in days.

[idiocracy]

I've used pfsense for the past 7 years. Installed opnsense on a new box about 5 days ago. Heard good things about it. So far so good.
It's very different.

One of the settings i've run with, which i thought i'd do again, is using memory for var/log and tmp. Old pfsense had 2gb memory and that worked just fine. New box has 8gb, so couldn't see a reason why that wouldn't work.

But, there seems to be something off with using memory for var/log. This is what dashboard looks like.



First thing, why is there reserved 4gb for logs? That seems like a lot.
And why is it filling up so fast, in this picture, you can see uptime is 3 days. So that's 3.7gb logs in 3 days.

So i messed about a bit. Eventually i found out, that if i go to firewall/logs/plain view and clear it. That'll empty var/log.
The max appears to be 5000 entries. I tried timing it, and it took about 4 min to fill up with 5000 new entries after being cleared.

What happens when it hits 100%?

[bartjsmit]

why is there reserved 4gb for logs?

4 GB is used for logs. How do you conclude that it is reserved? Would you rather it used 1 GB and leave the other 3 GB unused?

It is good security practice to log to a separate collector. This allows correlation with other parts of the network and stops an attacker hiding their tracks.

[idiocracy]

why is there reserved 4gb for logs?

4 GB is used for logs. How do you conclude that it is reserved? Would you rather it used 1 GB and leave the other 3 GB unused?

It is good security practice to log to a separate collector. This allows correlation with other parts of the network and stops an attacker hiding their tracks.

I imagine it's reserved because it's saying X out of 3.9gb, and bad things would happen if it tried using memory that was not available.
But you're saying it's normal. And if i understand correctly, it being 100% is actually what you want.
Running pfsense, it would use ~50/100mb on logs. Going from that to 4gb seemed like a big jump, considering that it takes ~4 days to fill up.

But i should just leave it, and let it do it's thing.

In the logs, it can show 5000 entries. This fills up in 4min. After that i am assuming the entries in the log viewer are being replaced by newer ones, but the log viewer still only shows 5000, even tho the logs continue to grow. Downloading the logs can only download those you select.
So why are we spending memory on logs that are no longer present in the log viewer, and can't be accessed?

[yourfriendarmando]

something is filling up the logs very fast, I wish there was a way to see each log area and get the totals of each size.

For now you need to do the following to see which one is swelling up. Log in to the box via console, ssh, serial.

cd /var/log
du -hscx -- *

From there it will depend what you need to address, maybe you have disk issues, network card causing too many interrupts, someone is trying to break in through the firewall, you have a device constantly requesting an ipv6 address

When you see which one is largest, it will have a latest.log, dmesg, syslog etc.

You can run this to watch the log and keep refreshing even if the file is rotated:

tail -F logFile

[idiocracy]

I'm not really experiencing any problems with the internet connection, everything appears to work as intended. Right now, it just appears as if logging in opnsense is broken.

Imagine if i had not changed logging to use memory, it would hammer my ssd and fill it up in like two weeks. That can't be normal.
If it was a disk issue, how would that affect stuff the firewall puts in memory?
Also, wouldn't smart notice this?

ipv6 is disabled.

For now you need to do the following to see which one is swelling up. Log in to the box via console, ssh, serial.

cd /var/log
du -hscx -- *

I don't know what that means.

When you see which one is largest, it will have a latest.log, dmesg, syslog etc.

You can run this to watch the log and keep refreshing even if the file is rotated:

tail -F logFile

I do not know what any of that means.

If it was network interrupts. Couldn't we see that in reporting/health quality?

I found that var/log is by default set up to use 50% of available memory. /tmp is also by default set up to use 50% of available memory.
What happens if both of them does that, wouldn't that crash the router?
I set them both to 10%.
Still not sure what happens when var/log is 100%, does it affect anything?

[franco]

> So i messed about a bit. Eventually i found out, that if i go to firewall/logs/plain view and clear it. That'll empty var/log.

So you already know you have excessive firewall logging, maybe you enabled the logging option on a stateless rule or one of the default logging options (which are disabled by default because they cause excessive logging).


Cheers,
Franco

[idiocracy]

So you already know you have excessive firewall logging, maybe you enabled the logging option on a stateless rule or one of the default logging options (which are disabled by default because they cause excessive logging).


Cheers,
Franco

No i don't know that. I just found that if i do the things i said, something happened.
I do not know which logging options you are referring to. What is a stateless rule, and which logging options are disabled by default?
Don't have any logging options that were off, turned on. I just set it to use memory for less wear on ssd.

[idiocracy]

I had no idea this would be such a tough one. Posted somewhere else as well about this. They were equally baffled.

I figured out that the reason it's reserving or using 4gb aka half the memory capacity. Is because it's literally set to use 50% of memory, by default when enabled.
Same is tmp, it's also set to use 50%. I wonder what  happens if both of them uses 50% each at the same time. Hmm.

I did not notice this, because, as i recall. In pfsense you set the size in mb, not %. So when it said 50 in the box underneath, i didn't think twice about it and was like "sure, probably fine" and moved on.
I have now manually set both of them to 10%, which should be more than enough imo. One could also just go into system/settings/logging and disable logging. That would work too.

[franco]

While this will fix the space consumption it will break log generation quickly as the space fills up and is capped.

Just go to the firewall log, find which rule logs thousands of lines and disable logging for it.


Cheers,
Franco

[idiocracy]

Going to firewall, it doesn't actually tell you which rules are being used. Now, looking at the ports, i can figure it out.
It took me a while to figure out that the "i" in rules is clickable, can enable and disable logging. Just found out 5 min ago. But they're all disabled.

I don't know how else i can disable logging for that specific rule.

In firewall/settings/advanced there's option to disable packets logging for blocked and passed traffic. I do not know if it helps on space if you do not log the packets. By default these are enabled.

[franco]

There's 3 ways to display the firewall log. The easiest is the live view. It will tell you what is being logged and by which rule.


Cheers,
Franco

[idiocracy]

Looking at the label.

let out anything from firewall host itself   
let out anything from firewall host itself (force gw)

These two fill the majority.

[idiocracy]

I just found out that if you go to firewall/rules/lan or wan. On the right side of "automatically generated rule" there's a small box, that can be expanded. And inside of that, i find stuff where logging is enabled.
But you cannot disable logging for any of those.