Show Posts
1
22.1 Legacy Series / RAM consumption - out of space memory - Log management
« on: March 08, 2022, 10:57:34 am »
Hello everyone.
This is my first post here.
First of all, we'd like to thank OPNSense technical team for their work.
We would just want to share our last production incident appearing from last upgrades.
Some new logs (logs level?) have to be added recently.
Probably since 21.+ versions.
In fact since we upgraded from 20 to 21+ versions we observed some incredible RAM usage on some of our OPNSense.
The RAM was consumed with a perfect rising ramp.
We discovered then that logs are indeed write in RAM in:
/var/log/...
We should have known better.
Here, you can read the "pfsense" documentation:
https://docs.netgate.com/pfsense/en/latest/monitoring/logs/size.html
As you can read logs default size should be 500KB with a rotation and a maximum of 7 files if our understanding is correct. This might not be the case for OPNSense but I did not find this level of details in OPNSense documentation. The restrictions on log file size and rotation may not be implemented.
I precise that ofc we did not change anything in default logging policy/settings.
The thing is that we observe some +300 MB log file wrote in a day for some of our infrastructure in:
"/var/log/filter/filter_date.log"
The disk will get filled and the OPNSense crash with a spamming error:
"..... out of swap space"
So if anyone is experimenting this kind of behaviour.
You can of course mount /var/log on an external disk...
Or you can temporarily clear and disable logging in
>> system > settings > logging
Hope this help the few of us having this kind of issue with RAM management on most recent OPNSense without having, like us, the knowledge to fix it straight away.
Kind regards.
This is my first post here.
First of all, we'd like to thank OPNSense technical team for their work.
We would just want to share our last production incident appearing from last upgrades.
Some new logs (logs level?) have to be added recently.
Probably since 21.+ versions.
In fact since we upgraded from 20 to 21+ versions we observed some incredible RAM usage on some of our OPNSense.
The RAM was consumed with a perfect rising ramp.
We discovered then that logs are indeed write in RAM in:
/var/log/...
We should have known better.
Here, you can read the "pfsense" documentation:
https://docs.netgate.com/pfsense/en/latest/monitoring/logs/size.html
As you can read logs default size should be 500KB with a rotation and a maximum of 7 files if our understanding is correct. This might not be the case for OPNSense but I did not find this level of details in OPNSense documentation. The restrictions on log file size and rotation may not be implemented.
I precise that ofc we did not change anything in default logging policy/settings.
The thing is that we observe some +300 MB log file wrote in a day for some of our infrastructure in:
"/var/log/filter/filter_date.log"
The disk will get filled and the OPNSense crash with a spamming error:
"..... out of swap space"
So if anyone is experimenting this kind of behaviour.
You can of course mount /var/log on an external disk...
Or you can temporarily clear and disable logging in
>> system > settings > logging
Hope this help the few of us having this kind of issue with RAM management on most recent OPNSense without having, like us, the knowledge to fix it straight away.
Kind regards.