Show Posts
1
21.1 Legacy Series / Limit size for '/var RAM disk' option
« on: July 01, 2021, 08:10:02 am »
I have been testing OPNsense in a virtualised environment (KVM) with modest resources (10GB disk, 4GB memory, 4 cores). The logic being it if behaves well here, it'll work just fine with more resources.
When investigating how OPNsense handles writing logs to disk, I enabled the '/var RAM disk' option in 'System: Settings: Miscellaneous'. With conservative logging options set in 'System: Settings: Logging' ('Disable circular logs': checked, 'Preserve logs (Days)': 5) and only basic services running on the system (pf, dhcp, unbound DNS), this worked just fine.
The problem came with enabling more services. Specifically NetFlow (required by 'Reporting: Insight'), ntopng (installed as a plugin) and redis (a required dependency of ntopng). Everything would work fine for a while, but certain aspects of the system would become unresponsive after some time had passed. Most noticeably unbound refused to respond to queries. The services widget on the dashboard would show unbound and flowd_aggregate as stopped. Manually restarting them again would work for a few minutes, but eventually they would stop again. The dashboard also showed high memory usage and disk usage on `/var`; not surprising given the additional services running on the system.
The only place that provided a hint as to what was happening was the OPNsense console: the system was running out of memory and killing processes. Unfortunately I didn't think to save the exact error messages at the time. (Does OPNsense regularly save `dmesg` output to a file?)
NetFilter, ntopng and redis obviously require memory to operate. The compounding factor is those services persisting a significant amount of data to memory-backed storage. In the case of NetFilter, it's actually a spiral of doom: `/var/log/flowd.log*` files produced by NetFliter are rotated by `flowd_aggregate`, which gets killed due to low memory conditions.
Looking at how the '/var RAM disk' option works, it simply triggers creation of tmpfs with minimal options. I.e.
For reference, enabling '/tmp RAM disk' does something similar
When no `size` parameter is specified, `tmpfs` defaults to using all of the available memory.
Understandably, having this particular combination of memory- and disk-hungry applications running on a system with constrained resources isn't a great idea. However I imagine there are legitimate cases where having a tmpfs-backed `/var` is desirable, and it would be preferable to exhaust logging space before the system runs out of memory and starts killing off processes.
When using '/var RAM disk' option in 'System: Settings: Miscellaneous', can tmpfs be capped at a size less than total system memory? Preferably user defined, either expressed as a percentage of system memory or an explicit value.
When investigating how OPNsense handles writing logs to disk, I enabled the '/var RAM disk' option in 'System: Settings: Miscellaneous'. With conservative logging options set in 'System: Settings: Logging' ('Disable circular logs': checked, 'Preserve logs (Days)': 5) and only basic services running on the system (pf, dhcp, unbound DNS), this worked just fine.
The problem came with enabling more services. Specifically NetFlow (required by 'Reporting: Insight'), ntopng (installed as a plugin) and redis (a required dependency of ntopng). Everything would work fine for a while, but certain aspects of the system would become unresponsive after some time had passed. Most noticeably unbound refused to respond to queries. The services widget on the dashboard would show unbound and flowd_aggregate as stopped. Manually restarting them again would work for a few minutes, but eventually they would stop again. The dashboard also showed high memory usage and disk usage on `/var`; not surprising given the additional services running on the system.
The only place that provided a hint as to what was happening was the OPNsense console: the system was running out of memory and killing processes. Unfortunately I didn't think to save the exact error messages at the time. (Does OPNsense regularly save `dmesg` output to a file?)
NetFilter, ntopng and redis obviously require memory to operate. The compounding factor is those services persisting a significant amount of data to memory-backed storage. In the case of NetFilter, it's actually a spiral of doom: `/var/log/flowd.log*` files produced by NetFliter are rotated by `flowd_aggregate`, which gets killed due to low memory conditions.
Looking at how the '/var RAM disk' option works, it simply triggers creation of tmpfs with minimal options. I.e.
Code: [Select]
mount -t tmpfs tmpfs /var
For reference, enabling '/tmp RAM disk' does something similar
Code: [Select]
mount -t tmpfs -o mode=01777 tmpfs /tmpWhen no `size` parameter is specified, `tmpfs` defaults to using all of the available memory.
Quote
size Specifies the total file system size in bytes, unless
suffixed with one of k, m, g, t, or p, which denote
byte, kilobyte, megabyte, gigabyte, terabyte and
petabyte respectively. If zero (the default) or a
value larger than SIZE_MAX - PAGE_SIZE is given, the
available amount of memory (including main memory and
swap space) will be used.
Understandably, having this particular combination of memory- and disk-hungry applications running on a system with constrained resources isn't a great idea. However I imagine there are legitimate cases where having a tmpfs-backed `/var` is desirable, and it would be preferable to exhaust logging space before the system runs out of memory and starts killing off processes.
When using '/var RAM disk' option in 'System: Settings: Miscellaneous', can tmpfs be capped at a size less than total system memory? Preferably user defined, either expressed as a percentage of system memory or an explicit value.