[SOLVED] Continuous growing arc size

[Raketenmeyer]

Hey,

I'm currently running 23.7.8_1 on a DEC740 and noted, that the ARC size is growing continuously. I don't know after which update that started exactly - I think it was 23.7.6, but I may be wrong. After a reboot it looks "normal" with about 300 MB ARC size, but then it starts growing. Yesterday it was around 2050 MB, just now I see 2385 MB. So currently the total RAM usage is 3263 MB (82%).

Is this expected or it here something going wrong?

The health check looks ok.

Code: [Select]

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 23.7.8_1 at Wed Nov 15 16:28:09 CET 2023
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 23.7.8 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 23.7.8 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense
>>> Check installed plugins
os-ddclient 1.16_2
os-mdns-repeater 1.1_1
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" has 69 dependencies to check.
Checking packages: ...................................................................... done
***DONE***

[Raketenmeyer]

Small update - the ARC size just dropped down to 113 MB and starts growing again.

[marunjar]

Unused memory is wasted memory
This behavior is totally fine for ARC, it will be freed if other processes needs more memory.

[Raketenmeyer]

Yes, I know that ZFS uses free memory for caching, but I find it unusual that
the ARC size is growing till most available memory is used - then dropping nearly
all used RAM and starting to grow again. In an endless loop, producing sawtooth
graphs. The DEC740 is running in my home - idling most time.

The two DEC3850 running in our company (opnsense business) don‘t show this
behavior - both with a quite stable ARC size around 2 GB.

[xavx]

Suggest you read the pfsense page about arc tuning so you can limit its usage https://docs.netgate.com/pfsense/en/latest/hardware/tune-zfs.html

[meyergru]

I can see a similar behaviour, but ARC seems to be only a symptom: It will fill up memory slowly e.g. because of writing logs. What causes the sawtooth must be something that is started regularly and uses much RAM, thus ARC gets evicted.

In my case, there are memory spikes at 00:20, 08:20 and 16:20. I have looked at the cronjobs, but saw nothing obvious with that rhythm. It may well be any cron job that is started more often than every 8 hours.

[Raketenmeyer]

Thanks @meyergru - that pointed me to the right direction. There were several devices in my IOT vlan spamming the firewall logs with ntp requests to a not reachable host. A restart of these devices solved this problem - and now also the ARC size seems to be under control again.