ARC RAM usage

[Isabella Borgward]

Just looking over my firewalls in central management, one firewall stands out above all the others with RAM usage at 85%. Every other box is 25-35%. It seems to be ARC, which I believe is part of ZFS.
All firewalls are on 26.4 or 25.10 and have 8GB of RAM. Why would ARC usage be so high on one firewall?

[nero355]

All firewalls are on 26.4 or 25.10 and have 8GB of RAM.

In theory ZFS should never use more than 50% of that IIRC so you should have nothing to worry about despite these values.

However...

Why would ARC usage be so high on one firewall?

Do all of them have Hostwatch Enabled or not ?

It's a process that's constantly active and gave many of us all sorts of issues after it's release and that's why many of us also have it set ti Disabled !!

So you could try that too and see if things will change ?
Might need a reboot just to be sure... :)

[Patrick M. Hausen]

In theory ZFS should never use more than 50% of that IIRC so you should have nothing to worry about despite these values.

That used to be a hard limit set in Linux because Linux memory management had difficulties flushing ARC in case of memory pressure. In FreeBSD such a limit does not exist and ARC will happily use all available memory. This is not a problem. It will be freed if needed.

I don't know the current state on Linux. At least TrueNAS removed the hard 50% limit AFAIK.

[OPNenthu]

At least TrueNAS remove the hard 50% limit AFAIK.

Indeed :P

You cannot view this attachment.

[Isabella Borgward]

Yes indeed this device has Hostwatch enabled and the others do not! I will disable it and see what happens. It had found 800 hosts, I wouldn't think that would require so much RAM to keep track of.

After a bit more reading, it seems like ARC memory usage on BSD can be considered as not really used, because it can be evicted as soon as memory is really needed by a process. That being the case, I think OpnSense management tools should take this in to account and not show me a scary yellow usage bar if it's not really "used".

[Isabella Borgward]

Before I managed to disable hostwatch, I started getting alerts about the Zabbix agent on this host.
I checked the logs and saw
[zone: pf states] PF states limit reached
being logged every few minutes.
Will ARC memory usage cause problems for PF states being created?

[Patrick M. Hausen]

Will ARC memory usage cause problems for PF states being created?

No. The number of pfstates has got a kernel internal hard limit. You can increase it at

Firewall > Settings > Advanced > Firewall Maximum States

[Isabella Borgward]

Default on this firewall is 791400. So wow, there must have been a lot of connections.
I need to work out how to track and alert on PF state exhaustion.

[pfry]

Default on this firewall is 791400. So wow, there must have been a lot of connections.[...]

I agree. The most I've seen is in the 10k range with 5k sustained, and I have a couple (unpopular) servers with static IPs. (Activity spikes are generally pingers and AI web scrapers.) Do you have popular servers, or perhaps many, active clients on one connection (e.g. a school system)? Your top output does not suggest a lot of activity.

[nero355]

In theory ZFS should never use more than 50% of that IIRC so you should have nothing to worry about despite these values.

That used to be a hard limit set in Linux because Linux memory management had difficulties flushing ARC in case of memory pressure.

In FreeBSD such a limit does not exist and ARC will happily use all available memory. This is not a problem. It will be freed if needed.

It appears to be totally different from what we both thought it should be : https://docs.freebsd.org/en/books/handbook/zfs/#zfs-advanced-tuning

vfs.zfs.arc.max starting with 13.x (vfs.zfs.arc_max for 12.x) - Upper size of the ARC.

The default is all RAM but 1 GB, or 5/8 of all RAM, whichever is more.

Use a lower value if the system runs any other daemons or processes that may require memory.
Adjust this value at runtime with sysctl(8) and set it in /boot/loader.conf or /etc/sysctl.conf.

/LearnedSomethingToday :)

[Isabella Borgward]

if the system runs any other daemons or processes that may require memory.

Right. Is there any system on earth that doesn't run 'other daemons or processes' than ARC? :D
Anyway, from the graphs, yes I can see that ARC tops out once it gets to 1GB free, so it is behaving as documented.