Just looking over my firewalls in central management, one firewall stands out above all the others with RAM usage at 85%. Every other box is 25-35%. It seems to be ARC, which I believe is part of ZFS.
All firewalls are on 26.4 or 25.10 and have 8GB of RAM. Why would ARC usage be so high on one firewall?
Quote from: Isabella Borgward on June 16, 2026, 12:57:35 PMAll firewalls are on 26.4 or 25.10 and have 8GB of RAM.
In theory ZFS should never use more than 50% of that IIRC so you should have nothing to worry about despite these values.
However...
QuoteWhy would ARC usage be so high on one firewall?
Do all of them have Hostwatch Enabled or not ?
It's a process that's constantly active and gave many of us all sorts of issues after it's release and that's why many of us also have it set ti Disabled !!
So you could try that too and see if things will change ?
Might need a reboot just to be sure... :)
Quote from: nero355 on June 16, 2026, 02:01:43 PMIn theory ZFS should never use more than 50% of that IIRC so you should have nothing to worry about despite these values.
That used to be a hard limit set in Linux because Linux memory management had difficulties flushing ARC in case of memory pressure. In FreeBSD such a limit does not exist and ARC will happily use all available memory. This is not a problem. It will be freed if needed.
I don't know the current state on Linux. At least TrueNAS removed the hard 50% limit AFAIK.
Quote from: Patrick M. Hausen on June 16, 2026, 02:22:40 PMAt least TrueNAS remove the hard 50% limit AFAIK.
Indeed :P
tn-25.10-mem.png
Yes indeed this device has Hostwatch enabled and the others do not! I will disable it and see what happens. It had found 800 hosts, I wouldn't think that would require so much RAM to keep track of.
After a bit more reading, it seems like ARC memory usage on BSD can be considered as not really used, because it can be evicted as soon as memory is really needed by a process. That being the case, I think OpnSense management tools should take this in to account and not show me a scary yellow usage bar if it's not really "used".
Before I managed to disable hostwatch, I started getting alerts about the Zabbix agent on this host.
I checked the logs and saw
[zone: pf states] PF states limit reached
being logged every few minutes.
Will ARC memory usage cause problems for PF states being created?
Quote from: Isabella Borgward on June 16, 2026, 03:01:00 PMWill ARC memory usage cause problems for PF states being created?
No. The number of pfstates has got a kernel internal hard limit. You can increase it at
Firewall > Settings > Advanced > Firewall Maximum States
Default on this firewall is 791400. So wow, there must have been a lot of connections.
I need to work out how to track and alert on PF state exhaustion.
Quote from: Isabella Borgward on June 16, 2026, 03:21:58 PMDefault on this firewall is 791400. So wow, there must have been a lot of connections.[...]
I agree. The most I've seen is in the 10k range with 5k sustained, and I have a couple (unpopular) servers with static IPs. (Activity spikes are generally pingers and AI web scrapers.) Do you have popular servers, or perhaps many, active clients on one connection (e.g. a school system)? Your top output does not suggest a lot of activity.
Quote from: Patrick M. Hausen on June 16, 2026, 02:22:40 PMQuote from: nero355 on June 16, 2026, 02:01:43 PMIn theory ZFS should never use more than 50% of that IIRC so you should have nothing to worry about despite these values.
That used to be a hard limit set in Linux because Linux memory management had difficulties flushing ARC in case of memory pressure.
In FreeBSD such a limit does not exist and ARC will happily use all available memory. This is not a problem. It will be freed if needed.
It appears to be totally different from what we both thought it should be : https://docs.freebsd.org/en/books/handbook/zfs/#zfs-advanced-tuning
Quotevfs.zfs.arc.max starting with 13.x (vfs.zfs.arc_max for 12.x) - Upper size of the ARC.
The default is all RAM but 1 GB, or 5/8 of all RAM, whichever is more.
Use a lower value if the system runs any other daemons or processes that may require memory.
Adjust this value at runtime with sysctl(8) and set it in /boot/loader.conf or /etc/sysctl.conf.
/LearnedSomethingToday :)
Quote from: nero355 on June 17, 2026, 12:34:13 PMif the system runs any other daemons or processes that may require memory.
Right. Is there any system on earth that
doesn't run 'other daemons or processes' than ARC? :D
Anyway, from the graphs, yes I can see that ARC tops out once it gets to 1GB free, so it is behaving as documented.
Quote from: Isabella Borgward on June 17, 2026, 03:52:37 PMQuote from: nero355 on June 17, 2026, 12:34:13 PMif the system runs any other daemons or processes that may require memory.
Right. Is there any system on earth that doesn't run 'other daemons or processes' than ARC? :D
Don't blame me for what they have written! LOL! :D
QuoteAnyway, from the graphs, yes I can see that ARC tops out once it gets to 1GB free, so it is behaving as documented.
NICE!!! :)