Messages

<3>pid 52659 (caddy), jid 0, uid 0, was killed: failed to reclaim memory

That didn't last long. Still had 4GB unused memory too.

Time for a beer or ten.

Update 2:

I have built a test instance of OPNsense 25.1 in Hyper-V and repeated the above periodic.conf and tests.

The output of dailysecurity.log is the same mentioning security_daily_compat_var : not found , so I know my system is not missing something that a rebuild will put back.

Also the inactive mem rises by 1000M after running periodic daily so that behaviour is the same. Again, no zfs and this is a vanilla install so no plug-ins etc.

Subsequent runs of periodic daily, periodic weekly and period monthly do not add that much to inactive mem so perhaps this is just how it is and I need to allow for the expansion when choosing plug-ins to avoid OOM reaper killing of Unbound, Caddy etc. ?

Update:

After changing anything set to "YES" to "NO" in periodic.conf and running #periodic daily then one by one setting No to YES repeating the test I was able to track the memory rise down to the security section.

I don't know why the daily security check causes Inactive Mem to go from 100M to 1250M but it does.

There were some issues reported in the security log as attached but searching forums.freebsd.org suggested this compat_var was deprecated so I don't know it is still present in the system.
https://github.com/freebsd/pkg/commit/6a077c32f445bfb10bab5536910b6b7329ce43d3

Something is messed up with my system that security check has issue with and for some reason keeps raising Inactive Mem.

Am I at the stage where I wipe and start again or can any guru shed light on the issue please ?

I have now added periodic.conf overrides so logs are saved to /var/log/daily.log etc instead of "root" as there is no mail installed.
That got rid of the mail error and on inspection of the daily.log I can see no issues.

I do not use ZFS and these tasks are not part enabled in the daily other than the default list zfs pools.

Again from a fresh boot I run periodic daily and Inactive memory rises from 100M to 1280M.
From another ssh shell I can see pkg appear in top processes during the time the inactive memory rises. Then pkg goes and I briefly see 'xz' then it is over.

I am thinking to turn off all parts of the periodic daily then enable one at a time to see which element is causing the issue.

I think I might be getting somewhere.

I tried the boguns update mentioned earlier which made no difference to inactive memory so  I decided to run from SSH  the 'periodic daily' cron task.

I watched as from a freshly booted system the inactive memory climb from 80M to 1200M within a few seconds and stay there.

The prompt took a good 2 minutes to come back then said 'eval: mail: not found'

I cannot see anywhere in the gui to configure mail. I think older releases had it in System/Settings/Notifications but that is not present.

I'm assuming the mail error is the cause of the inactive memory issue here ?

Can anyone point me in the right direction please ?

No, I am not using zfs just ufs.

Yes 43 for me - counted them with ps -faxd | grep php-cgi

The only thing left I can think of doing is disabling or uninstalling CrowdSec, Caddy and Unbound DNS.

I also have a ProofPoint Emerging Threats alias blocklist that updates every 12 hours which I could stop.

I need OpenVPN running to access the firewall from work so can't lose that too!

It would not leave much left for OPNsense firewall to do and I'd lose most of the functionality.

I installed the firewall over three years ago on a HP T730 8GB and have added a Intel I350 2 port NIC and it's been fantastic till the memory issues lately.

Is it normal to have 43 php-cgi processes running ?

Hi Cedrik

I am still using Caddy but have found that the memory issues I have are also affecting Unbound DNS.

My inactive memory still creeps up and up even without Caddy installed so it's not the plugin's issue.

Thanks for your help.

Inactive memory still on the rise.

What could be using it up and other than rebooting the firewall, how do I get it back ?

Dashboard still says 1.2/7GB in use.

I searched for others with similar memory issues - I am not with Zen Internet and not using IPv6 so that's not it.

Thanks Meyergru I checked them (attached) and nothing jumps out at me ?

Just to add I tried one by one restarting services from the dashboard and this did claw back a 10% increase in free mem but only temporarily.
A reboot of the firewall and free mem went back up to near 80%.

I have noticed over the last few weeks, since 24.7 and now 25.1.1 that around 3:18 to 3:38 am every morning the Inactive Memory rises by around 20 %.

Free memory is at 78% after reboot then the next morning it drop so 52%, then the next day 40%, then 30% at which point OOM kicks in killing off Caddy Plugin or Unbound or both.

I have tried disabling IDS and removing most of my blocklists from Crowdsec but the behaviour is the same. With Caddy and IDS off I get a few more days before OOM starts killing things.

I have no cronjobs around this time and have been through every log file available from the GUI but cannot figure out why or what is going on at that time.

The dashboard memory widget happily says I am using 1.2/7GB RAM (I am using a 1GB MFS).

Can anyone point me in the right direction to track down why this could be happening please ?

I finally managed to get my cert exported to Proxmox.
I just followed this recent guide:

https://sysadmin102.com/2025/02/proxmox-opnsense-acme-certificate-automation/

It goes on about using a limited new user-token just for the purpose of the acme automation which seemed a good idea.

I had to make sure I have a DNS override setup in my OpnSense Unbound DNS so it would resolve my Proxmox host.

Works a treat :)

Have you checked "/var/log/caddy/caddy.log" specifically? It's not exposed in the GUI.

Yes I checked all the logs from that folder.

I do not have Layer4 setup and only two Reverse Proxies. With Caddy disabled the memory usage sits at 3.1/8GB RAM, With Caddy running it is initially 3.2/8GB then after a day 4/8GB so not running low on RAM.

I do appreciate that with issues you cannot reproduce or have any meaningful errors in the logs it is a bit difficult to progress a solution.

My system does seem to have had remnants of ZenArmor from previous  setups that have not quite removed cleanly so it could be something there but everything else seems ok and I am reluctant to wipe my firewall to try Caddy on a clean machine so will leave without it for now.

Thanks

I have a similar issue and have updated to the latest OPnsense 25.1.1 and Caddy plugin 1.8.2 but still get the memory issue after running for a day:

2025-02-14T07:52:15   Notice   kernel   <3>pid 87856 (caddy), jid 0, uid 0, was killed: failed to reclaim memory   
2025-02-13T13:18:23   Notice   kernel   <118>Log: /var/log/caddy/caddy.log   
2025-02-13T13:18:23   Notice   kernel   <118>Starting caddy... done   
2025-02-11T13:10:43   Notice   kernel   <3>pid 47033 (caddy), jid 0, uid 0, was killed: failed to reclaim memory   
2025-01-30T15:34:06   Notice   kernel   <3>pid 87676 (caddy), jid 0, uid 0, was killed: failed to reclaim memory

No panic messages in caddy logs