pid xxx (suricata), jid 0, uid 0, was killed: failed to reclaim memory

[proutfoo]

Hello,

I am new to the IDS setup and i created a schedule to update the rules once a day. However when it comes to reloading after the successful download, both suricata and unbound crash and do not restart;

2024-02-24T02:14:23   Notice   kernel   <3>pid 61010 (unbound), jid 0, uid 59, was killed: failed to reclaim memory   
2024-02-24T02:14:23   Notice   kernel   <3>pid 97109 (suricata), jid 0, uid 0, was killed: failed to reclaim memory   
2024-02-24T02:12:03   Notice   rule-updater.py   download completed for https://rules.emergingthreats.net/open/suricata-7.0/emerging.rules.tar.gz   
2024-02-24T02:12:02   Notice   rule-updater.py   version response for https://rules.emergingthreats.net/open/suricata-7.0/version.txt : 10539   
2024-02-24T02:12:01   Notice   rule-updater.py   download completed for https://threatfox.abuse.ch/downloads/threatfox_suricata.rules   
2024-02-24T02:12:01   Notice   rule-updater.py   download completed for https://feodotracker.abuse.ch/downloads/feodotracker.rules   
2024-02-24T02:12:01   Notice   rule-updater.py   download completed for https://sslbl.abuse.ch/blacklist/sslipblacklist.rules   
2024-02-24T02:12:00   Notice   rule-updater.py   download completed for https://sslbl.abuse.ch/blacklist/sslblacklist_tls_cert.rules

any tips how to fix this?  I have 6GB available to this opnsense VM, going to try and move it up to 8 and see if its running out of RAM perhaps?

[proutfoo]

I doubled RAM to 12G and 48 hours later its still going so I guess I was running out of available memory.

[chris-stl]

Same - OPNsense 25.1.1-amd64, 16GB RAM and suricata (in IDS mode) is dying every 16-18h with this error:

Code Select Expand

kernel - - [meta sequenceId="1"] <3>pid 70333 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Enabled subscriptions of ET Pro telemetry rules.

It is growing every rule reload starting/complete task and not releasing memory (every 3h in my case)

I suspect memory leak.

PS: suricata are using about 4.1GB of memory after fresh start and stay around 4.3GB.

[IvahLind]

Hello,

I am new to the IDS setup and i created a schedule to update the rules once a day. However when it comes to reloading after the successful download, both suricata and unbound crash and do not restart;

2024-02-24T02:14:23   Notice   kernel   <3>pid 61010 (unbound), jid 0, uid 59, was killed: failed to reclaim memory   
2024-02-24T02:14:23   Notice   kernel   <3>pid 97109 (suricata), jid 0, uid 0, was killed: failed to reclaim memory   
2024-02-24T02:12:03   Notice   rule-updater.py   download completed for https://rules.emergingthreats.net/open/suricata-7.0/emerging.rules.tar.gz   
2024-02-24T02:12:02   Notice   rule-updater.py   version response for https://rules.emergingthreats.net/open/suricata-7.0/version.txt : 10539   
2024-02-24T02:12:01   Notice   rule-updater.py   download completed for https://threatfox.abuse.ch/downloads/threatfox_suricata.rules   
2024-02-24T02:12:01   Notice   rule-updater.py   download completed for https://feodotracker.abuse.ch/downloads/feodotracker.rules   
2024-02-24T02:12:01   Notice   rule-updater.py   download completed for https://sslbl.abuse.ch/blacklist/sslipblacklist.rules   
2024-02-24T02:12:00   Notice   rule-updater.py   download completed for https://sslbl.abuse.ch/blacklist/sslblacklist_tls_cert.rules
Geometry Dash Lite[/colorl]
any tips how to fix this?  I have 6GB available to this opnsense VM, going to try and move it up to 8 and see if its running out of RAM perhaps?

Hello proutfoo!
After increasing your RAM, monitor your system's memory usage during the rule update to see if it actually runs out of memory. Use top or htop to monitor memory usage in real time.
Could this be helpful?

[chris-stl]

More findings (my OPNSense router has 16GB of RAM):
- the default OPNSense setup has log configuration to use 50% of memory for ramdisk (/var/log) to store logs (in my case about 8GB of ram)
- the last matching rule of default deny any and default pass is configured (by default) to do logging (/var/log/filter/filter*.log)
- so after 16-20hours of working, the /var/log ramdisk is about full usage.
- then schedule task for Suricata is fired, new rules downloaded and applied
- based that (in my case) Suricata i using about 6.5-7GB of RAM, it use about 10-11GB during new rules applying process
- 8GB of ramdisk + 10GB Suricata process = OOM killer job

PS: I reconfigured /var/log ram disk to take 3.5GB and 4 days log rotating, then the remote logging is configured to store logs for longer time. I will watch, what will happen to Suricata then.