Update to 21.7.6 breaks Suricata (out of memory)

[QuisaZaderak]

Since upgrade to 21.7.6 (from 21.7.5_2) Suricata process 6.0.4 crashes regularely. In logfile I see error e.g.:

suricata[70864]   [101573] <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - opening devname netmap:ix0-2/R failed: Cannot allocate memory

After that Suricata needs to be restarted. Memory usage of system during that crash was: 35 % ( 5732/16270 MB ), so plenty of space free...

A rollback to Suricata 6.0.3 via opnsense-revert -r 21.7.5 suricata resolves the issue with memory.

[franco]

Please dial back your ruleset use or add more RAM. Ultimately if you fail to properly configure your system according to its capabilities then yes it will run out of RAM.

This can and will break on updates or even simple upstream ruleset changes.

You can avoid updating but you can't avoid administration of your device.


Cheers,
Franco

[QuisaZaderak]

I have 16GB of RAM in this machine and ruleset / out of memory with Suricata was no problem until 6.0.4.
System is a dedicated 12 core Ryzen with 16GB RAM.

With 6.0.4 it crashes within 2-5 minutes. With 6.0.3 with same ruleset / same policy it runs weeks without issues and RAM usage is currently ~3,8GB of total 16GB for all services running (with Suricata 6.0.3 active). Im using OPNsense since 19.1.x

There is one main policies rule and 34 rule adjustments active. Using ET Telemetry (44 of 50), ET Open (8 of 8 ) and Abuse.ch (5 of 5) Rulesets. A total of 94400 rules are reported by system.

[ejprice]

I am experiencing the same issue with the recent update. 21.7.6 8GiB of memory, ET ruleset. Everything was fine prior.

[ejprice]

Just reporting back that rolling back to Suricata 6.0.3 fixed the issue.

[chemlud]

What kind of trash software eats up 16GB? These suricata guys (lots of DoD money involved) should go back and fix their code...

[Fright]

Hi all.
Has anyone found a link between memory depletion and rulsets updates? maybe it's just because of a live reload of the rules? (slightly increased memory consumption in 6.0 but does it have a significant impact on live reloads?)

[QuisaZaderak]

Every time when i try to run with 6.0.4 i see
<Notice> -- This is Suricata version 6.0.4 RELEASE running in SYSTEM mode
then some warnings about unknown class types in emerging rules (also present in 6.0.3 log) and ~2 minutes later a
[101573] <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - opening devname netmap:ix0-2/R failed: Cannot allocate memory

On the 6.0.3 run at this point i would see all messages from opened netmap on my devices like:
[100831] <Notice> -- opened netmap:igb6/R from igb6: 0x1d272dfc000
[100831] <Notice> -- opened netmap:igb6^ from igb6^: 0x1d272dfc300
[100879] <Notice> -- opened netmap:igb6^ from igb6^: 0x1d275b03000
[100879] <Notice> -- opened netmap:igb6/T from igb6: 0x1d275b03300
[100882] <Notice> -- opened netmap:igb1/R from igb1: 0x1d277d86000
etc etc
[101552] <Notice> -- all 12 packet processing threads, 4 management threads initialized, engine started.
[101552] <Notice> -- rule reload starting

So on 6.0.4 i'm missing all this messages above as it crashes with out of memory before.

[wuwzy]

Just reporting back that rolling back to Suricata 6.0.3 fixed the issue.

consult. I encountered the same upgrade as yours and the problem persisted. If possible, I would like to ask you how to return to 6.0.3. Even I want to return 21.7.5   

[franco]

> [101573] <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - opening devname netmap:ix0-2/R failed: Cannot allocate memory

It looks like either the kernel is trying to open a kernel-side buffer that is bigger than a memory limit allows or the libnetmap library throws this error because it tries to request a huge buffer size as opposed to Suricata 6.0.3 itself handling the netmap open.

We are looking into it.


Cheers,
Franco

[jaylow]

Hi,

same problem here, but i got a kernel panic, too. The corresponding dmesg messages can be found in the attached file. I faced the problem while trying to activate more than six network interfaces for intrusion detection.

After getting the error "<Error> -- [ERRCODE: SC_ERR_FATAL(171)] - opening devname netmap:igb2-2/R failed: Cannot allocate memory" the command "sysctl -a | grep -i netmap" showed "netmap ouch, double free on buffer ..."

Increasing the value of dev.netmap.buf_num did not help.

Suricata legacy mode works well, but we are lack of the blocking option (IPS mode) now.

Perhaps this thread in the pfsense forum points out the underlying problem:
https://forum.pfsense.com/topic/165601/help-understanding-a-crash-kernel-panic

Hope i could provide further information.

Cheers,

Josef

[chemlud]

...if you update to 21.7.7 nmap will roll back und peace there is for the moment.