OPNsense Forum

Archive => 21.7 Legacy Series => Topic started by: QuisaZaderak on November 29, 2021, 11:00:28 am

Title: Update to 21.7.6 breaks Suricata (out of memory)
Post by: QuisaZaderak on November 29, 2021, 11:00:28 am
Since upgrade to 21.7.6 (from 21.7.5_2) Suricata process 6.0.4 crashes regularely. In logfile I see error e.g.:

suricata[70864]   [101573] <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - opening devname netmap:ix0-2/R failed: Cannot allocate memory

After that Suricata needs to be restarted. Memory usage of system during that crash was: 35 % ( 5732/16270 MB ), so plenty of space free...

A rollback to Suricata 6.0.3 via opnsense-revert -r 21.7.5 suricata resolves the issue with memory.
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: franco on November 29, 2021, 11:02:59 am
Please dial back your ruleset use or add more RAM. Ultimately if you fail to properly configure your system according to its capabilities then yes it will run out of RAM.

This can and will break on updates or even simple upstream ruleset changes.

You can avoid updating but you can't avoid administration of your device.


Cheers,
Franco
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: QuisaZaderak on November 29, 2021, 05:56:42 pm
I have 16GB of RAM in this machine and ruleset / out of memory with Suricata was no problem until 6.0.4.
System is a dedicated 12 core Ryzen with 16GB RAM.

With 6.0.4 it crashes within 2-5 minutes. With 6.0.3 with same ruleset / same policy it runs weeks without issues and RAM usage is currently ~3,8GB of total 16GB for all services running (with Suricata 6.0.3 active). Im using OPNsense since 19.1.x

There is one main policies rule and 34 rule adjustments active. Using ET Telemetry (44 of 50), ET Open (8 of 8 ) and Abuse.ch (5 of 5) Rulesets. A total of 94400 rules are reported by system.
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: ejprice on December 02, 2021, 10:26:59 pm
I am experiencing the same issue with the recent update. 21.7.6 8GiB of memory, ET ruleset. Everything was fine prior.
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: ejprice on December 03, 2021, 07:26:48 pm
Just reporting back that rolling back to Suricata 6.0.3 fixed the issue.
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: chemlud on December 03, 2021, 08:26:00 pm
What kind of trash software eats up 16GB? These suricata guys (lots of DoD money involved) should go back and fix their code...
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: Fright on December 04, 2021, 03:24:39 pm
Hi all.
Has anyone found a link between memory depletion and rulsets updates? maybe it's just because of a live reload of the rules? (slightly increased memory consumption in 6.0 but does it have a significant impact on live reloads?)
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: QuisaZaderak on December 04, 2021, 03:47:12 pm
Every time when i try to run with 6.0.4 i see
<Notice> -- This is Suricata version 6.0.4 RELEASE running in SYSTEM mode
then some warnings about unknown class types in emerging rules (also present in 6.0.3 log) and ~2 minutes later a
[101573] <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - opening devname netmap:ix0-2/R failed: Cannot allocate memory

On the 6.0.3 run at this point i would see all messages from opened netmap on my devices like:
[100831] <Notice> -- opened netmap:igb6/R from igb6: 0x1d272dfc000
[100831] <Notice> -- opened netmap:igb6^ from igb6^: 0x1d272dfc300
[100879] <Notice> -- opened netmap:igb6^ from igb6^: 0x1d275b03000
[100879] <Notice> -- opened netmap:igb6/T from igb6: 0x1d275b03300
[100882] <Notice> -- opened netmap:igb1/R from igb1: 0x1d277d86000
etc etc
[101552] <Notice> -- all 12 packet processing threads, 4 management threads initialized, engine started.
[101552] <Notice> -- rule reload starting

So on 6.0.4 i'm missing all this messages above as it crashes with out of memory before.
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: wuwzy on December 06, 2021, 07:00:18 am
Just reporting back that rolling back to Suricata 6.0.3 fixed the issue.

consult. I encountered the same upgrade as yours and the problem persisted. If possible, I would like to ask you how to return to 6.0.3. Even I want to return 21.7.5   ;D
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: franco on December 06, 2021, 08:40:33 am
> [101573] <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - opening devname netmap:ix0-2/R failed: Cannot allocate memory

It looks like either the kernel is trying to open a kernel-side buffer that is bigger than a memory limit allows or the libnetmap library throws this error because it tries to request a huge buffer size as opposed to Suricata 6.0.3 itself handling the netmap open.

We are looking into it.


Cheers,
Franco
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: jaylow on December 15, 2021, 07:51:27 pm
Hi,

same problem here, but i got a kernel panic, too. The corresponding dmesg messages can be found in the attached file. I faced the problem while trying to activate more than six network interfaces for intrusion detection.

After getting the error "<Error> -- [ERRCODE: SC_ERR_FATAL(171)] - opening devname netmap:igb2-2/R failed: Cannot allocate memory" the command "sysctl -a | grep -i netmap" showed "netmap ouch, double free on buffer ..."

Increasing the value of dev.netmap.buf_num did not help.

Suricata legacy mode works well, but we are lack of the blocking option (IPS mode) now.

Perhaps this thread in the pfsense forum points out the underlying problem:
https://forum.pfsense.com/topic/165601/help-understanding-a-crash-kernel-panic

Hope i could provide further information.

Cheers,

Josef
Title: Re: Update to 21.7.6 breaks Suricata (out of memory)
Post by: chemlud on December 15, 2021, 08:09:56 pm
...if you update to 21.7.7 nmap will roll back und peace there is for the moment.  ;)