OPNsense Forum

Archive => 18.1 Legacy Series => Topic started by: crt333 on July 16, 2018, 12:28:51 am

Title: 18.1.12 suricata crash
Post by: crt333 on July 16, 2018, 12:28:51 am
After the latest update suricata is crashing. The dashboard shows that the memory use grows up to about double what it used in 18.1.11 and then it dies. The log shows:

Jul 15 16:23:42   kernel: pid 532 (suricata), uid 0: exited on signal 6 (core dumped)
Jul 15 16:20:19   kernel: -> pid: 300 ppid: 97610 p_pax: 0x850<SEGVGUARD,ASLR,NODISALLOWMAP32BIT>
Jul 15 16:20:19   kernel: [HBSD SEGVGUARD] [/usr/local/bin/suricata (300)] Suspension expired.

hardware is a quotom q355g4, using openssl.

restarting it the memory slowly increases and then it crashes again.
Title: Re: 18.1.12 suricata crash
Post by: johan on July 17, 2018, 10:31:29 am
I see this too, exact same behavior. Other hardware though, Supermicro x10sba J1900.
Title: Re: 18.1.12 suricata crash
Post by: mimugmail on July 17, 2018, 10:41:29 am
Do you have all rules enabled? Also the app-detect rules?

Can you give us some logs from suricata.log
Title: Re: 18.1.12 suricata crash
Post by: crt333 on July 17, 2018, 06:22:01 pm
All abuse and ET Open rules enabled.

OPNsense-App-detect/test enabled, other App-detect disabled.

I'd be happy to give you the log file, but don't know how to get it. Can it be accessed from the GUI?
Title: Re: 18.1.12 suricata crash
Post by: GOCE on July 17, 2018, 06:23:02 pm
Same here on both HA firewalls after update to 18.1.12.
Title: Re: 18.1.12 suricata crash
Post by: mimugmail on July 17, 2018, 07:47:33 pm
Do you have URL haus from abuse enabled? There are two rule errors. Please disable for testing.
Title: Re: 18.1.12 suricata crash
Post by: crt333 on July 17, 2018, 08:58:08 pm
URL haus from abuse was enabled, so I disabled and tried again. Same as before, memory use grows beyond the 1.1GB it used to use up to around 2.3GB, then drops down when suricata dies.

I rebooted and tried again, same results.

I disabled IPS and just ran IDS, same result, dies after a couple of minutes.
Title: Re: 18.1.12 suricata crash
Post by: ruggerio on July 17, 2018, 09:02:27 pm
Have reinstalled suricata and downloaded again all rules. After that, the following message occurs

Jul 17 15:09:44    suricata: [100271] <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "drop http $HOME_NET any -> $EXTERNAL_NET any (msg:"URLhaus Known malware download URL detected"; flow:established,from_client; content:"GET"; http_method; content:"/kor/kapkap5.yarn"; http_uri; depth:17; isdataat:!1,relative; content:"bcxvjwqhewqe.com"; http_host; depth:16; isdataat:!1,relative; metadata:created_at " from file /usr/local/etc/suricata/opnsense.rules/abuse.ch.urlhaus.rules at line 19366

1st my pulse was on 200 (wtf, malware), until i saw i must be the mentionned false rule :) :)

hope this helps.
Title: Re: 18.1.12 suricata crash
Post by: ruggerio on July 17, 2018, 09:06:06 pm
what i can confirm after disabling url haus, is that suricata started working againg. I will monitor and post, if it breaks again.
Title: Re: 18.1.12 suricata crash
Post by: JohnnyBeee on July 18, 2018, 03:28:39 pm
I have the same problem.

I have deactivated url.haus entirely, but this did not help.
I still get kernel: pid 5613 (suricata), uid 0: exited on signal 6 (core dumped)
I don't think that a rule is the culprit because, when I check the Intrusion detection log, I do not even see the usual message
suricata: [100137] <Notice> -- rule reload starting
All i get now is thie message and then I see the crash in the system log.
    suricata: [100180] <Notice> -- This is Suricata version 4.0.4 RELEASE

The memory usage peaks at 50% on suricata start.
Restarting the firewall did not help. I have a APU.2C4.

The problem started 3 days ago, not sure what triggered it. I never had it before 18.1.11, that I know.

Please help.
Title: Re: 18.1.12 suricata crash
Post by: ruggerio on July 18, 2018, 04:13:28 pm
I reinstalled Suricata und downloaded the rules again. Disabled URL-Haus, this made it working for me.

The Thing is, that i never saw any Action from suricata except the start in the logs. Reinstalling fixed it for me.

Roger
Title: Re: 18.1.12 suricata crash
Post by: mimugmail on July 18, 2018, 04:16:32 pm
Can you disable all rules for testing?
Title: Re: 18.1.12 suricata crash
Post by: JohnnyBeee on July 18, 2018, 04:17:23 pm
It would seem that Pattern matcher "Hyperscan" is the culprit.
When I switch to "Aho-Corasick" Suricata starts. Performance is way lower than with Hyperscan though.

Note that the messages
suricata: [100185] <Notice> -- rule reload starting
suricata: [100185] <Notice> -- rule reload complete
still no longer show as of 18.1.11

How can I get this fixed to get Hyperscan to work again?
Title: Re: 18.1.12 suricata crash
Post by: JohnnyBeee on July 18, 2018, 05:23:29 pm
Can you disable all rules for testing?

Thank you for your help.

I have deactivated all the rules and switched back to Hyperscan and Suricata still crashes.
Note that the memory usage, with all the rules discabled, still peaks at about 50% during Suricata startup, then drops to about 20%.
Title: Re: 18.1.12 suricata crash
Post by: JohnnyBeee on July 18, 2018, 05:45:17 pm
I reinstalled Suricata und downloaded the rules again. Disabled URL-Haus, this made it working for me.

The Thing is, that i never saw any Action from suricata except the start in the logs. Reinstalling fixed it for me.

Roger

Thanks for your help.

Please forgive my ignorance, but  how do I reinstall Suricata?

Thanks.
Title: Re: 18.1.12 suricata crash
Post by: mimugmail on July 18, 2018, 05:49:55 pm
System : Firmware : Packages

There's a reinstall button for every pkg
Title: Re: 18.1.12 suricata crash
Post by: JohnnyBeee on July 18, 2018, 06:13:29 pm
System : Firmware : Packages

There's a reinstall button for every pkg

Thank you.
I reinstalled Suricata but the problem persists.

The messages I still get are:
Jul 18 17:34:29    kernel: pid 52626 (suricata), uid 0: exited on signal 6 (core dumped)
Jul 18 17:29:46    kernel: -> pid: 52293 ppid: 49789 p_pax: 0x850<SEGVGUARD,ASLR,NODISALLOWMAP32BIT>
Jul 18 17:29:46    kernel: [HBSD SEGVGUARD] [/usr/local/bin/suricata (52293)] Suspension expired.
Title: Re: 18.1.12 suricata crash
Post by: ruggerio on July 18, 2018, 10:00:24 pm
Just click System->Firmware->Packages

Search for Suricata and click on the symbol, which looks like a recycling-sign (Reinstall Suricata) :)

Afterwards go to Suricata and download the rules again, but without url-haus.

Roger
Title: Re: 18.1.12 suricata crash
Post by: franco on July 18, 2018, 10:08:51 pm
Suricata 4.0.5 was released today. Not sure if the crashes are related to the CVEs...

https://suricata-ids.org/2018/07/18/suricata-4-0-5-available/

You can install for amd64/OpenSSL:

# pkg add -f https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/MINT/18.7.r2/OpenSSL/All/suricata-4.0.5.txz

or amd64/LibreSSL:

# pkg add -f https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/MINT/18.7.r2/LibreSSL/All/suricata-4.0.5.txz

to see if that helps.

Reinstall/revert to the current version using:

# opnsense-revert suricata


Cheers,
Franco
Title: Re: 18.1.12 suricata crash
Post by: crt333 on July 18, 2018, 10:52:46 pm
- I downloaded all the rules and restarted -> crash
- I reinstalled suricata -> crash
- I switched from hypersan to aho-corasick -> running
Title: Re: 18.1.12 suricata crash
Post by: mimugmail on July 18, 2018, 10:58:02 pm
Does you CPU support SSS3?
Title: Re: 18.1.12 suricata crash
Post by: crt333 on July 19, 2018, 12:56:16 am
My CPU is an I5 5250U, SSE 4.1, 4.2, AVX2....
Title: Re: 18.1.12 suricata crash
Post by: mimugmail on July 19, 2018, 06:02:21 am
Strange. Have you checked latest Update from Franco?
Title: Re: 18.1.12 suricata crash
Post by: JohnnyBeee on July 19, 2018, 06:17:29 am
Suricata 4.0.5 was released today. Not sure if the crashes are related to the CVEs...

https://suricata-ids.org/2018/07/18/suricata-4-0-5-available/

You can install for amd64/OpenSSL:

# pkg add -f https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/MINT/18.7.r2/OpenSSL/All/suricata-4.0.5.txz

or amd64/LibreSSL:

# pkg add -f https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/MINT/18.7.r2/LibreSSL/All/suricata-4.0.5.txz

to see if that helps.

Reinstall/revert to the current version using:

# opnsense-revert suricata


Cheers,
Franco

It works for me :)
Title: Re: 18.1.12 suricata crash
Post by: ruggerio on July 19, 2018, 07:14:54 am
cool, so whats the difference, why it is working for me?

Hardware:  apu2
Scanengine: hyperscan

Title: Re: 18.1.12 suricata crash
Post by: franco on July 19, 2018, 09:44:33 am
My guess is different rules and different local traffic. It's hard to pin down without looking at the specifics of the 4.0.5 update if that indeed magically solves it.


Cheers,
Franco
Title: Re: 18.1.12 suricata crash
Post by: crt333 on July 19, 2018, 03:56:44 pm
I updated to 4.0.5, and switched to hyperscan, and after a few minutes suricata crashed again.

Reverted to current and switched to aho-corasick and it is working again.
Title: Re: 18.1.12 suricata crash
Post by: rjb4526 on July 24, 2018, 05:02:56 pm
For me this was definitely due to the abuse.ch\urlhaus rule set.  Once I disabled that and re-downloaded/reloaded the rules, Suricata stopped crashing.

It seems they have an issue with this rule set currently.
Title: Re: 18.1.12 suricata crash
Post by: franco on July 24, 2018, 06:13:16 pm
This may be relevant. https://twitter.com/abuse_ch/status/1020172320378417154


Cheers,
Franco
Title: Re: 18.1.12 suricata crash
Post by: rjb4526 on July 25, 2018, 02:36:10 am
This may be relevant. https://twitter.com/abuse_ch/status/1020172320378417154


Cheers,
Franco

Well, it wasn't fixed... Re-enabling the abuse.ch\urlhaus rules on 18.1.12 resulted in Suricata crashing again until I disabled the rule set.

After updating to 18.1.13 I am now experiencing the same issue with memory usage growing until Suricata crashes, then going back to normal.  This only seems to occur if the abuse.ch\urlhaus rule set is enabled.  Disabling it again seems to have stopped the issue for now.
Title: Re: 18.1.12 suricata crash
Post by: mimugmail on July 25, 2018, 05:53:11 am
How much RAM do you have?
Title: Re: 18.1.12 suricata crash
Post by: max on July 25, 2018, 04:10:36 pm
After updating to 18.1.13 I am now experiencing the same issue with memory usage growing until Suricata crashes, then going back to normal.  This only seems to occur if the abuse.ch\urlhaus rule set is enabled.  Disabling it again seems to have stopped the issue for now.

I can confirm the issue on several apu2c4 based systems on OPNsense 18.1.12 and 18.1.13.

As stated by rjb4526, the only workaround that prevents Suricata from crashing currently is to disable abuse.ch/URLhaus.

Some more background:

Title: Re: 18.1.12 suricata crash
Post by: rjb4526 on July 25, 2018, 05:11:41 pm
How much RAM do you have?

8GB.  RAM usage doesn't grow to 100%, though.  Last I saw it grew to about 2.5GB before Suricata crashed.
Title: Re: 18.1.12 suricata crash
Post by: crt333 on July 25, 2018, 06:35:29 pm
I just upgraded to 18.1.13 and noticed some odd things, so I thought I'd try and add some details in the hope that it helps the always helpful OpnSense team

System is a Qotom Q355G4
CPU: i5-5250U
RAM: 8GB

NOTE: after every setting change below I rebooted to make sure results were "clean"

phase 1 - update to 18.1.13, reboot, reload all suricata rules with URLhaus disabled

a) using hyperscan - runs ok
    system memory 1059M

    suricata memory usage (from System->Diagnostics->Activity)
    Mem Size: 2805M
    Res: 350M

b) using aho-corasick - runs ok
    system memory 1129M

    suricata memory usage (from System->Diagnostics->Activity)
    Mem Size: 2913M
    Res: 443M

phase 2 - enable URLhaus,  download all rules

a) using aho-corasick - runs ok
    system memory 1794M

    suricata memory usage (from System->Diagnostics->Activity)
    Mem Size: 3565M
    Res: 1069M

b) using hyperscan: crashes

phase 3: disable URLhaus

a) using hyperscan: crashes (but it worked above!!)

so, the only thing that was different in 1a above was that the rules for URLhaus showed "not installed" at the start, rather than just disabled. So, I downloaded all the rules again so it showed "not installed" again for URLhaus, and rebooted:

using hyperscan: works again

so, the URLhaus rule can't even be installed it seems, even if disabled.

FURTHER ODDITY:

after the crashes, when I changed settings and clicked reboot, the screen paused for a while but then came back to show stats rather than to the login screen. Also, uptime showed no reboot occurred. Tried twice, same thing. Also tried shutdown, which also didn't work!!

So, I had to power cycle after the failures to cause a reboot to happen. Very odd, thought I'd mention it.

I hope the above is helpful, please let me know if I can provide other info.

Title: Re: 18.1.12 suricata crash
Post by: mimugmail on July 25, 2018, 07:03:30 pm
Perhaps the ruleset wont get deleted. There was a fix around, not sure If only for 18.7.
Title: Re: 18.1.12 suricata crash
Post by: rjb4526 on July 26, 2018, 04:25:20 am
Despite abuse.ch's claims to the contrary on Twitter, the issue still isn't fixed.  Enabling abuse.ch/urlhaus rules still results in Suricata crashing.
Title: Re: 18.1.12 suricata crash
Post by: mimugmail on July 26, 2018, 06:01:51 am
So you get an error in the logs? If not perhaps the ruleset is too big for you system?
Title: Re: 18.1.12 suricata crash
Post by: rjb4526 on July 27, 2018, 01:51:09 am
So you get an error in the logs? If not perhaps the ruleset is too big for you system?
The only thing that shows up in the logs is Suricata crashing.
Title: Re: 18.1.12 suricata crash
Post by: mimugmail on July 27, 2018, 05:52:01 am
Can you watch the memory Bar in the Dashboard shortly after enabling Suricata?
Title: Re: 18.1.12 suricata crash
Post by: rjb4526 on July 28, 2018, 05:01:44 am
Can you watch the memory Bar in the Dashboard shortly after enabling Suricata?
Yes.  I refresh it several times over the course of a couple of minutes and watch the memory usage grow to about 2.8-3GB before it crashes.  Once it crashes, memory usage goes back to a more normal number like 750MB-1GB depending on usage.
Title: Re: 18.1.12 suricata crash
Post by: wefinet on September 03, 2018, 02:25:36 pm
[Updated as I have answered in the wrong thread]:

I'm having issues here, too.

See https://forum.opnsense.org/index.php?topic=9512.msg43639#msg43639 for my details.