OPNsense Forum
Archive => 18.1 Legacy Series => Topic started by: crt333 on July 16, 2018, 12:28:51 am
-
After the latest update suricata is crashing. The dashboard shows that the memory use grows up to about double what it used in 18.1.11 and then it dies. The log shows:
Jul 15 16:23:42 kernel: pid 532 (suricata), uid 0: exited on signal 6 (core dumped)
Jul 15 16:20:19 kernel: -> pid: 300 ppid: 97610 p_pax: 0x850<SEGVGUARD,ASLR,NODISALLOWMAP32BIT>
Jul 15 16:20:19 kernel: [HBSD SEGVGUARD] [/usr/local/bin/suricata (300)] Suspension expired.
hardware is a quotom q355g4, using openssl.
restarting it the memory slowly increases and then it crashes again.
-
I see this too, exact same behavior. Other hardware though, Supermicro x10sba J1900.
-
Do you have all rules enabled? Also the app-detect rules?
Can you give us some logs from suricata.log
-
All abuse and ET Open rules enabled.
OPNsense-App-detect/test enabled, other App-detect disabled.
I'd be happy to give you the log file, but don't know how to get it. Can it be accessed from the GUI?
-
Same here on both HA firewalls after update to 18.1.12.
-
Do you have URL haus from abuse enabled? There are two rule errors. Please disable for testing.
-
URL haus from abuse was enabled, so I disabled and tried again. Same as before, memory use grows beyond the 1.1GB it used to use up to around 2.3GB, then drops down when suricata dies.
I rebooted and tried again, same results.
I disabled IPS and just ran IDS, same result, dies after a couple of minutes.
-
Have reinstalled suricata and downloaded again all rules. After that, the following message occurs
Jul 17 15:09:44 suricata: [100271] <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "drop http $HOME_NET any -> $EXTERNAL_NET any (msg:"URLhaus Known malware download URL detected"; flow:established,from_client; content:"GET"; http_method; content:"/kor/kapkap5.yarn"; http_uri; depth:17; isdataat:!1,relative; content:"bcxvjwqhewqe.com"; http_host; depth:16; isdataat:!1,relative; metadata:created_at " from file /usr/local/etc/suricata/opnsense.rules/abuse.ch.urlhaus.rules at line 19366
1st my pulse was on 200 (wtf, malware), until i saw i must be the mentionned false rule :) :)
hope this helps.
-
what i can confirm after disabling url haus, is that suricata started working againg. I will monitor and post, if it breaks again.
-
I have the same problem.
I have deactivated url.haus entirely, but this did not help.
I still get kernel: pid 5613 (suricata), uid 0: exited on signal 6 (core dumped)
I don't think that a rule is the culprit because, when I check the Intrusion detection log, I do not even see the usual message
suricata: [100137] <Notice> -- rule reload starting
All i get now is thie message and then I see the crash in the system log.
suricata: [100180] <Notice> -- This is Suricata version 4.0.4 RELEASE
The memory usage peaks at 50% on suricata start.
Restarting the firewall did not help. I have a APU.2C4.
The problem started 3 days ago, not sure what triggered it. I never had it before 18.1.11, that I know.
Please help.
-
I reinstalled Suricata und downloaded the rules again. Disabled URL-Haus, this made it working for me.
The Thing is, that i never saw any Action from suricata except the start in the logs. Reinstalling fixed it for me.
Roger
-
Can you disable all rules for testing?
-
It would seem that Pattern matcher "Hyperscan" is the culprit.
When I switch to "Aho-Corasick" Suricata starts. Performance is way lower than with Hyperscan though.
Note that the messages
suricata: [100185] <Notice> -- rule reload starting
suricata: [100185] <Notice> -- rule reload complete
still no longer show as of 18.1.11
How can I get this fixed to get Hyperscan to work again?
-
Can you disable all rules for testing?
Thank you for your help.
I have deactivated all the rules and switched back to Hyperscan and Suricata still crashes.
Note that the memory usage, with all the rules discabled, still peaks at about 50% during Suricata startup, then drops to about 20%.
-
I reinstalled Suricata und downloaded the rules again. Disabled URL-Haus, this made it working for me.
The Thing is, that i never saw any Action from suricata except the start in the logs. Reinstalling fixed it for me.
Roger
Thanks for your help.
Please forgive my ignorance, but how do I reinstall Suricata?
Thanks.
-
System : Firmware : Packages
There's a reinstall button for every pkg
-
System : Firmware : Packages
There's a reinstall button for every pkg
Thank you.
I reinstalled Suricata but the problem persists.
The messages I still get are:
Jul 18 17:34:29 kernel: pid 52626 (suricata), uid 0: exited on signal 6 (core dumped)
Jul 18 17:29:46 kernel: -> pid: 52293 ppid: 49789 p_pax: 0x850<SEGVGUARD,ASLR,NODISALLOWMAP32BIT>
Jul 18 17:29:46 kernel: [HBSD SEGVGUARD] [/usr/local/bin/suricata (52293)] Suspension expired.
-
Just click System->Firmware->Packages
Search for Suricata and click on the symbol, which looks like a recycling-sign (Reinstall Suricata) :)
Afterwards go to Suricata and download the rules again, but without url-haus.
Roger
-
Suricata 4.0.5 was released today. Not sure if the crashes are related to the CVEs...
https://suricata-ids.org/2018/07/18/suricata-4-0-5-available/
You can install for amd64/OpenSSL:
# pkg add -f https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/MINT/18.7.r2/OpenSSL/All/suricata-4.0.5.txz
or amd64/LibreSSL:
# pkg add -f https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/MINT/18.7.r2/LibreSSL/All/suricata-4.0.5.txz
to see if that helps.
Reinstall/revert to the current version using:
# opnsense-revert suricata
Cheers,
Franco
-
- I downloaded all the rules and restarted -> crash
- I reinstalled suricata -> crash
- I switched from hypersan to aho-corasick -> running
-
Does you CPU support SSS3?
-
My CPU is an I5 5250U, SSE 4.1, 4.2, AVX2....
-
Strange. Have you checked latest Update from Franco?
-
Suricata 4.0.5 was released today. Not sure if the crashes are related to the CVEs...
https://suricata-ids.org/2018/07/18/suricata-4-0-5-available/
You can install for amd64/OpenSSL:
# pkg add -f https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/MINT/18.7.r2/OpenSSL/All/suricata-4.0.5.txz
or amd64/LibreSSL:
# pkg add -f https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/MINT/18.7.r2/LibreSSL/All/suricata-4.0.5.txz
to see if that helps.
Reinstall/revert to the current version using:
# opnsense-revert suricata
Cheers,
Franco
It works for me :)
-
cool, so whats the difference, why it is working for me?
Hardware: apu2
Scanengine: hyperscan
-
My guess is different rules and different local traffic. It's hard to pin down without looking at the specifics of the 4.0.5 update if that indeed magically solves it.
Cheers,
Franco
-
I updated to 4.0.5, and switched to hyperscan, and after a few minutes suricata crashed again.
Reverted to current and switched to aho-corasick and it is working again.
-
For me this was definitely due to the abuse.ch\urlhaus rule set. Once I disabled that and re-downloaded/reloaded the rules, Suricata stopped crashing.
It seems they have an issue with this rule set currently.
-
This may be relevant. https://twitter.com/abuse_ch/status/1020172320378417154
Cheers,
Franco
-
This may be relevant. https://twitter.com/abuse_ch/status/1020172320378417154
Cheers,
Franco
Well, it wasn't fixed... Re-enabling the abuse.ch\urlhaus rules on 18.1.12 resulted in Suricata crashing again until I disabled the rule set.
After updating to 18.1.13 I am now experiencing the same issue with memory usage growing until Suricata crashes, then going back to normal. This only seems to occur if the abuse.ch\urlhaus rule set is enabled. Disabling it again seems to have stopped the issue for now.
-
How much RAM do you have?
-
After updating to 18.1.13 I am now experiencing the same issue with memory usage growing until Suricata crashes, then going back to normal. This only seems to occur if the abuse.ch\urlhaus rule set is enabled. Disabling it again seems to have stopped the issue for now.
I can confirm the issue on several apu2c4 based systems on OPNsense 18.1.12 and 18.1.13.
As stated by rjb4526, the only workaround that prevents Suricata from crashing currently is to disable abuse.ch/URLhaus.
Some more background:
- Reinstalling Suricata didn't help.
- New downloads of all the rules didn't help.
- Issue is present on OPNsense 18.1.12 and 18.1.13.
- Suricata crashes with both Hyperscan and Aho-Corasick pattern matcher.
- The apu2c4 board contains an AMD GX-412TC CPU and 4 GB DRAM (which supports SSSE3)
-
How much RAM do you have?
8GB. RAM usage doesn't grow to 100%, though. Last I saw it grew to about 2.5GB before Suricata crashed.
-
I just upgraded to 18.1.13 and noticed some odd things, so I thought I'd try and add some details in the hope that it helps the always helpful OpnSense team
System is a Qotom Q355G4
CPU: i5-5250U
RAM: 8GB
NOTE: after every setting change below I rebooted to make sure results were "clean"
phase 1 - update to 18.1.13, reboot, reload all suricata rules with URLhaus disabled
a) using hyperscan - runs ok
system memory 1059M
suricata memory usage (from System->Diagnostics->Activity)
Mem Size: 2805M
Res: 350M
b) using aho-corasick - runs ok
system memory 1129M
suricata memory usage (from System->Diagnostics->Activity)
Mem Size: 2913M
Res: 443M
phase 2 - enable URLhaus, download all rules
a) using aho-corasick - runs ok
system memory 1794M
suricata memory usage (from System->Diagnostics->Activity)
Mem Size: 3565M
Res: 1069M
b) using hyperscan: crashes
phase 3: disable URLhaus
a) using hyperscan: crashes (but it worked above!!)
so, the only thing that was different in 1a above was that the rules for URLhaus showed "not installed" at the start, rather than just disabled. So, I downloaded all the rules again so it showed "not installed" again for URLhaus, and rebooted:
using hyperscan: works again
so, the URLhaus rule can't even be installed it seems, even if disabled.
FURTHER ODDITY:
after the crashes, when I changed settings and clicked reboot, the screen paused for a while but then came back to show stats rather than to the login screen. Also, uptime showed no reboot occurred. Tried twice, same thing. Also tried shutdown, which also didn't work!!
So, I had to power cycle after the failures to cause a reboot to happen. Very odd, thought I'd mention it.
I hope the above is helpful, please let me know if I can provide other info.
-
Perhaps the ruleset wont get deleted. There was a fix around, not sure If only for 18.7.
-
Despite abuse.ch's claims to the contrary on Twitter, the issue still isn't fixed. Enabling abuse.ch/urlhaus rules still results in Suricata crashing.
-
So you get an error in the logs? If not perhaps the ruleset is too big for you system?
-
So you get an error in the logs? If not perhaps the ruleset is too big for you system?
The only thing that shows up in the logs is Suricata crashing.
-
Can you watch the memory Bar in the Dashboard shortly after enabling Suricata?
-
Can you watch the memory Bar in the Dashboard shortly after enabling Suricata?
Yes. I refresh it several times over the course of a couple of minutes and watch the memory usage grow to about 2.8-3GB before it crashes. Once it crashes, memory usage goes back to a more normal number like 750MB-1GB depending on usage.
-
[Updated as I have answered in the wrong thread]:
I'm having issues here, too.
See https://forum.opnsense.org/index.php?topic=9512.msg43639#msg43639 for my details.