18.7 abuse.ch/urlhaus rules still cause Suricata crash

[milkywaygoodfellas]

Enabling this rule set still causes RAM usage to grow until Suricata crashes... there is nothing in the Suricata log and the only entry in the general log is "kernel: pid (suricata), uid 0: exited on signal 6 (core dumped)."

Disabling the rule set remains the only way to keep Suricata from crashing.  I've tried reporting the issue to abuse.ch as well but haven't really gotten any response except "fixed," which it isn't...

Please let me know if there's anything I can provide to help narrow down the cause of the issue.

[mimugmail]

You can only check If you double the size and see if it happens again.

[guest18611]

You can only check If you double the size and see if it happens again.

Double the RAM Size? I have 8GB RAM and have the same issue. I could test it later with 16GB RAM...

[mimugmail]

When you grad the file it's nearly 30000 rules:

https://urlhaus.abuse.ch/downloads/ids/

This will eat up many ram. I'll try to ping the author perhaps he can integrate a cirular limit

[milkywaygoodfellas]

You can only check If you double the size and see if it happens again.

Unfortunately I don't have UDIMMs around that I can do that with... quite expensive for a home user

I have 8GB of RAM in it already and the RAM usage never goes to 100%.  Last time, it went to around 25-30% before crashing.  I don't think it's an issue with the amount of RAM.

This also never happened before whatever change was made to the rule set that introduced the HTML parsing error just recently.  Since then, I have not been able to enable this ruleset.  Considering Suricata's version was not changed in OPNsense I'm inclined to believe there's still an issue in the rule set itself but... either way, it was not crashing Suricata a couple of weeks ago and it is now.

[mimugmail]

I talked with the author, there will come many more rules, so I think using this list is only for high end hardware. Probably abuse.ch will stop this ruleset.

[guest18611]

tested with 16GB RAM...same Error. RAM is filling and then crashes.
I only have activated abuse.ch/spamhaus rules.

Log says:

Code: [Select]

Aug 2 21:06:41 suricata: [100247] <Notice> -- rule reload starting
Aug 2 21:05:50 suricata: [100247] <Notice> -- all 2 packet processing threads, 4 management threads initialized, engine started.
Aug 2 21:05:50 suricata: [100243] <Notice> -- This is Suricata version 4.0.5 RELEASE

[milkywaygoodfellas]

I talked with the author, there will come many more rules, so I think using this list is only for high end hardware. Probably abuse.ch will stop this ruleset.

How much more "high end" do you need to get with an edge firewall than an x86 box with 8-16GB of RAM?  Again, it doesn't even get to 100% RAM usage before crashing out.
There is clearly an issue with the rule set itself here, and it's clear now that abuse.ch is not interested in fixing it.  RAM is clearly not the issue here.