Suricata IDS/IPS ~56% slower than before update

[grimm26]

any better on 21.1.1?

[Helle]

I have issues with high cpu load and monit alerts me every now and then that CPU is too high and it seems very strange, The high load doesn't seem to be caused by excessive traffic and is present even when almost nothing is going on.

Cpu load is between 50% and 100% even if traffic is as low as 3Mbit/s in and out
This was never a problem before 21.1

Nothing has been changed except the Opensense upgrades, no firewall changes or IPS rules,

My system is an APU 2d4 which has intel nics

[mimugmail]

Did you check which process spikes?

[Helle]

I did not but I will keep an eye on this.

Did some testing changing IPS from Hyperscan to Aho-Corasick and it made a big change in CPU Load... After a few hours, I switched back to Hyperscan and the load issue did not reappear.

I suspect Suricata at the moment but it may need some more invstigation.

[ilikenwf]

So this update to 21.7 coincided with the time when my ISP bumped me from 300 down to gigabit...

I'm using a 2nd gen i5 with minimal rules enabled, and even just monitoring LAN, I also get CPU spikes in the suricata process above 100%, so threads I assume...

My down speed suffers as a result, and I get 300-400 megabits down, versus bursting at/above 920 with suricata disabled.

With the hardware and config I have this really shouldn't be the case...

[mimugmail]

How many rules and which pattern matcher? My oldest i5 to test was 5th gen

[ilikenwf]

I'm using Hyperscan, with 46355 rules enabled...that may not be "not very many" I suppose, but it's a lot less than I ran without issue before the update and before the ISP upgraded me to gigabit from 300 down...

Without suricata disabled, I run 0-3% cpu most of the time, sometimes spiking if I do a speed test to 9%...

And yes, this is with all the hardware acceleration disabled on the NICs, as netmap still doesn't support any of it/there are bugs in some hardware and drivers...

I have tried with Aho-Corasick but it didn't seem to affect CPU.

[mimugmail]

On the newest Atom you get around 300mbit too, maybe the second gen doesnt give you more

https://www.routerperformance.net/opnsense/opnsense-performance-scope7-1510-21-1-6/

[ilikenwf]

Interesting...it could be - or it could be some kind of thread config is needed to really get suricata running reliably on the machine.

The entire reason I use this machine (a corebooted thinkpad t430) is that it runs coreboot...I could virtualize on my rackmount vm hosts, but that defeats the purpose of my paranoia's love of coreboot.

[ilikenwf]

So, in IDS only mode, it can get up around 400-500 megabits, with only 325 or so rules enabled...

I find all of this troubling because someone else with a protectli i5 dual core on reddit mentions getting up to 600 down (maxing out his connection) with an older version of opnsense and suricata around 11 months ago.

[mimugmail]

20.1 is indeed a bit faster than 20.7. after 20.7 everything is same. IDS mode usually gives you wirespeed, then you CPU might be just too old

[klamath]

So some things that might help, decrease ring size of the network interface that is being monitored. Disable HT, Enable/Disable some of these bios settings [1], make sure the system is set to performance mode for CPU Freq scaling.


https://www.academia.edu/33882347/Suricata_Extreme_Performance_Tuning

[ilikenwf]

I ended up adjusting ring size, and also am using stream dropping - most of the things I catch tend to be small and not involved in large streams of data...

[carlcedin]

I'm also having the same issue right now.
currently running version 21.1

2021-08-23T23:22:08   dpinger[15656]   WAN-GW 10.x.x.x: sendto error: 64   
2021-08-23T23:22:04   dpinger[15656]   WAN-GW 10.x.x.x: sendto error: 50   
2021-08-23T23:21:21   dpinger[84453]   GATEWAY ALARM: WAN-GW (Addr: 10.x.x.x Alarm: 0 RTT: 37243us RTTd: 157235us Loss: 0%)   
2021-08-23T23:21:21   dpinger[15656]   WAN-GW 10.x.x.x: Clear latency 37243us stddev 157235us loss 0%   
2021-08-23T23:20:24   dpinger[60798]   GATEWAY ALARM: WAN-GW (Addr: 10.x.x.x Alarm: 1 RTT: 558880us RTTd: 2270229us Loss: 11%)   
2021-08-23T23:20:22   dpinger[15656]   WAN-GW 10.x.x.x: Alarm latency 558880us stddev 2270229us loss 11%   
2021-08-23T23:20:15   dpinger[15656]   WAN-GW 10.x.x.x: sendto error: 55   
2021-08-23T23:16:31   dpinger[99790]   GATEWAY ALARM: WAN-GW (Addr: 10.x.x.x Alarm: 0 RTT: 33898us RTTd: 191777us Loss: 5%)   
2021-08-23T23:16:31   dpinger[15656]   WAN-GW 10.x.x.x: Clear latency 33898us stddev 191777us loss 5%   
2021-08-23T23:15:13   dpinger[15656]   WAN-GW 10.x.x.x: sendto error: 55   
2021-08-23T23:15:12   dpinger[15656]   WAN-GW 10.x.x.x: sendto error: 55   
2021-08-23T23:12:08   dpinger[42452]   GATEWAY ALARM: WAN-GW (Addr: 10.x.x.x Alarm: 0 RTT: 13809us RTTd: 67732us Loss: 0%)   
2021-08-23T23:12:08   dpinger[15656]   WAN-GW 10.x.x.x: Clear latency 13809us stddev 67732us loss 0%   
2021-08-23T23:11:32   dpinger[63388]   GATEWAY ALARM: WAN-GW (Addr: 10.x.x.x Alarm: 1 RTT: 664889us RTTd: 1880460us Loss: 10%)   
2021-08-23T23:11:32   dpinger[15656]   WAN-GW 10.x.x.x: Alarm latency 664889us stddev 1880460us loss 10%   
2021-08-23T23:10:38   dpinger[65637]   GATEWAY ALARM: WAN-GW (Addr: 10.x.x.x Alarm: 1 RTT: 1367329us RTTd: 4004130us Loss: 28%)   
2021-08-23T23:10:38   dpinger[15656]   WAN-GW 10.x.x.x: Alarm latency 1367329us stddev 4004130us loss 28%   
2021-08-23T23:01:29   dpinger[39719]   GATEWAY ALARM: WAN-GW (Addr: 10.x.x.x Alarm: 0 RTT: 21698us RTTd: 87994us Loss: 0%)   
2021-08-23T23:01:29   dpinger[15656]   WAN-GW 10.x.x.x: Clear latency 21698us stddev 87994us loss 0%   
2021-08-23T23:01:20   dpinger[38796]   GATEWAY ALARM: WAN-GW (Addr: 10.x.x.x Alarm: 1 RTT: 25364us RTTd: 94972us Loss: 15%)

I have some intermittent issues here.
After I turn off the IPS mode, connection is stable.

I'm also having some problem with Sensei keep crashing when enable the Suricata.
WAN using 1G and my LAN are connected with 10G
Do you guys have any ideas about this?

[jordaneliza97]