Suricata not using all system resource?

[annoniempjuh]

i still looking why with suricata on its not possible to get the 10Gbe speeds in the LAN.

i found the following thing that i don't understand:

Code: [Select]

2021-06-06T13:28:05 suricata[73887] [100195] <Notice> -- all 4 packet processing threads, 4 management threads initialized, engine started.
2021-06-06T13:28:05 suricata[73887] [100533] <Notice> -- opened netmap:pppoe1/T from pppoe1: 0x5f809706300
2021-06-06T13:28:05 suricata[73887] [100533] <Notice> -- opened netmap:pppoe1^ from pppoe1^: 0x5f809706000
2021-06-06T13:28:05 suricata[73887] [100526] <Notice> -- opened netmap:pppoe1^ from pppoe1^: 0x5f808c62300
2021-06-06T13:28:05 suricata[73887] [100526] <Notice> -- opened netmap:pppoe1/R from pppoe1: 0x5f808c62000
2021-06-06T13:28:05 suricata[73887] [100523] <Notice> -- opened netmap:ix0/T from ix0: 0x5f802b84300
2021-06-06T13:28:05 suricata[73887] [100523] <Notice> -- opened netmap:ix0^ from ix0^: 0x5f802b84000
2021-06-06T13:28:04 suricata[73887] [100506] <Notice> -- opened netmap:ix0^ from ix0^: 0x5f8007fc300
2021-06-06T13:28:04 suricata[73887] [100506] <Notice> -- opened netmap:ix0/R from ix0: 0x5f8007fc000
why is suriata only use 4 thread while the NIC has 6?:

Code: [Select]

ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> mem 0x7fffc00000-0x7fffdfffff,0x7fffe04000-0x7fffe07fff at device 0.0 on pci9
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 6 RX queues 6 TX queues
ix0: Using MSI-X interrupts with 7 vectors
ix0: allocated for 6 queues
ix0: allocated for 6 rx queues
ix0: Ethernet address:
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: netmap queues/slots: TX 6/2048, RX 6/2048
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> mem 0x7fffa00000-0x7fffbfffff,0x7fffe00000-0x7fffe03fff at device 0.1 on pci9
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 6 RX queues 6 TX queues
ix1: Using MSI-X interrupts with 7 vectors
ix1: allocated for 6 queues
ix1: allocated for 6 rx queues
ix1: Ethernet address:
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: netmap queues/slots: TX 6/2048, RX 6/2048

and suricata is running on one core at time:

Code: [Select]

27412 root 103 0 801 774 CPU9 9 0:42 100.00% /usr/local/bin/suricata -D --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
i have 6 cores (12 threads) available, why is suricata only using one core at time?

What can i do to get suricata use more threads on the NIC (2x 6 instead of 2x 4) and use more then one core at a time...

OPNsense 21.1.6-amd64
Ryzen 5 3600
Intel X540-T2

[lebertu]

I think the suricata conf is set by opnsense to auto for threading, I have a similar card to yours (x550) and 4 CPUS and when I set threads to 4 I get poor performance, 2 threads for 4 queues is ok and able to achieve wire rate.

https://suricata.readthedocs.io/en/suricata-5.0.6/capture-hardware/netmap.html

I have noticed the same behaviour you're describing on pfsense, suricata seems to be pinned to one core not sure why ?

 

[annoniempjuh]

i tried to make some manual changes in the config file, not much improve and sometimes suricata just wont start...
can't found much info on how to optimize Suricata on OPNsense...
I like Suricata and i don't wanna change to sensei yet, but its frustration with not getting the 10Gbe speeds is getting bigger and bigger, especially with a powerful cpu doing almost nothing...

hope someone cant help me to the solution...

[tuto2]

Hi,

Suricata on FreeBSD uses Netmap to achieve IPS functionality. Judging by your logs, you are indeed using netmap to bypass the host stack and enable Suricata to inspect packets straight off the wire.

Note the way ports are opened:

ix0/R (Receive thread) --> ix0^ (Host stack)
ix0^ (Host stack) --> ix0/T (Transmit thread)

This simply means that on initialization, netmap opens two "ports" - one on which to capture packets, at which point Suricata will be able to do it's thing, and another port that represents the host stack (using the '^' symbol), which is used by Suricata to forward inspected packets back to the host stack. The same principle applies on the transmit side (but reversed) - totalling a thread usage of 4 in a default setup.

The way Netmap is currently implemented does not allow for more than one thread to connect to the host stack on both the receive and transmit side. Manually increasing the amount of threads will not ensure a gain in throughput, and any measured increase in throughput will be wrong, since packets on different threads might not even reach Suricata and thus could potentially even skip by Suricata, due to a lack of synchronization.

In conclusion, Suricata on FreeBSD currently only supports one thread in IPS mode. However, Netmap has recently committed support for multiple threads towards the host stack in FreeBSD, and Suricata is in the process of integrating this into their software - so keep an eye on that.

Cheers,

Stephan

[binaryanomaly]

I am able to push through ~7 Gbps peak with Suricata enabled, virtualized with an intel X550-T2 with SR-IOV enabled.

The whole setup is still sort of experimental as I have some rather strange SR-IOV issues with OPNsense but with a bit of tinkering it works.

Important performance considerations were:
- In the KVM config set CPU type to host, to maximally benefit from CPU features/optimizations (I have an 11th Gen Intel(R) Core(TM) i5-11600 @ 2.80GHz)
- Use SR-IOV (I got worse performance passing through the whole PCI device - I still have no idea why ~3 Gpbs)
- Pattern Matcher: Hyperscan

[annoniempjuh]

i decided to give Sensei a change, and i have to say i kinda liked it. it also won't use all system resources but it get 1Gbe more  iperf speeds than Suricata...

at this moment i disabled Suricata, manly because the WAN uses PPPoE and Surciata(netmap) wont work with that. At least not in IPS mode. maybe i'm going to set it to IDS but is have to do more reading on that.

[binaryanomaly]

I'm coming from there, respectively had both activated which drags down the performance even more to around 3 Gbps here.

I somehow have more confidence in the

Code: [Select]

ET Pro Telemetry ruleset and this is why I decided to go with suricata only for the time being.

I may change my mind again at some point because the usability and out of the box reporting capabilities of suricata leave a lot to desire when compared to sensei.

Concerning the ruleset I think ET Pro Telemetry is superior.

[binaryanomaly]

at this moment i disabled Suricata

To my surprise I just ended up here as well.
After experiencing a lot of strange and unstable behavior I finally pulled the plug.

It may work better as a dedicated Linux based VM but on OPNsense/BSD for the time being the cost in terms of stability and performance is currently too high for me with my virtual infrastructure setup.

Also the whole network latency seems to have improved perceiveably w/o suricata.

A significant part of the suricata rules seem to be DNS and IP blacklists anyway which can easily be implemented by other means.

[ryanhaver]

at this moment i disabled Suricata

To my surprise I just ended up here as well.

Unless I have missed something...we are almost two years out from your posts and still dealing with this on FreeBSD with Netmap.

I have opted to roll with Zenarmor alone for now and stop using Suricata due to major stability and latency issues, along with a significant reduction in throughput due to only a single CPU core being utilized by Suricata. With that said Zenarmor still doesn't have multicore support either, although they advertise it on their website.

[binaryanomaly]

I had in the meanwhile switched on suricata again shortly after and have deactivated it again since ca. 2-3 months because suddenly strange failures occured that were even crashing the vmhost without any logs or hints.

Also the UI to configure suricata is imho not very good in terms of usability.

With both the above I currently do not see any value or benefit of activating suricata again.
I used to use zenarmor as well but couldn't find any value in addition to the DNS and L3/4 blocklists I am already using.

I think suricata could be a powerful component once the performance, stability and usability issues are addressed but I it's likely going to take years until we're there.