Suricata IPS 10Gbps

[seed]

I have been interested in intrusion prevention and Suricata for several years. I find suricata fascinating.

However, it is hard to find information on setups with 10Gbps IPS. I would be very interested if one of you is running such a setup on OPNsense and which HW is used.

Currently the stream of an interface runs through a CPU core. So the throughput is very limited.

Alternatively to the HW question I would be interested when and if Suricata (IPS mode) on OPNsense uses all CPU cores.

The thing is, I find it a bit unfortunate that "commercial providers" offload IPS.
On the HW systems then e.g. a FPGA or ASIC is installed which then executes IPS with wirespeed. Of course, OPNsense cannot implement this solution, but I just wanted to mention this for the context.

[guenti_r]

Don´t compare apples with pears.
The Asic´s or FPGA´s looks good on paper, but not in real scenarios.
Most of these "commercial providers" using a similar IPS-Engine like Snort or Suricata 

For example, for an "High-Performance-Setup" we using OPNSense in an virtualized HA-Stack (Proxmox).
Search for CPU´s with high clock rate.
Some "standard"-Blades with modern Xeon´s or AMD Epyc should be enough for Suricata 

Example above, 2 OPNSense in HA with Suricata (with a lot! of rules), average 20 TB mixed traffic per day, the CPU idles around 2-4%.

Edit: Suricata uses all cores!

Edit2: One OPNSense HA-Setup with 2* HP DL380 G10+ (Xeon Gold 5218), another with 2 Supermicros with AMD Epyc 7443p.

[seed]

Don´t compare apples with pears.
The Asic´s or FPGA´s looks good on paper, but not in real scenarios.

I mentioned this for context.
Fortigate uses something like this, for example. Alternatively, there are the NICs from napatech.
Please explain which "real" scenarios you refer to.

Search for CPU´s with high clock rate.

I know. Since Suricata distributes the individual interfaces to individual CPUs in IPS mode (runmode: workers), the single core performance is so important. A 10G inteface would therefore need a CPU that can do this in single core. Alternatively 2x 10G as lagg and then distribute the traffic over it. Then the Suricata distributes the traffic of the individual lagg members on cores = number of lagg members.

Example above, 2 OPNSense in HA with Suricata (with a lot! of rules), average 20 TB mixed traffic per day, the CPU idles around 2-4%.

Does this refer to the IPS mode? Which NIC? Lagg? how many interfaces?

[guenti_r]

"Real scenario" -> Master-Gateway to protect Datacenters (Cloud/Applicationprovider).
Before we using Fortigates with catastrophal experience (buggy Firmware, slow IPS, alot Hardware defects)

Single core performance is always important 

Maybe our situation is not the same as yours because we virtualized OPNSense with Proxmox, so these things are hardware-independend for us (virtio-nics with multiqueue).

Yes, IPS-Mode. Virtio-NIC. No LAGG. Underlying Hardware-NICs: Mellanox 100GB Dualport or Intel X7** Dualport.

[lilsense]

Um, Yeah,... I don't buy one bit of "Fortinet sucks" junk...

Every appliance including OPNsense have bugs up the yingyang and need to be fixed.

Now, Fortinet does use their processing of various functions on ASIC as opposed to OPNsense which uses CPU. It is one the reasons why Fortinet can push terabits and it's why they are ISP class firewalls including their itty-bitty boxes.

When I read Virtualization, then all this high performances stuff is out the door.

One thing is fact, hardware always performs better than any software.
Here's the Matrix for your reference...

https://www.fortinet.com/content/dam/fortinet/assets/data-sheets/Fortinet_Product_Matrix.pdf

[seed]

A real guide with which hardware which speed is achievable does not really exist (apart from DECISOs own HW).

I suspect that iflib together with the igb driver and suricata is slowing things down somewhere. This would explain why guenti_r achieves good performance with the HW:

Underlying hardware NICs: Mellanox 100GB Dualport or Intel X7** Dualport

.

It would be ideal, and I hope so, if Deciso would produce hardware with acceleration e.g. via FPGA. Then interface speed would be equal to IPS speed.

Until then I can only buy the fastest CPU (single thread) possible to get some speed.

[lilsense]

As per the understanding, all the performance is based on hardware used. I do not believe Decisio is going to test everything and only test their own and provide the data like they currently are. If you look at the Fortinet Matrix, you can also see that all those are really dependent on how much services are running on the system.

[seed]

I did not write that DECISO should test all hardware, but only that DECISO logically knows only the test results from its own hardware ;-)

What would be nice is if someone with 10Gbps IPS comes forward and shares their experiences or hardware selection to provide others with an orientation in the hardware selection.

In other words, bare metal.

Which CPU, RAM (size and speed), motherboard, NIC, storage, number of Suricata rules, network connectivity, benchmark results......

[lilsense]

It really depends on the type of traffic, when it comes to IPS. if you look at the Fortinet, there's a * which mentions Enterprise, but not sure what that is... LOL

[seed]

no shit sherlock.

tools like cisco trex exist to measure those things.

[guenti_r]

Why not test it in a virtual appliance?
Use an "old" server, put OPNSense in a VM and measure.
I have done this several times.

Fortigate´s look good on paper, but are horrible in real world.
Here we have around 60 Forti´s for the trash can....
Every Forti is replaced with OPNSense Appliances, from DEC690 up to big HA-Clusters and VMs.

These are daily experiences. We have a lot of Customers where i had replaced these horrible Forti-Boxes.

IPS is a little beast, Suricata is fast enough to compete with these "Commercial" boxes.

Edit: Answer the question, on an HA-OPNSense-Cluster, 222450 rules are enabled.

[seed]

As I had already written at the beginning:
"I have been interested in intrusion prevention and Suricata for several years"
I have already been using Suricata for 4 years.
I have used OPNsense on various hardware appliances and have also run benchmarks several times. In the most diverse configurations. And achieved results from 500Mbit to 2Gbit. On different hardware.

All this is known to me. I only mentioned fortigate because they do IPS in hardware. I hope that is understandable. It was never about the direct comparison of OPNsense to Fortigate!

It is simply about the question of what hardware is required for 10Gbps IPS. Nothing else! I hope that is now understandable for all.

[guenti_r]

It really depends on the type of traffic, when it comes to IPS. if you look at the Fortinet, there's a * which mentions Enterprise, but not sure what that is... LOL

Full agree. Try for fun on a Fortigate enabling more than 10000 IPS Rules and it crashes instantly.
Good Marketing, bad (stolen GPL-Code) Product.
Only the highest priced Forti´s have enough power to play reasonable with IPS in real world scenario.
These Asic´s are slower than some Smartphones out there 
Take a look on de hardware specs, it will bring you a smile in your face:
https://yurisk.info/2021/03/14/Fortigate-Firewalls-Hardware-CPU-model-and-number-Memory-size-datasheet-table/

[guenti_r]

It is simply about the question of what hardware is required for 10Gbps IPS. Nothing else! I hope that is now understandable for all.

Did you read the answers? Re-read my first one.

[guenti_r]

Until then I can only buy the fastest CPU (single thread) possible to get some speed.

Suricata is multi-threaded, take a look:
https://suricata.readthedocs.io/en/suricata-5.0.3/configuration/suricata-yaml.html#threading