Speedtests Sensei and Suricata

[binaryanomaly]

Hi,

Just wanted to share my speedtestresults to see if folks here observe similar behaviour.
VM-to-VM, Suricata using Hyperscan pattern matcher with ET Pro Telemetry rules, speedtest-cli (Ookla).

• Suricata only ~8 Gbs
• Sensei only ~4.7 Gbs
• Suricata plus Sensei ~3 Gbs

The speed decrease when using Suricata and Sensei together is quite significant.
Anything one can do about this?




Detailed test results:

Suricata❌, Sensei❌

Code: [Select]

    Latency:     1.90 ms   (0.49 ms jitter)
   Download:  8047.50 Mbps (data used: 4.0 GB)
     Upload:  7408.12 Mbps (data used: 8.7 GB)

Suricata✅, Sensei❌

Code: [Select]

    Latency:     1.49 ms   (0.19 ms jitter)
   Download:  8036.28 Mbps (data used: 5.6 GB)
     Upload:  7309.16 Mbps (data used: 8.9 GB)

Suricata❌, Sensei✅

Code: [Select]

    Latency:     2.84 ms   (0.19 ms jitter)
   Download:  4705.11 Mbps (data used: 7.4 GB)
     Upload:  5264.34 Mbps (data used: 6.4 GB)

Suricata✅[, Sensei✅

Code: [Select]

    Latency:     2.92 ms   (0.28 ms jitter)
   Download:  2819.69 Mbps (data used: 3.0 GB)
     Upload:  1090.61 Mbps (data used: 670.7 MB)


[sy]

Hi,

Did you try Sensei in bypass mode?

[binaryanomaly]

Hi,

Did you try Sensei in bypass mode?

No, stopped the engine completely.
Why should I test in bypass mode? I either would want it off completely or fully functional.

[sy]

Hi,

The bypass will provide us the info that slowness is due to engine or netmap. In Bypass mode, Sensei just forward the packets. If the results are the same in bypass mode, we should check the netmap side.

[binaryanomaly]

Hi,

The bypass will provide us the info that slowness is due to engine or netmap. In Bypass mode, Sensei just forward the packets. If the results are the same in bypass mode, we should check the netmap side.

Ok thanks, I understand.

The results are almost identical in bypass mode:

Code: [Select]

    Latency:     2.79 ms   (0.23 ms jitter)
   Download:  2705.55 Mbps (data used: 2.6 GB)
     Upload:  1146.04 Mbps (data used: 1.2 GB)


[W0nderW0lf]

I have the same problem, but since I dont need that much speed for the stuff I do, I have to life with 1/3 of my full speed.
It's depending on your hardware. I still use a J1900 CPU and thats a pretty old and slow one

[binaryanomaly]

Well, I can live with ~3 Gbps that's not really an issue 

Nevertheless the drop in performance is significant and probably not without impact on weaker hardware. Also it would allow to waste less resources in virtualization scenarios, generate less heat, etc., etc.

I have a 11th Gen Intel(R) Core(TM) i5-11600 @ 2.80GHz at work here, 4 cores for OPNsense, somewhat doubt that it is the bottleneck. On the other hand it seems to be somewhat stuck around 2.7/8 Gbps also with the previous hardware.

[manos]

I had a similar issue with speedtest results dropping by 50% when both Suricata and Zenarmor were enabled until I set the "Pattern matcher" to "Hyperscan" under Services->Intrusion Detection->Administration.
Your processor needs to support the SSE3 extension though.

[johndchch]

I have a 11th Gen Intel(R) Core(TM) i5-11600 @ 2.80GHz at work here, 4 cores for OPNsense, somewhat doubt that it is the bottleneck. On the other hand it seems to be somewhat stuck around 2.7/8 Gbps also with the previous hardware.

since the 11600 is a 6-core/12-thread cpu I presume '4 cores for opnsense' means you're running virtualised? Are passing the NICs thru to the VM, or you using vmxnet3/virtio?

[Vilhonator]

To my knowledge that's pretty normal.

 IDS and IPS both will slow down connections and there's very little you can do to improve it. In fact, if your network speed wouldn't slow down at all or to some degree, it would pretty much mean things aren't working as they should.

In otherwords, IPS and IDS do slow down connections, but it's more important to see how many clients can use network services, before connection slows down to a level, in which you can see it's slow without actually using any speedtest methods.

[binaryanomaly]

since the 11600 is a 6-core/12-thread cpu I presume '4 cores for opnsense' means you're running virtualised? Are passing the NICs thru to the VM, or you using vmxnet3/virtio?

I run it virtualized on promox. I am currently using bridged virtio NICs but have also tested passthrough but the difference was not significant if I recall correctly. I will set it up in passthrough again, soon.

Regarding the values of the initial post, I realized later that when switching on/off suricata there's a delay until it's fully up so the the 8 Gbps measurement is not accurate it's rather 3 Gbps with Suricata on. I think it's related to single CPU restrictions.

[johndchch]

I run it virtualized on promox. I am currently using bridged virtio NICs but have also tested passthrough but the difference was not significant if I recall correctly. I will set it up in passthrough again, soon.

I'm running under esxi - full NIC passthrough is fine, but using portgroup level vlans ( together with a trunk port on the switch ) to share the 10gbe link between the opnsense vm ( with wan and lan on separate vlans - as I only have 1gbps internet so don't really need a full 10gbe connection just for opnsense)  and an ubuntu vm I get a MASSIVE hit ( as in <1gb/s ) on iperf3 speed both in the opnsense vm and the other vm when sensei comes up ( as sensei puts the LAN port it's monitoring into promiscuous mode - vmware's KB warns with good reason about speed issues when allowing promiscuous mode )

I can get around it by inhibiting the ability to enable promiscuous mode in the vswitch settings - at which point iperf3 speed goes back up to what i'd expect ( full 9.4gb/s to the ubuntu vm and about 6gb/s to the opnsense vm ) - since promiscuous on the LAN side doesn't really make much sense anyway ( I want to monitor traffic thru the firewall only)  that works for me



Regarding the values of the initial post, I realized later that when switching on/off suricata there's a delay until it's fully up so the the 8 Gbps measurement is not accurate it's rather 3 Gbps with Suricata on. I think it's related to single CPU restrictions.
[/quote]

[lilsense]

The speed decrease when using Suricata and Sensei together is quite significant.
Anything one can do about this?

Yes, it's quite simple.

-Build an ASIC chip for Suricata and another for Sensei.
-Write code for the OS to talk to the ASIC chips.
-test and let us know...

There's reason why people (ISP's) buy Fortinet/Palo Alto/Juniper is because of the ASIC chips eat this stuff at line rates.