Suricata IDS/IPS ~56% slower than before update

[annoniempjuh]

today i did upgrade OPNsense end my server to 10Gbit NICs

hardware:
Intel Ethernet Converged Network Adapter X540-T2  (OPNsense)
Mellanox ConnectX-3 CX311A (unRAID server)
MikroTik Cloud Smart Switch 326-24G-2S+RM (switch)


Iperf results:

suricata OFF = cpu usage 40% / 51%

Code Select Expand

iperf3 -c 10.0.3.1 -t 60 -i 10
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 35558 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  3.03 GBytes  2.60 Gbits/sec    0    252 KBytes       
[  5]  10.00-20.00  sec  2.99 GBytes  2.57 Gbits/sec    0    246 KBytes       
[  5]  20.00-30.00  sec  2.98 GBytes  2.56 Gbits/sec    0    243 KBytes       
[  5]  30.00-40.00  sec  2.96 GBytes  2.54 Gbits/sec    0    209 KBytes       
[  5]  40.00-50.00  sec  2.93 GBytes  2.52 Gbits/sec    0    277 KBytes       
[  5]  50.00-60.00  sec  2.97 GBytes  2.55 Gbits/sec    0    260 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  17.9 GBytes  2.56 Gbits/sec    0             sender
[  5]   0.00-60.00  sec  17.9 GBytes  2.56 Gbits/sec                  receiver

iperf Done.

iperf3 -c 10.0.3.1 -t 60 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[  5] local 10.0.3.2 port 36642 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.82 GBytes  3.28 Gbits/sec                 
[  5]  10.00-20.00  sec  3.89 GBytes  3.35 Gbits/sec                 
[  5]  20.00-30.00  sec  3.82 GBytes  3.28 Gbits/sec                 
[  5]  30.00-40.00  sec  3.75 GBytes  3.22 Gbits/sec                 
[  5]  40.00-50.00  sec  3.60 GBytes  3.09 Gbits/sec                 
[  5]  50.00-60.00  sec  3.76 GBytes  3.23 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  22.6 GBytes  3.24 Gbits/sec  8384             sender
[  5]   0.00-60.00  sec  22.6 GBytes  3.24 Gbits/sec                  receiver

iperf Done.


suricata ON = cpu usage 59% / 76%

Code Select Expand

iperf3 -c 10.0.3.1 -t 60 -i 10
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 37868 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  2.80 GBytes  2.40 Gbits/sec    0   5.66 KBytes       
[  5]  10.00-20.00  sec  2.81 GBytes  2.42 Gbits/sec    0    272 KBytes       
[  5]  20.00-30.00  sec  2.78 GBytes  2.38 Gbits/sec    0    223 KBytes       
[  5]  30.00-40.00  sec  2.79 GBytes  2.40 Gbits/sec    0    240 KBytes       
[  5]  40.00-50.00  sec  1.53 GBytes  1.32 Gbits/sec    4   1.41 KBytes       
[  5]  50.00-60.01  sec  0.00 Bytes  0.00 bits/sec    2   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.01  sec  12.7 GBytes  1.82 Gbits/sec    6             sender
[  5]   0.00-61.65  sec  12.7 GBytes  1.77 Gbits/sec                  receiver

iperf Done.

iperf3 -c 10.0.3.1 -t 60 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[  5] local 10.0.3.2 port 38420 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.40 GBytes  1.21 Gbits/sec                 
[  5]  10.00-20.00  sec  1.37 GBytes  1.17 Gbits/sec                 
[  5]  20.00-30.00  sec  1.40 GBytes  1.20 Gbits/sec                 
[  5]  30.00-40.00  sec  1.39 GBytes  1.19 Gbits/sec                 
[  5]  40.00-50.00  sec  1.40 GBytes  1.20 Gbits/sec                 
[  5]  50.00-60.00  sec  1.41 GBytes  1.21 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  8.37 GBytes  1.20 Gbits/sec   18             sender
[  5]   0.00-60.00  sec  8.37 GBytes  1.20 Gbits/sec                  receiver

iperf Done.

[klamath]

I upgraded the kernel in my 20.7 install to 21.1 kernel and notice a speed drop when IPS is enabled.

IDS only:
Speedtest by Ookla

Server: ZochNet - Lincoln, TX (id = 20875)
ISP: Grande Communications
Latency:    22.87 ms   (0.73 ms jitter)
Download:   926.23 Mbps (data used: 1.0 GB)
Upload:    46.38 Mbps (data used: 66.3 MB)
Packet Loss:     0.0%

IDS/IPS:
Speedtest by Ookla

Server: ZochNet - Lincoln, TX (id = 20875)
ISP: Grande Communications
Latency:    19.63 ms   (1.04 ms jitter)
Download:   666.98 Mbps (data used: 626.3 MB)
Upload:    45.06 Mbps (data used: 67.3 MB)
Packet Loss:     0.0%

I did not have these speed issues when using the 20.7.5-next kernel branch with my hardware.

[Voodoo]

My monitoring shows increased CPU usage.

Before with 20.7.x it maxed out at 22%.

With 21.1.x it goes up to 80%.

With all tests I get my max bandwidth of 200mbit.

It does not happen with suricata disabled, the weird thing is that suricata is not set to listen on lan, only on dmz.

[thowe]

Is it possible that only XEN virtualized OPNsense instances could be affected? I have the impression that I have read a few times here in the forum about performance drops when switching to 21.1 in connection with XEN. But please take it with a grain of salt: I can't judge that myself, since I don't use XEN.

[andreaslink]

Is it possible that only XEN virtualized OPNsense instances could be affected?

No, I do not think so, I'm also running directly on bare metal and I was affected here as well. Franco pointed out the kernel to be responsible and this could be confirmed. So I can currently live with only IDS and no IPS activated until this is under better control, but downgrading kernel is in my mind currently the approbate solution in relation to the performance drop.

[franco]

Well, the issue still lies with FreeBSD stable/12 branch going backwards on performance. It is really an uphill battle to get fixes AND avoid regressions.


Cheers,
Franco

[AdSchellevis]

I haven't seen performance issues on my setup between these versions, but since there have been quite some fixes around netmap in different kernels, it's often a good idea to check if it's Suricata causing issues or netmap.

There is a simple "bridge" tool for netmap available in the kernel source directory, if people want to check netmap behaviour on their hardware they can always build the tool and create a bridge between the card and the host-stack to rule out certain driver issues.

To install it, you need the kernel source directory in place (/sur/src), you can use the build tools (https://github.com/opnsense/tools) to checkout all sources on your machine.

When the sources are in place, you can build the tools using the following commands (on amd64):

Code Select Expand


cd /usr/src/tools/tools/netmap/
make bridge



Next make sure netmap isn't used (no Suricata or Sensei) and create  a bridge between the physical connection and the host stack, assuming the interface in question is called vmx2, the command would look like:

Code Select Expand


/usr/obj/usr/src/amd64.amd64/tools/tools/netmap/bridge -i netmap:vmx2 -i netmap:vmx2


Wait a few seconds and start the test again with OPNsense in between. When netmap isn't interfering the test with or without bridge should show roughly the same numbers.

Best regards,

Ad

[klamath]

Is there any tooling that opnsense can provide or recommend the users experiencing slowness with IDS/IPS can consume to help close loop this issue?  I started using opnsense a few months ago and it seems that every minor and major release has some regression with netmap/iflib.  I think having some tooling the users can use to provide meaningful feedback to the dev team to chase down these issues would be useful to all.

I haven't seen performance issues on my setup between these versions, but since there have been quite some fixes around netmap in different kernels, it's often a good idea to check if it's Suricata causing issues or netmap.

There is a simple "bridge" tool for netmap available in the kernel source directory, if people want to check netmap behaviour on their hardware they can always build the tool and create a bridge between the card and the host-stack to rule out certain driver issues.

To install it, you need the kernel source directory in place (/sur/src), you can use the build tools (https://github.com/opnsense/tools) to checkout all sources on your machine.

When the sources are in place, you can build the tools using the following commands (on amd64):

Code Select Expand


cd /usr/src/tools/tools/netmap/
make bridge



Next make sure netmap isn't used (no Suricata or Sensei) and create  a bridge between the physical connection and the host stack, assuming the interface in question is called vmx2, the command would look like:

Code Select Expand


/usr/obj/usr/src/amd64.amd64/tools/tools/netmap/bridge -i netmap:vmx2 -i netmap:vmx2


Wait a few seconds and start the test again with OPNsense in between. When netmap isn't interfering the test with or without bridge should show roughly the same numbers.

Best regards,

Ad

[franco]

> I started using opnsense a few months ago and it seems that every minor and major release has some regression with netmap/iflib.

Interesting analysis...maybe because 20.7 was the first release that had netmap/iflib combo in the OS to deal with in kernel and FreeBSD support on it still is meagre at best.

Basically Sensei people had to go and sponsor work around netmap to get that somewhere back to where any version before 20.7 was. ;)

The iflib author hasn't done anything of value on his iflib work since and moved on to work on the wireguard kernel integration instead. It's a great outlook really with things like this going on in an OS.


Cheers,
Franco

[AdSchellevis]

@klamath I'm not sure why your quoting my message, but as said if it's purely about netmap, one could try to use bridge to pinpoint issues if they exists for their setup. It's definitely not the case that there are issues in all releases, we test the hardware we provide on periodic bases and haven't seen a lot of (major) issues ourselves.

Quite some reports about performance are related to too optimistic assumptions (e.g. expecting 1Gbps IPS on an apu board for example) or drivers which aren't very well supported (we ship what's being offered upstream, if support isn't great in FreeBSD for netmap, it highly likely isn't great on our end). IPS needs quite some computing power and isn't comparable to normal routing/firewall functions at all in terms of requirements.

When it comes to testing,  we tend to offer test kernels and release candidates on periodic bases. To help catching issues up front, please do test, document behaviour when experiencing issues, and try to track them to FreeBSD bug reports if they exists. When there are fixes available upstream, we often assess if we can backport them into our system. Quite some fixes have been merged in the last versions for various drivers (with quite some help from the Sensei people as Franco mentioned), I haven't seen side affects in terms of performance myself, but that doesn't mean they don't exist for some drivers.


Best regards,

Ad

[klamath]

Thank you for the response, the reason I quoted you I was trying to pin-point what you are wanting us to test for and what artifacts we can transmit back for follow up.   I am not trying to finger point or assign blame to anyone or any project, I am trying to see if there is some tooling that we the users can consume and send back on what we are seeing on our end to cut down on the "we didn't see that in our testing" back and forth as it doesn't help resolve the issues at hand.

If at the end of the day this level of back and forth is helpful and there is no standardized troubleshooting tooling we can use so be it, but I would like to work on getting this issue resolved.

That being said what are the next steps I and the other people having issues with the 21.1 kernel with IDS/IPS enabled provide you with narrowing down the issue?

@klamath I'm not sure why your quoting my message, but as said if it's purely about netmap, one could try to use bridge to pinpoint issues if they exists for their setup. It's definitely not the case that there are issues in all releases, we test the hardware we provide on periodic bases and haven't seen a lot of (major) issues ourselves.

Quite some reports about performance are related to too optimistic assumptions (e.g. expecting 1Gbps IPS on an apu board for example) or drivers which aren't very well supported (we ship what's being offered upstream, if support isn't great in FreeBSD for netmap, it highly likely isn't great on our end). IPS needs quite some computing power and isn't comparable to normal routing/firewall functions at all in terms of requirements.

When it comes to testing,  we tend to offer test kernels and release candidates on periodic bases. To help catching issues up front, please do test, document behaviour when experiencing issues, and try to track them to FreeBSD bug reports if they exists. When there are fixes available upstream, we often assess if we can backport them into our system. Quite some fixes have been merged in the last versions for various drivers (with quite some help from the Sensei people as Franco mentioned), I haven't seen side affects in terms of performance myself, but that doesn't mean they don't exist for some drivers.


Best regards,

Ad

[AdSchellevis]

@klamath no problem, to increase chances of gaining traction with an issue, best thing todo is to write down what you tested (exact equipment) and between which (kernel) versions you noticed a difference in performance. Like I tried to tell earlier, a lot of these issues aren't alike, so conditions matter.   Between 20.7.x and 21.1.x the number of changes are more limited, 20.1 isn't very comparable due to the upstream change to iflib (not our choice, just a fact of life).

If the number of moving parts are limited, it's easier to point to specific changes. Sometimes a simple iperf3 test with machines on both ends of the firewall is already enough to notice a difference. When the problem is "my network card worked great before iflib", I'm afraid people are really looking for voluntary kernel engineers. Maybe it helps to ask the vendor for better FreeBSD support, I don't know, realistically if it's not equipement we use, chances aren't very large people will spend a lot of time on these type of issues over here. (sometimes tracking these issues down costs many days, if the vendor doesn't really care, there aren't a lot of people spending that much of their spare time)

Best regards,

Ad

[klamath]

Running version 20.7.8 Business Edition 
Kernel 21.1 (FreeBSD cerberus.underworld.local 12.1-RELEASE-p12-HBSD FreeBSD 12.1-RELEASE-p12-HBSD #0  3c6040c7243(stable/21.1)-dirty: Mon Jan 25 12:27:52 CET 2021     root@sensey:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64)

Hardware: https://www.supermicro.com/en/products/system/Mini-ITX/SYS-E300-9D-8CN8TP.cfm
Ram: 64 GB ECC

When Suricata is enabled with IDS/IPS protection the max WAN speed is capped at around 650-670Mbps, with IPS mode disabled I can achieve full 1Gb/s down.

I did not see any of this issue with 20.7.5-NEXT, but since I upgraded to 20.7.8 and the -next kernels got removed I have no way to restore my performance back to satisfactory levels.

[klamath]

Seems that Freenas is having the same issues around iflib: https://jira.ixsystems.com/plugins/servlet/mobile#issue/NAS-107593

They posted a work around that may or may not help people here:

sysctl net.iflib.min_tx_latency=1

[Space]

Hi,

that workaround does not help with my system (igb driver). iperf3 stays at ~600Mb/s ...

Best regards, Space