Suricata IDS/IPS ~56% slower than before update

[andreaslink]

I updated Thursday evening to OPNsense 21.1-amd64 and realized next morning that routed permanent video streams between LAN and WAN were significant slower until they broke very soon.

To the background, I have OPNsense running within my local network separating networks, so there is another router before reaching the Internet, which allows me 1GBit speedtests via OPNsense within my infrastructure.
So, in short I have LAN <-> OPNsense --> WAN <--> FritzBox --> Internet, this allows me stress tests from my LAN into the WAN without going through the slower internet connection of the provider. If it matters, I am running OPNsense on a 4 core Intel Xeon E5506, 20GB RAM, 2x Broadcom NetXtreme II BCM5709, 4x Intel 82580. Sensei currently deactivated.

After analysis I figured out that IDS/IPS is the root cause here. I came updated from 20.7.8_4 were everything was fine and as I read here https://opnsense.org/opnsense-21-1-marvelous-meerkat-released/ there are no changes made to Suricata within the release. I did not make any changes to the related setup or rules etc.

So, I made some interesting iperf3 measurements.

OPNsense v20.7.8_4:
Host LAN-net <-> Host WAN-net with IDS/IPS activated --> ~550 MBit/s

OPNsense v21.1:
Host LAN-net <-> Host WAN-net with IDS/IPS activated:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  1.68 GBytes   240 Mbits/sec  128             sender
[  5]   0.00-60.17  sec  1.68 GBytes   239 Mbits/sec                  receiver

Host LAN-net <-> Host WAN-net no IDS/IPS:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  2.54 GBytes   729 Mbits/sec  978             sender
[  5]   0.00-30.01  sec  2.54 GBytes   728 Mbits/sec                  receiver


OPNsense <-> Host WAN-net no IDS/IPS:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  3.29 GBytes   942 Mbits/sec  408             sender
[  5]   0.00-30.00  sec  3.29 GBytes   941 Mbits/sec                  receiver

As a result I can see within the update a performance dropped from ~550 Mbit/s down to ~240 Mbit/s, which is a performance drop of ~310 MBit/s aka 56%, which I cannot explain but measure. My overall routing power between LAN and WAN seems to be around 729 Mbit/s, which is acceptable for me as quite some video streams were passing through the firewall during measurement and where I do not have a comparable value from before the update.

Any suggestions what causes this IDS/IPS impact? Can someone second this behavior on his setup as well? I know for future only-internet-connections, this might be sufficient, but currently I feel unhappy with the result as it just came with the update to 21.1.
Looking forward for hints, ideas and comments.

[seed]

I have seen the same bahavior. 50% - 60% performance loss in suricata.
It feels like every update somehow reduces the overall suricata performance.

Why is that?

HW:
https://bsd-hardware.info/?probe=453d257afe

EDIT:
with the hardware listed above i was able to reach gigabit speed with more then ~40000 Suricata rules with older Software builds. After sorting out rules that i dont need im down to ~25000. But the performance is still decreasing.

[Superduke]

Posted in the Suricata sub-board, but it certainly appears that Suricata performance has degraded with the 21.1 upgrade.

In my case, my Xbox access to Gamepass basically became Null with Suricata enabled and back to normal (~500MBit/s) disabled....no performance issues prior to the OPN upgrade, same rules (ALL) etc.

I tried disabling the new policy approach but it didn't seem to matter.

FWIW....

[franco]

It's probably more iflib patching on FreeBSD stable/12 ... You can install the older kernel to see...

# opnsense-update -zkr 20.7.8
# opnsense-shell reboot


Cheers,
Franco

[seed]

Thanks for the reply.

i downgraded the kernel and rebootet.
The performance is nonetheless far under my expectations:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  44.5 MBytes   373 Mbits/sec   47    626 KBytes       
[  5]   1.00-2.00   sec  78.7 MBytes   660 Mbits/sec    0    711 KBytes       
[  5]   2.00-3.00   sec  77.4 MBytes   649 Mbits/sec    1    559 KBytes       
[  5]   3.00-4.00   sec  78.7 MBytes   660 Mbits/sec    0    656 KBytes       
[  5]   4.00-5.00   sec  77.4 MBytes   650 Mbits/sec    0    741 KBytes       
[  5]   5.00-6.00   sec  74.9 MBytes   628 Mbits/sec    5    585 KBytes       
[  5]   6.00-7.00   sec  78.7 MBytes   660 Mbits/sec    0    680 KBytes       
[  5]   7.00-8.00   sec  78.7 MBytes   660 Mbits/sec    0    764 KBytes       
[  5]   8.00-9.00   sec  78.6 MBytes   660 Mbits/sec    8    618 KBytes       
[  5]   9.00-10.00  sec  78.7 MBytes   660 Mbits/sec    0    710 KBytes       

[franco]

I don't understand this:

> The performance is nonetheless far under my expectations

Well, it is the same as 20.7.8 with the old kernel or not?

Because... there have been no other moving parts in the major iteration that would cause this.

If your expectations are higher in any case you should probably skip right to https://bugs.freebsd.org/bugzilla/


Cheers,
Franco

[seed]

I made an odd finding.

The Performance stays at around 650mbits. It doesnt matter if 22.000 or 10.000 rules are loaded.

[klamath]

I think a lot of the fixes in the -next kernels helped Suricata performance.  I recently upgraded to 20.7.8 and also upgraded my 20.7.5-next kernel to 20.7.8 and noticed a speed drop.  My WAN inspection went from 1Gb down to 630-650Mbps.  Turning off Suricata while running 20.7.8 kernel and the speed returned to -next levels.

Tim

[annoniempjuh]

i am on OPNsense 21.1 and i don't have any problem?

Code: [Select]

iperf3 -c 10.0.3.1 -u -t 60 -i 10 -b 1000M
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 60596 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec  813711 
[  5]  10.00-20.00  sec  1.10 GBytes   943 Mbits/sec  813645 
[  5]  20.00-30.00  sec  1.10 GBytes   943 Mbits/sec  813746 
[  5]  30.00-40.00  sec  1.10 GBytes   943 Mbits/sec  813787 
[  5]  40.00-50.00  sec  1.10 GBytes   943 Mbits/sec  813730 
[  5]  50.00-60.00  sec  1.10 GBytes   943 Mbits/sec  813777 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  6.58 GBytes   943 Mbits/sec  0.000 ms  0/4882396 (0%)  sender
[  5]   0.00-60.00  sec  6.56 GBytes   939 Mbits/sec  0.011 ms  20901/4882368 (0.43%)  receiver

iperf Done.
Hardware:
AMD Ryzen 3 2200G with Radeon Vega Graphics (4 cores)
8GB RAM
Intel PRO/1000 PT Dual Port Server Adapter (PCI-e 4x) (driver: EM)

when i was on OPNsense 20.1.8_1 it was:

Code: [Select]

iperf3 -c 10.0.3.31 -u -t 60 -i 10 -b 1000M
Connecting to host 10.0.3.31, port 5201
[  5] local 10.0.3.1 port 44924 connected to 10.0.3.31 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec  856118
[  5]  10.00-20.00  sec  1.16 GBytes  1.00 Gbits/sec  856870
[  5]  20.00-30.00  sec  1.16 GBytes  1000 Mbits/sec  857061
[  5]  30.00-40.00  sec  1.16 GBytes  1.00 Gbits/sec  856166
[  5]  40.00-50.00  sec  1.16 GBytes  1000 Mbits/sec  857113
[  5]  50.00-60.00  sec  1.16 GBytes  1.00 Gbits/sec  857192
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  6.98 GBytes  1000 Mbits/sec  0.000 ms  0/5140520 (0%)  sender
[  5]   0.00-60.00  sec  3.34 GBytes   479 Mbits/sec  0.046 ms  2680818/5140353 (52%)  receiver

iperf Done.
next week i'm going to upgrade to 10Gbe nic and fiber, will test if there will be a decrease of performance...

[seed]

@annoniempjuh looking at this numbers i assume that you are using suricata in IDS mode. My throughput was with IPS Mode.

What decreased the performance between 20.7.5 and 20.7.8?

[annoniempjuh]

Suricata is in IPS mode
i only tested v20.1.8_1 and v21.1

[seed]

@annoniempjuh you tested iperf3 with UDP. using udp i get simila numbers.
My result [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  44.5 MBytes   373 Mbits/sec   47    626 KBytes       
[  5]   1.00-2.00   sec  78.7 MBytes   660 Mbits/sec    0    711 KBytes       
[  5]   2.00-3.00   sec  77.4 MBytes   649 Mbits/sec    1    559 KBytes       
[  5]   3.00-4.00   sec  78.7 MBytes   660 Mbits/sec    0    656 KBytes       
[  5]   4.00-5.00   sec  77.4 MBytes   650 Mbits/sec    0    741 KBytes       
[  5]   5.00-6.00   sec  74.9 MBytes   628 Mbits/sec    5    585 KBytes       
[  5]   6.00-7.00   sec  78.7 MBytes   660 Mbits/sec    0    680 KBytes       
[  5]   7.00-8.00   sec  78.7 MBytes   660 Mbits/sec    0    764 KBytes       
[  5]   8.00-9.00   sec  78.6 MBytes   660 Mbits/sec    8    618 KBytes       
[  5]   9.00-10.00  sec  78.7 MBytes   660 Mbits/sec    0    710 KBytes 


was with plain settings: iperf3 -c <serverip>

What i ment with "What decreased the performance between 20.7.5 and 20.7.8?" was refering to klamath post.
Still this question remains unanswered. Maybe franco can shine a little light on this.

[annoniempjuh]

@annoniempjuh you tested iperf3 with UDP. using udp i get simila numbers.
My result [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  44.5 MBytes   373 Mbits/sec   47    626 KBytes       
[  5]   1.00-2.00   sec  78.7 MBytes   660 Mbits/sec    0    711 KBytes       
[  5]   2.00-3.00   sec  77.4 MBytes   649 Mbits/sec    1    559 KBytes       
[  5]   3.00-4.00   sec  78.7 MBytes   660 Mbits/sec    0    656 KBytes       
[  5]   4.00-5.00   sec  77.4 MBytes   650 Mbits/sec    0    741 KBytes       
[  5]   5.00-6.00   sec  74.9 MBytes   628 Mbits/sec    5    585 KBytes       
[  5]   6.00-7.00   sec  78.7 MBytes   660 Mbits/sec    0    680 KBytes       
[  5]   7.00-8.00   sec  78.7 MBytes   660 Mbits/sec    0    764 KBytes       
[  5]   8.00-9.00   sec  78.6 MBytes   660 Mbits/sec    8    618 KBytes       
[  5]   9.00-10.00  sec  78.7 MBytes   660 Mbits/sec    0    710 KBytes 


was with plain settings: iperf3 -c <serverip>

What i ment with "What decreased the performance between 20.7.5 and 20.7.8?" was refering to klamath post.
Still this question remains unanswered. Maybe franco can shine a little light on this.

didn't notice i was using UDP...

Code: [Select]

iperf3 -c 10.0.3.1
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 44238 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  80.9 MBytes   679 Mbits/sec    0    243 KBytes       
[  5]   1.00-2.00   sec  63.4 MBytes   532 Mbits/sec    0    243 KBytes       
[  5]   2.00-3.00   sec  39.6 MBytes   332 Mbits/sec    0    243 KBytes       
[  5]   3.00-4.00   sec  49.5 MBytes   416 Mbits/sec    1    243 KBytes       
[  5]   4.00-5.00   sec  56.8 MBytes   476 Mbits/sec    0    243 KBytes       
[  5]   5.00-6.00   sec  54.5 MBytes   457 Mbits/sec    0    246 KBytes       
[  5]   6.00-7.00   sec  48.3 MBytes   405 Mbits/sec    1    246 KBytes       
[  5]   7.00-8.00   sec  44.4 MBytes   372 Mbits/sec    0    243 KBytes       
[  5]   8.00-9.00   sec  74.6 MBytes   626 Mbits/sec    0    246 KBytes       
[  5]   9.00-10.00  sec  35.9 MBytes   301 Mbits/sec    0   5.66 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   548 MBytes   460 Mbits/sec    2             sender
[  5]   0.00-10.00  sec   546 MBytes   458 Mbits/sec                  receiver

iperf Done.
its indeed slower then i expected

[klamath]

@seed

There is a -next release of the 20.7.X branch that included a lot of fixes for intel drivers with iflib and netmap.   Here is the thread, https://forum.opnsense.org/index.php?topic=17363.0

I think the -next kernels have been removed for some reason, Im trying to see if someone can restore them as they are the fix to IDS/IPS running on intel cards.

My NIC hardware:
Ethernet Connection X722 for 10GbE SFP+
Ethernet Connection X722 for 10GBASE-T
I350 Gigabit Network Connection

[franco]

Guys, -next is what lead to 21.1. The test kernels have been removed.

So if you want to compare stock 21.1 and 20.7.x is the best option.


Cheers,
Franco