Performance tuning for IPS maximum performance

[mimugmail]

Which hardware?

[dl3it]

Board: https://www.biostar.com.tw/app/en/mb/introduction.php?S_ID=935
NIC: intel i350-T2 https://ark.intel.com/content/www/us/en/ark/products/59062/intel-ethernet-server-adapter-i350-t2.html
8G RAM

[mimugmail]

Sounds reasonable for such a board 

[annoniempjuh]

i was thinking of some performance tuning, did disabled:
- Hardware CRC
- Hardware TCO
- Hardware LRO
- VLAN Hardware Filtering
changed the Pattern matcher to 'hyperscan'
enabled  IPS mode and Promiscuous mode.
i didn't change anything else.

iperf3:

Code: [Select]

iperf3 -c 10.0.3.31 -u -t 60 -i 10 -b 1000M
Connecting to host 10.0.3.31, port 5201
[  5] local 10.0.3.1 port 44924 connected to 10.0.3.31 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec  856118 
[  5]  10.00-20.00  sec  1.16 GBytes  1.00 Gbits/sec  856870 
[  5]  20.00-30.00  sec  1.16 GBytes  1000 Mbits/sec  857061 
[  5]  30.00-40.00  sec  1.16 GBytes  1.00 Gbits/sec  856166 
[  5]  40.00-50.00  sec  1.16 GBytes  1000 Mbits/sec  857113 
[  5]  50.00-60.00  sec  1.16 GBytes  1.00 Gbits/sec  857192 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  6.98 GBytes  1000 Mbits/sec  0.000 ms  0/5140520 (0%)  sender
[  5]   0.00-60.00  sec  3.34 GBytes   479 Mbits/sec  0.046 ms  2680818/5140353 (52%)  receiver

iperf Done.
server statics say: 962Mbit/sec.

well.... i don't need any tuning? 

Suricata is active on WAN and LAN, tested iperf on Lan.
if i change the pattern match to aho-corasick its around the 450Mbit.

rules: 56019
is this command the right one?:

Code: [Select]

root@OPNsense:/usr/local/etc/suricata/rules # cat *.rules | sed 's/^ *#.*//' | sed '/^ *$/d' | wc -lHardware:
AMD Ryzen 3 2200G with Radeon Vega Graphics (4 cores)
8GB RAM
Intel PRO/1000 PT Dual Port Server Adapter (PCI-e 4x) (driver: EM)
OPNsense 20.1.8_1

[mimugmail]

Looks good

[seed]

I just reach 712 MBit Max on my System:

Xeon E-2236
Asus P11c-M/4L
32 GB 2666 mhz ECC RAM
NIC: i340-t4 + 4 x Intel I210AT (onboard)


Powerd shows this output:
root@OPNsense:~ # powerd -v
powerd: unable to determine AC line status
load 156%, current freq 3401 MHz ( 0), wanted freq 6802 MHz
load 100%, current freq 3401 MHz ( 0), wanted freq 6802 MHz
load 100%, current freq 3401 MHz ( 0), wanted freq 6802 MHz
load 114%, current freq 3401 MHz ( 0), wanted freq 6802 MHz
load 157%, current freq 3401 MHz ( 0), wanted freq 6802 MHz


so i assume the Cpu is using its turbo of max 4,80 GHz

I testted with a iperf3 Server in my management vlan and the client in my lan.
OPNsense is fresh installed. Tunables are default. Top Shows one CPU core fully utilised.


root@OPNsense:/usr/local/etc/suricata/rules # cat *.rules | sed 's/^ *#.*//' | sed '/^ *$/d' | wc -l
   47263

With suricata disabled i reach 112 Mbyte (good).

[seed]

I just reach 712 MBit Max on my System:

Xeon E-2236
Asus P11c-M/4L
32 GB 2666 mhz ECC RAM
NIC: i340-t4 + 4 x Intel I210AT (onboard)


Powerd shows this output:
root@OPNsense:~ # powerd -v
powerd: unable to determine AC line status
load 156%, current freq 3401 MHz ( 0), wanted freq 6802 MHz
load 100%, current freq 3401 MHz ( 0), wanted freq 6802 MHz
load 100%, current freq 3401 MHz ( 0), wanted freq 6802 MHz
load 114%, current freq 3401 MHz ( 0), wanted freq 6802 MHz
load 157%, current freq 3401 MHz ( 0), wanted freq 6802 MHz


so i assume the Cpu is using its turbo of max 4,80 GHz

I testted with a iperf3 Server in my management vlan and the client in my lan.
OPNsense is fresh installed. Tunables are default. Top Shows one CPU core fully utilised.


root@OPNsense:/usr/local/etc/suricata/rules # cat *.rules | sed 's/^ *#.*//' | sed '/^ *$/d' | wc -l
   47263

With suricata disabled i reach 112 Mbyte (good).

Sorry. i forgot the sceenshot showing my suricata settings.

[mimugmail]

Try only WAN and disable promisc

[seed]

I testet only with the WAN interface (which is nating) with disables Promisc mode.
This is What i got:

before bios "optimisations"

[  5]   0.00-1.00   sec  70.3 MBytes   589 Mbits/sec   48    636 KBytes       
[  5]   1.00-2.00   sec  94.9 MBytes   796 Mbits/sec    0    744 KBytes       
[  5]   2.00-3.00   sec  97.4 MBytes   817 Mbits/sec    2    625 KBytes       
[  5]   3.00-4.00   sec  98.6 MBytes   827 Mbits/sec    0    737 KBytes       
[  5]   4.00-5.00   sec  98.6 MBytes   828 Mbits/sec    6    617 KBytes       
[  5]   5.00-6.00   sec  97.4 MBytes   817 Mbits/sec    0    728 KBytes       
[  5]   6.00-7.00   sec  94.9 MBytes   796 Mbits/sec    3    602 KBytes       
[  5]   7.00-8.00   sec  96.1 MBytes   806 Mbits/sec    0    714 KBytes       
[  5]   8.00-9.00   sec  97.3 MBytes   817 Mbits/sec    9    588 KBytes       
[  5]   9.00-10.00  sec  91.1 MBytes   764 Mbits/sec    0    697 KBytes       
[  5]  10.00-11.00  sec  96.2 MBytes   807 Mbits/sec    6    564 KBytes       
[  5]  11.00-12.00  sec  97.4 MBytes   817 Mbits/sec    0    683 KBytes       
[  5]  12.00-13.00  sec   100 MBytes   839 Mbits/sec    1    554 KBytes       
[  5]  13.00-14.00  sec  97.5 MBytes   818 Mbits/sec    0    679 KBytes       
[  5]  14.00-15.00  sec  96.2 MBytes   807 Mbits/sec    9    546 KBytes       
[  5]  15.00-16.00  sec  96.2 MBytes   807 Mbits/sec    0    667 KBytes       
[  5]  16.00-17.00  sec  96.2 MBytes   807 Mbits/sec    0    772 KBytes       
[  5]  17.00-18.00  sec  96.2 MBytes   807 Mbits/sec    5    655 KBytes       
^C[  5]  18.00-18.60  sec  58.7 MBytes   818 Mbits/sec    0    721 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-18.60  sec  1.73 GBytes   799 Mbits/sec   89             sender
[  5]   0.00-18.60  sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated

with "optimized bios"

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  98.4 MBytes   826 Mbits/sec   54    694 KBytes       
[  5]   1.00-2.00   sec  95.0 MBytes   797 Mbits/sec    5    563 KBytes       
[  5]   2.00-3.00   sec  96.2 MBytes   807 Mbits/sec    0    683 KBytes       
[  5]   3.00-4.00   sec  97.5 MBytes   818 Mbits/sec    5    550 KBytes       
[  5]   4.00-5.00   sec  97.5 MBytes   818 Mbits/sec    0    672 KBytes       
[  5]   5.00-6.00   sec  96.2 MBytes   807 Mbits/sec    3    542 KBytes       
[  5]   6.00-7.00   sec  97.5 MBytes   818 Mbits/sec    0    665 KBytes       
[  5]   7.00-8.00   sec  96.2 MBytes   807 Mbits/sec    0    769 KBytes       
[  5]   8.00-9.00   sec  98.7 MBytes   828 Mbits/sec    7    653 KBytes       
[  5]   9.00-10.00  sec  97.5 MBytes   818 Mbits/sec    0    759 KBytes       
[  5]  10.00-11.00  sec  96.2 MBytes   807 Mbits/sec    8    639 KBytes       
[  5]  11.00-12.00  sec  97.5 MBytes   818 Mbits/sec    0    748 KBytes       
[  5]  12.00-13.00  sec  95.0 MBytes   797 Mbits/sec    1    629 KBytes       
[  5]  13.00-14.00  sec  95.0 MBytes   797 Mbits/sec    0    734 KBytes       
[  5]  14.00-15.00  sec  96.2 MBytes   807 Mbits/sec    2    612 KBytes       
[  5]  15.00-16.00  sec  96.2 MBytes   807 Mbits/sec    0    725 KBytes       
^C[  5]  16.00-16.06  sec  5.00 MBytes   686 Mbits/sec    0    730 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-16.06  sec  1.52 GBytes   811 Mbits/sec   85             sender
[  5]   0.00-16.06  sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated

Very close. but still not what i expected to see.
Why is the result different from the "lan" Interface? What stops the system from performing better?
I mean. The Xeon E-2236 is really good.

@mimugmail:
I Read your blogpost testing with the Xeon E3-1240 v6. You got better results. The CPU is slightly older. So what black magic is happening here?

[mimugmail]

I tested with 10g interfaces

[webdb]

Hi
I have a 1Gig connection and OPNsense works perfectly fine with IPS enabled (approx 3k rules). But when I download big files from Usenet (e.g. 5-10 gig) the performance goes from 900Mbps down to a few Kbps and up again. This isn't really an issue for me as I have no time constraints for such downloads. However teh firewall/DNS seems to freez as my 60 devices can't connect to the internet after such a download and I always have to restart Opnsense.
When I turn on my old Kerio Control and do the same scenario I see drops to approx 50mbps and the firewall doesn't freeze.

Has anyone similar issues and found a solution? I love Opnsense and don't want to go back to Kerio again or switch to another product such as Zyxel ATP 200

Thanks
Daniel

Hardware: Initel Core i7, 16GB Memory, SSD, only Dyndns and IPS running on Opnsense

[alexroz]

I have mini-pc https://www.aliexpress.com/item/4000859041000.html  based on Celeron 3865U with 4GB RAM.
And I am experiencing sharp download bandwidth drop when I turn IPS on. I get download throughput just below 1GBps when Suricata is OFF and between 300 to 400 when Suricata is ON.
Any performance tuning suggestions?

[mimugmail]

Only enable Rules you really need. No phpnuke stuff and so on

[alexroz]

Can someone explain how promiscuous mode can improve Suricata's performance?

[spetrillo]

Only enable Rules you really need. No phpnuke stuff and so on

Is there a guide on what we should enable?