Performance tuning for IPS maximum performance

[mimugmail]

Can someone explain how promiscuous mode can improve Suricata's performance?

It doesn't, you only need this when you listen to igb0 while running several vlans on it so you don't have to select every single interface

[mimugmail]

Only enable Rules you really need. No phpnuke stuff and so on

Is there a guide on what we should enable?

At first, read those descriptions:
https://tools.emergingthreats.net/docs/ETPro%20Rule%20Categories.pdf

After this I'm sure you know which you don't need.

[alexroz]

Only enable Rules you really need. No phpnuke stuff and so on

Is there a guide on what we should enable?

$1000000 question.....

[harryincs]

what would be the correct setting for disabling flow control with a Realtek driver?
hw.igb.0.fc=0 is for Intel as I understand.

my Nic cards show up as em0 and re0

thanks,

[mimugmail]

sysctl -a | grep fc and check if its available. I'd guess a RE doesnt even support it

[harryincs]

This is what I get and these are probably the statements of interest:

hw.ixl.enable_tx_fc_filter: 1
dev.em.0.fc_low_water: 20552
dev.em.0.fc_high_water: 23584
dev.em.0.fc: 3

---------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------

root@IncsFW1:~ # sysctl -a | grep fc
device   ocs_fc
2 LABEL gptid/ffcdb6e8-434e-11eb-b1ee-64006a9003d0 209715200 512 i 0 o 0
z0xfffff80003a3ca00 [shape=box,label="DEV\ngptid/ffcdb6e8-434e-11eb-b1ee-64006a9003d0\nr#4"];
z0xfffff80003a3c800 [shape=hexagon,label="gptid/ffcdb6e8-434e-11eb-b1ee-64006a9003d0\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
      <name>gptid/ffcdb6e8-434e-11eb-b1ee-64006a9003d0</name>
       <rawuuid>ffcdb6e8-434e-11eb-b1ee-64006a9003d0</rawuuid>
       <efimedia>HD(1,GPT,ffcdb6e8-434e-11eb-b1ee-64006a9003d0,0x28,0x64000)</efimedia>
     <name>gptid/ffcdb6e8-434e-11eb-b1ee-64006a9003d0</name>
vfs.reassignbufcalls: 10443885
vfs.getnewbufcalls: 4196070
net.inet.ip.rfc6864: 1
net.inet.tcp.rfc1323: 1
net.inet.tcp.rfc3465: 1
net.inet.tcp.rfc3390: 1
net.inet.tcp.rfc3042: 1
net.inet.tcp.rfc6675_pipe: 0
net.link.generic.system.ifcount: 7
net.inet6.ip6.rfc6204w3: 1
net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0
hw.ixl.enable_tx_fc_filter: 1
     ConventionalMemory 000000100000          0x0 0003befc UC WC WT WB
             LoaderData 00003bffc000          0x0 00004004 UC WC WT WB
dev.em.0.fc_low_water: 20552
dev.em.0.fc_high_water: 23584
dev.em.0.fc: 3

[annoniempjuh]

a view days ago i did upgrade OPNsense and my server to 10Gbit NICs

hardware:
Intel Ethernet Converged Network Adapter X540-T2  (OPNsense)
Mellanox ConnectX-3 CX311A (unRAID server)
MikroTik Cloud Smart Switch 326-24G-2S+RM (switch)


Iperf3 results:

suricata OFF = cpu usage 40% / 51%

Code Select Expand

iperf3 -c 10.0.3.1 -t 60 -i 10
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 35558 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  3.03 GBytes  2.60 Gbits/sec    0    252 KBytes       
[  5]  10.00-20.00  sec  2.99 GBytes  2.57 Gbits/sec    0    246 KBytes       
[  5]  20.00-30.00  sec  2.98 GBytes  2.56 Gbits/sec    0    243 KBytes       
[  5]  30.00-40.00  sec  2.96 GBytes  2.54 Gbits/sec    0    209 KBytes       
[  5]  40.00-50.00  sec  2.93 GBytes  2.52 Gbits/sec    0    277 KBytes       
[  5]  50.00-60.00  sec  2.97 GBytes  2.55 Gbits/sec    0    260 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  17.9 GBytes  2.56 Gbits/sec    0             sender
[  5]   0.00-60.00  sec  17.9 GBytes  2.56 Gbits/sec                  receiver

iperf Done.

iperf3 -c 10.0.3.1 -t 60 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[  5] local 10.0.3.2 port 36642 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.82 GBytes  3.28 Gbits/sec                 
[  5]  10.00-20.00  sec  3.89 GBytes  3.35 Gbits/sec                 
[  5]  20.00-30.00  sec  3.82 GBytes  3.28 Gbits/sec                 
[  5]  30.00-40.00  sec  3.75 GBytes  3.22 Gbits/sec                 
[  5]  40.00-50.00  sec  3.60 GBytes  3.09 Gbits/sec                 
[  5]  50.00-60.00  sec  3.76 GBytes  3.23 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  22.6 GBytes  3.24 Gbits/sec  8384             sender
[  5]   0.00-60.00  sec  22.6 GBytes  3.24 Gbits/sec                  receiver

iperf Done.


suricata ON = cpu usage 59% / 76%

Code Select Expand

iperf3 -c 10.0.3.1 -t 60 -i 10
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 43546 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec   753 MBytes   632 Mbits/sec    2   5.66 KBytes       
[  5]  10.00-20.00  sec   748 MBytes   627 Mbits/sec    8    219 KBytes       
[  5]  20.00-30.00  sec   745 MBytes   625 Mbits/sec    5    209 KBytes       
[  5]  30.00-40.00  sec   774 MBytes   649 Mbits/sec   12    188 KBytes       
[  5]  40.00-50.00  sec   744 MBytes   624 Mbits/sec    5    218 KBytes       
[  5]  50.00-60.00  sec   795 MBytes   667 Mbits/sec    7    215 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  4.45 GBytes   637 Mbits/sec   39             sender
[  5]   0.00-60.00  sec  4.45 GBytes   637 Mbits/sec                  receiver

iperf Done.

iperf3 -c 10.0.3.1 -t 60 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[  5] local 10.0.3.2 port 38420 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.40 GBytes  1.21 Gbits/sec                 
[  5]  10.00-20.00  sec  1.37 GBytes  1.17 Gbits/sec                 
[  5]  20.00-30.00  sec  1.40 GBytes  1.20 Gbits/sec                 
[  5]  30.00-40.00  sec  1.39 GBytes  1.19 Gbits/sec                 
[  5]  40.00-50.00  sec  1.40 GBytes  1.20 Gbits/sec                 
[  5]  50.00-60.00  sec  1.41 GBytes  1.21 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  8.37 GBytes  1.20 Gbits/sec   18             sender
[  5]   0.00-60.00  sec  8.37 GBytes  1.20 Gbits/sec                  receiver

iperf Done.

UDP:

Code Select Expand

iperf3 -c 10.0.3.1 -u -t 60 -i 10 -b 10000M
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.2 port 59369 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  2.88 GBytes  2.48 Gbits/sec  2138663 
[  5]  10.00-20.00  sec  2.89 GBytes  2.48 Gbits/sec  2143473 
[  5]  20.00-30.00  sec  2.85 GBytes  2.45 Gbits/sec  2110755 
[  5]  30.00-40.00  sec  2.81 GBytes  2.41 Gbits/sec  2081894 
[  5]  40.00-50.00  sec  2.87 GBytes  2.46 Gbits/sec  2126508 
[  5]  50.00-60.00  sec  2.92 GBytes  2.51 Gbits/sec  2167670 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  17.2 GBytes  2.47 Gbits/sec  0.000 ms  0/12768963 (0%)  sender
[  5]   0.00-60.01  sec  12.5 GBytes  1.79 Gbits/sec  0.001 ms  3471092/12768963 (27%)  receiver

iperf Done.

i know that there are some posts going on that on OPNsense 21.1 there is slowdown...

its looks like i am not the only one who doesn't get 10Gb speeds...

i tried a view tuneables but it didn't do anything:

Code Select Expand

kern.ipc.maxsockbuf:  16777216
net.inet.ip.intr_queue_maxlen:  2048
net.inet.tcp.recvspace:  4194304
net.inet.tcp.sendspace:  2097152
net.inet.tcp.recvbuf_max:  16777216
net.inet.tcp.recvbuf_inc:  524288
net.inet.tcp.sendbuf_max:  16777216
net.inet.tcp.sendbuf_inc:  32768
net.route.netisr_maxqlen:  2048
net.link.ifqmaxlen:  2048

need to do more investigation why it won't do 10Gb, maybe its the switch who has wrong settings (it using the defaults settings) or maybe it's the unRAID server...

[mimugmail]

Does this also happen with 20.7.8?

[annoniempjuh]

Does this also happen with 20.7.8?

i didn't test it on 20.7.8
i tried to downgrade to 20.7.8 but it didn't succeed:

Code Select Expand

opnsense-update -r 20.7.8[/s]
Fetching base-20.7.8-amd64.txz: .. failed, no signature found


edit:
did a clean install of 20.7, upgraded it to 20.7.8_4
same results...
not sure what the problem is, guess i have to investigate if its not OPNsense but unRAID or the switch.

[annoniempjuh]

view months later and did an another rounds of trail and error. this time i got some better results:
OPNsense 21.1.5
currently i use the following custom configs:

Code Select Expand

net.inet.tcp.tso=0
net.inet.udp.checksum=0
net.isr.maxthreads=-1
net.isr.dispatch=deferred
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
kern.ipc.maxsockbuf=16777216
kern.ipc.nmbclusters=1000000
kern.ipc.nmbjumbop=524288
hw.bce.tso_enable=0
hw.vtnet.lro_disable=1
hw.ix.flow_control=0
hw.ix.rx_process_limit=-1
hw.ix.tx_process_limit=-1
hw.intr_storm_threshold=10000

net.inet6.ip6.redirect=0
net.inet.ip.intr_queue_maxlen=3000
net.inet.tcp.mssdflt=1460
net.inet.tcp.minmss=1300
net.inet.tcp.syncookies=0

in /boot/loader.conf.local:
#cc_htcp_load="YES"
if_ix_updated_load="YES"
hw.ix.tx_process_limit="-1"
hw.ix.rx_process_limit="-1"
hw.ix.enable_aim="1"
hw.ix.max_interrupt_rate="64000"
hw.ix.rxd="4096"
hw.ix.txd="4096"
net.link.ifqmaxlen="8192"
hw.ix.num_queues="8"

Iperf3 results:
suricata on:

Code Select Expand

iperf3 -c 10.0.3.1 -t 20 -i 10
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.47 port 38402 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  1.44 GBytes  1.24 Gbits/sec    0    580 KBytes       
[  5]  10.00-20.00  sec  1.49 GBytes  1.28 Gbits/sec    0    580 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.94 GBytes  1.26 Gbits/sec    0             sender
[  5]   0.00-20.00  sec  2.93 GBytes  1.26 Gbits/sec                  receiver

iperf Done.
iperf3 -c 10.0.3.1 -t 20 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[  5] local 10.0.3.47 port 38744 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.58 GBytes  1.35 Gbits/sec                 
[  5]  10.00-20.00  sec  1.61 GBytes  1.38 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  3.19 GBytes  1.37 Gbits/sec   20             sender
[  5]   0.00-20.00  sec  3.19 GBytes  1.37 Gbits/sec                  receiver

iperf Done.
iperf3 -c 10.0.3.1 -t 20 -i 10 -u -b 10000M
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.47 port 44492 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  4.91 GBytes  4.22 Gbits/sec  3641085 
[  5]  10.00-20.00  sec  4.91 GBytes  4.21 Gbits/sec  3637990 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-20.00  sec  9.82 GBytes  4.22 Gbits/sec  0.000 ms  0/7279075 (0%)  sender
[  5]   0.00-20.01  sec  5.74 GBytes  2.46 Gbits/sec  0.002 ms  3022771/7279075 (42%)  receiver

iperf Done.
iperf3 -c 10.0.3.1 -t 20 -i 10 -u -b 10000M -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[  5] local 10.0.3.47 port 41435 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  2.56 GBytes  2.20 Gbits/sec  0.007 ms  146252/2042299 (7.2%) 
[  5]  10.00-20.00  sec  2.53 GBytes  2.18 Gbits/sec  0.008 ms  126068/2004900 (6.3%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-20.00  sec  5.09 GBytes  2.19 Gbits/sec  0.000 ms  0/4048582 (0%)  sender
[  5]   0.00-20.00  sec  5.09 GBytes  2.19 Gbits/sec  0.008 ms  272320/4047199 (6.7%)  receiver

iperf Done.

suricata off:

Code Select Expand


iperf3 -c 10.0.3.1 -t 10 -i 10
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.47 port 43458 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  3.43 GBytes  2.95 Gbits/sec    0    577 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.43 GBytes  2.95 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  3.43 GBytes  2.95 Gbits/sec                  receiver

iperf Done.

iperf3 -c 10.0.3.1 -t 10 -i 10 -u -b 10000M
Connecting to host 10.0.3.1, port 5201
[  5] local 10.0.3.47 port 47876 connected to 10.0.3.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  4.82 GBytes  4.14 Gbits/sec  3571818 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  4.82 GBytes  4.14 Gbits/sec  0.000 ms  0/3571818 (0%)  sender
[  5]   0.00-10.00  sec  4.70 GBytes  4.04 Gbits/sec  0.001 ms  84942/3571817 (2.4%)  receiver

iperf Done.


those test are with a direct connection wit OPNsense and my Desktop (its using a Intel X550 t2)
i did also test the connection between my unRAID server and desktop with a direct connection between those two, 9.90Gbit...

but i also see that the cpu usage of OPNsense spike up to 90%, i guess that the AMD Ryzen 3 2200G is struggling with those speeds...

and i know, using OPNsense as iperf server isn't recommeded, i did those iperf test also with my unraid server as server and desktop connected on its switch..

[annoniempjuh]

i just tried some new iperf test, this time i did remove the loader.conf.local and reset the tunables to default and only apply those custom setting:

Code Select Expand


kern.ipc.nmbclusters=1000000
net.inet.tcp.tso=0
net.inet.ip.redirect=0
net.inet6.ip6.redirect=0
net.isr.bindthreads=1
net.isr.maxthreads=-1
hw.intr_storm_threshold=10000
hw.ix.flow_control=0
net.isr.numthreads=-1
net.route.netisr_maxqlen=2048
hw.ibrs_disable=1      <<<<<-- the Ryzen 5 3600 isn't vulnerable
vm.pmap.pti=0      <<<<<-- the Ryzen 5 3600 isn't vulnerable

in > Interfaces > Settings > hardware offloading, everything disabled.


did some iperf test to a iperf server on a vlan on unRAID: (suricata off)

Code Select Expand


iperf3 -c 10.0.15.6 -P 8
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.32 GBytes  2.85 Gbits/sec   32             sender
[  5]   0.00-10.00  sec  3.31 GBytes  2.84 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.25 GBytes  1.07 Gbits/sec   20             sender
[  7]   0.00-10.00  sec  1.24 GBytes  1.06 Gbits/sec                  receiver
[  9]   0.00-10.00  sec   429 MBytes   359 Mbits/sec   14             sender
[  9]   0.00-10.00  sec   425 MBytes   356 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   881 MBytes   739 Mbits/sec   18             sender
[ 11]   0.00-10.00  sec   872 MBytes   731 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec  2.12 GBytes  1.82 Gbits/sec   26             sender
[ 13]   0.00-10.00  sec  2.11 GBytes  1.81 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  1.24 GBytes  1.06 Gbits/sec   22             sender
[ 15]   0.00-10.00  sec  1.24 GBytes  1.06 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec   937 MBytes   786 Mbits/sec   16             sender
[ 17]   0.00-10.00  sec   930 MBytes   780 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec   795 MBytes   667 Mbits/sec   21             sender
[ 19]   0.00-10.00  sec   791 MBytes   663 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  10.9 GBytes  9.36 Gbits/sec  169             sender
[SUM]   0.00-10.00  sec  10.8 GBytes  9.31 Gbits/sec                  receiver

iperf Done.


iperf test on OPNsense: (suricata off)

Code Select Expand


iperf3 -c 10.0.3.1 -P 8
[SUM]   0.00-10.00  sec  8.81 GBytes  7.57 Gbits/sec    0             sender
[SUM]   0.00-10.01  sec  8.79 GBytes  7.54 Gbits/sec                  receiver

iperf Done.


iperf to unraid (vlan) with Sucicata on:

Code Select Expand


iperf3 -c 10.0.15.6 -P 8
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  72.2 MBytes  60.6 Mbits/sec  616             sender
[  5]   0.00-10.01  sec  69.7 MBytes  58.4 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   127 MBytes   106 Mbits/sec  512             sender
[  7]   0.00-10.01  sec   124 MBytes   104 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   398 MBytes   334 Mbits/sec  315             sender
[  9]   0.00-10.01  sec   396 MBytes   332 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   458 MBytes   384 Mbits/sec  883             sender
[ 11]   0.00-10.01  sec   456 MBytes   382 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec   227 MBytes   190 Mbits/sec  1091             sender
[ 13]   0.00-10.01  sec   224 MBytes   187 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec  48.4 MBytes  40.6 Mbits/sec  220             sender
[ 15]   0.00-10.01  sec  46.6 MBytes  39.1 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec   249 MBytes   209 Mbits/sec  712             sender
[ 17]   0.00-10.01  sec   247 MBytes   207 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec   168 MBytes   141 Mbits/sec  962             sender
[ 19]   0.00-10.01  sec   166 MBytes   139 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.71 GBytes  1.47 Gbits/sec  5311             sender
[SUM]   0.00-10.01  sec  1.69 GBytes  1.45 Gbits/sec                  receiver

iperf Done.


iperf single stream to unRAID (vlan) with Suricata on:

Code Select Expand


iperf3 -c 10.0.15.6
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.61 GBytes  1.38 Gbits/sec   96             sender
[  5]   0.00-10.01  sec  1.61 GBytes  1.38 Gbits/sec                  receiver

iperf Done.


iperf test to unRAID (vlan) using UDP with Suricata on:

Code Select Expand


iperf3 -c 10.0.15.6 -u -b 10000M
Connecting to host 10.0.15.6, port 5201
[  5] local 10.0.3.47 port 42165 connected to 10.0.15.6 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   500 MBytes  4.19 Gbits/sec  362060 
[  5]   1.00-2.00   sec   499 MBytes  4.18 Gbits/sec  361268 
[  5]   2.00-3.00   sec   502 MBytes  4.21 Gbits/sec  363826 
[  5]   3.00-4.00   sec   502 MBytes  4.21 Gbits/sec  363828 
[  5]   4.00-5.00   sec   502 MBytes  4.22 Gbits/sec  363884 
[  5]   5.00-6.00   sec   502 MBytes  4.21 Gbits/sec  363707 
[  5]   6.00-7.00   sec   504 MBytes  4.23 Gbits/sec  365292 
[  5]   7.00-8.00   sec   507 MBytes  4.26 Gbits/sec  367461 
[  5]   8.00-9.00   sec   505 MBytes  4.24 Gbits/sec  365711 
[  5]   9.00-10.00  sec   503 MBytes  4.22 Gbits/sec  364081 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  4.91 GBytes  4.22 Gbits/sec  0.000 ms  0/3641118 (0%)  sender
[  5]   0.00-10.01  sec  2.19 GBytes  1.88 Gbits/sec  0.002 ms  2016349/3641118 (55%)  receiver

iperf Done.


iperf test to unRAID (vlan) using UDP with Suricata on: (reverse)

Code Select Expand


iperf3 -c 10.0.15.6 -u -b 10000M -R
Connecting to host 10.0.15.6, port 5201
Reverse mode, remote host 10.0.15.6 is sending
[  5] local 10.0.3.47 port 45202 connected to 10.0.15.6 port 5201
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec   270 MBytes  2.26 Gbits/sec  0.007 ms  12487/207721 (6%) 
[  5]   1.00-2.00   sec   256 MBytes  2.15 Gbits/sec  0.008 ms  26181/211449 (12%) 
[  5]   2.00-3.00   sec   242 MBytes  2.03 Gbits/sec  0.004 ms  22832/198170 (12%) 
[  5]   3.00-4.00   sec   278 MBytes  2.33 Gbits/sec  0.021 ms  6834/208187 (3.3%) 
[  5]   4.00-5.00   sec   278 MBytes  2.33 Gbits/sec  0.006 ms  6573/207709 (3.2%) 
[  5]   5.00-6.00   sec   278 MBytes  2.33 Gbits/sec  0.005 ms  10012/211426 (4.7%) 
[  5]   6.00-7.00   sec   279 MBytes  2.34 Gbits/sec  0.015 ms  3345/205200 (1.6%) 
[  5]   7.00-8.00   sec   266 MBytes  2.23 Gbits/sec  0.006 ms  1160/193969 (0.6%) 
[  5]   8.00-9.00   sec   257 MBytes  2.15 Gbits/sec  0.011 ms  0/185941 (0%) 
[  5]   9.00-10.00  sec   251 MBytes  2.11 Gbits/sec  0.008 ms  0/181951 (0%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  2.71 GBytes  2.33 Gbits/sec  0.000 ms  0/2011786 (0%)  sender
[  5]   0.00-10.00  sec  2.59 GBytes  2.23 Gbits/sec  0.008 ms  89424/2011723 (4.4%)  receiver

iperf Done.


Iperf3 test IPv6 to OPNsense (Suricata On): UDP

Code Select Expand


iperf3 -c [IPv6 address of OPNsense] -u -b 10000M
Connecting to host [IPv6 address of OPNsense], port 5201
[  5] local [desktop IPV6 address] port 35200 connected to [IPv6 address of OPNsense] port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   499 MBytes  4.19 Gbits/sec  368714 
[  5]   1.00-2.00   sec   490 MBytes  4.11 Gbits/sec  361542 
[  5]   2.00-3.00   sec   493 MBytes  4.13 Gbits/sec  363771 
[  5]   3.00-4.00   sec   493 MBytes  4.14 Gbits/sec  364085 
[  5]   4.00-5.00   sec   493 MBytes  4.14 Gbits/sec  364046 
[  5]   5.00-6.00   sec   492 MBytes  4.13 Gbits/sec  363624 
[  5]   6.00-7.00   sec   493 MBytes  4.13 Gbits/sec  363868 
[  5]   7.00-8.00   sec   493 MBytes  4.13 Gbits/sec  363977 
[  5]   8.00-9.00   sec   492 MBytes  4.13 Gbits/sec  363473 
[  5]   9.00-10.00  sec   493 MBytes  4.14 Gbits/sec  364084 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  4.82 GBytes  4.14 Gbits/sec  0.000 ms  0/3641184 (0%)  sender
[  5]   0.00-10.00  sec  3.02 GBytes  2.60 Gbits/sec  0.002 ms  1354919/3640263 (37%)  receiver
iperf Done.


Iperf test IPv6 to OPNsense (Suricata on): UDP (reverse)

Code Select Expand


iperf3 -c [IPv6 address of OPNsense] -u -b 10000M -R
Connecting to host [IPv6 address of OPNsense], port 5201
Reverse mode, remote host [IPv6 address of OPNsense] is sending
[  5] local [desktop IPV6 address] port 47373 connected to [IPv6 address of OPNsense] port 5201
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec   491 MBytes  4.12 Gbits/sec  0.003 ms  23388/385996 (6.1%) 
[  5]   1.00-2.00   sec   533 MBytes  4.47 Gbits/sec  0.004 ms  31061/424933 (7.3%) 
[  5]   2.00-3.00   sec   535 MBytes  4.49 Gbits/sec  0.022 ms  25175/419985 (6%) 
[  5]   3.00-4.00   sec   543 MBytes  4.55 Gbits/sec  0.003 ms  20191/420819 (4.8%) 
[  5]   4.00-5.00   sec   532 MBytes  4.46 Gbits/sec  0.001 ms  28162/421172 (6.7%) 
[  5]   5.00-6.00   sec   533 MBytes  4.47 Gbits/sec  0.002 ms  26736/420413 (6.4%) 
[  5]   6.00-7.00   sec   532 MBytes  4.47 Gbits/sec  0.003 ms  29979/423165 (7.1%) 
[  5]   7.00-8.00   sec   532 MBytes  4.46 Gbits/sec  0.002 ms  34746/427258 (8.1%) 
[  5]   8.00-9.00   sec   532 MBytes  4.46 Gbits/sec  0.004 ms  25774/418755 (6.2%) 
[  5]   9.00-10.00  sec   533 MBytes  4.47 Gbits/sec  0.002 ms  22481/415869 (5.4%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  5.38 GBytes  4.62 Gbits/sec  0.000 ms  0/4179500 (0%)  sender
[  5]   0.00-10.00  sec  5.17 GBytes  4.44 Gbits/sec  0.002 ms  267693/4178365 (6.4%)  receiver
iperf Done.


so conclution: the default setting of OPNsense are fine, no need to customize the settings

[NoncarbonatedClack]

# File starts below this line, use Copy/Paste #####################
# Check for interface specific settings and add accordingly.
# These ae tunables to improve network performance on Intel igb driver NICs

# Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
# This tunable must be set according to your configuration. VERY IMPORTANT!
# Set FC to 0 (<x>) on all interfaces
hw.igb.<x>.fc=0 #Also put this in System Tunables hw.igb.<x>.fc: value=0

Just wanted to throw my .02 at this in case anyone else sees it... no matter what I did above, I could not get FC disabled. The solution was to add "dev.igb.0.fc" to tuneables, with a value of 0. That resolved it for me.

I'm trying to verify now that "hw.igb.enable_aim=1" works, but I'm not really sure how.
I'm also wondering how to check/enable handling VLANs on the hardware level, not sure how to go about that.


This is on OPNsense OPNsense 21.7.5-amd64, FreeBSD 12.1-RELEASE-p21-HBSD

The system is an HP ProLiant ML310e Gen8
Intel Xeon E3-1220 V2 3.10 GHz 4c/4t
16 GBDDR3 ECC
120 GB SSD
Intel i350 T4 V1

Symmetrical 1Gbps internet connection.

I'm not running IDS/IPS yet but preparing to.

[dcol]

I think changing to dev was introduced in a newer version of FreeBSD. I will change my original post

[LOTRouter]

I have been trying to tune IPS for the Intel i225 2.5G NIC on my N5105 based router using 22.7_4.  When I disable IPS and just leave IDS on, I can consistently get 1.4G down (after adding the below tuning). If I enable IPS, even with just the opnsense.test.rules enabled, I can only get between 800M to 1.2G down and with significant jitter introduced.

The most relevant tuning I made was disabling flow control.  Before doing so, I could never get above 1.2G down even with IDS disabled.  I added these tunables:

SYSTEM | SETTINGS | TUNABLES
Interface igc0 Flow Control | dev.igc.0.fc | 0
Interface igc1 Flow Control | dev.igc.1.fc | 0
Interface igc2 Flow Control | dev.igc.2.fc | 0
Interface igc3 Flow Control | dev.igc.3.fc | 0


Bottom line, if you want the full 1.4G Comcast provisions and you want IPS, then an N5105 is probably a bit to underpowered for it.  If you just want IDS, then it can handle that just fine at full speed.  I have both an N6005 and an i5 1135G7 on its way I will also try, but I don't expect them until Late August.

[lilsense]

Are there any hi performance tuning sets specifically for Decisio DEC850?

I am using RSS as well.
https://forum.opnsense.org/index.php?topic=24409.msg116941