46
Intrusion Detection and Prevention / Re: Performance tuning for IPS maximum performance
« on: February 07, 2021, 10:24:53 am »
a view days ago i did upgrade OPNsense and my server to 10Gbit NICs
hardware:
Intel Ethernet Converged Network Adapter X540-T2 (OPNsense)
Mellanox ConnectX-3 CX311A (unRAID server)
MikroTik Cloud Smart Switch 326-24G-2S+RM (switch)
Iperf3 results:
suricata OFF = cpu usage 40% / 51%
suricata ON = cpu usage 59% / 76%
UDP:
i know that there are some posts going on that on OPNsense 21.1 there is slowdown...
its looks like i am not the only one who doesn't get 10Gb speeds...
i tried a view tuneables but it didn't do anything:
need to do more investigation why it won't do 10Gb, maybe its the switch who has wrong settings (it using the defaults settings) or maybe it's the unRAID server...
hardware:
Intel Ethernet Converged Network Adapter X540-T2 (OPNsense)
Mellanox ConnectX-3 CX311A (unRAID server)
MikroTik Cloud Smart Switch 326-24G-2S+RM (switch)
Iperf3 results:
suricata OFF = cpu usage 40% / 51%
Code: [Select]
iperf3 -c 10.0.3.1 -t 60 -i 10
Connecting to host 10.0.3.1, port 5201
[ 5] local 10.0.3.2 port 35558 connected to 10.0.3.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-10.00 sec 3.03 GBytes 2.60 Gbits/sec 0 252 KBytes
[ 5] 10.00-20.00 sec 2.99 GBytes 2.57 Gbits/sec 0 246 KBytes
[ 5] 20.00-30.00 sec 2.98 GBytes 2.56 Gbits/sec 0 243 KBytes
[ 5] 30.00-40.00 sec 2.96 GBytes 2.54 Gbits/sec 0 209 KBytes
[ 5] 40.00-50.00 sec 2.93 GBytes 2.52 Gbits/sec 0 277 KBytes
[ 5] 50.00-60.00 sec 2.97 GBytes 2.55 Gbits/sec 0 260 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 17.9 GBytes 2.56 Gbits/sec 0 sender
[ 5] 0.00-60.00 sec 17.9 GBytes 2.56 Gbits/sec receiver
iperf Done.
iperf3 -c 10.0.3.1 -t 60 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[ 5] local 10.0.3.2 port 36642 connected to 10.0.3.1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 3.82 GBytes 3.28 Gbits/sec
[ 5] 10.00-20.00 sec 3.89 GBytes 3.35 Gbits/sec
[ 5] 20.00-30.00 sec 3.82 GBytes 3.28 Gbits/sec
[ 5] 30.00-40.00 sec 3.75 GBytes 3.22 Gbits/sec
[ 5] 40.00-50.00 sec 3.60 GBytes 3.09 Gbits/sec
[ 5] 50.00-60.00 sec 3.76 GBytes 3.23 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 22.6 GBytes 3.24 Gbits/sec 8384 sender
[ 5] 0.00-60.00 sec 22.6 GBytes 3.24 Gbits/sec receiver
iperf Done.
suricata ON = cpu usage 59% / 76%
Code: [Select]
iperf3 -c 10.0.3.1 -t 60 -i 10
Connecting to host 10.0.3.1, port 5201
[ 5] local 10.0.3.2 port 43546 connected to 10.0.3.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-10.00 sec 753 MBytes 632 Mbits/sec 2 5.66 KBytes
[ 5] 10.00-20.00 sec 748 MBytes 627 Mbits/sec 8 219 KBytes
[ 5] 20.00-30.00 sec 745 MBytes 625 Mbits/sec 5 209 KBytes
[ 5] 30.00-40.00 sec 774 MBytes 649 Mbits/sec 12 188 KBytes
[ 5] 40.00-50.00 sec 744 MBytes 624 Mbits/sec 5 218 KBytes
[ 5] 50.00-60.00 sec 795 MBytes 667 Mbits/sec 7 215 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 4.45 GBytes 637 Mbits/sec 39 sender
[ 5] 0.00-60.00 sec 4.45 GBytes 637 Mbits/sec receiver
iperf Done.
iperf3 -c 10.0.3.1 -t 60 -i 10 -R
Connecting to host 10.0.3.1, port 5201
Reverse mode, remote host 10.0.3.1 is sending
[ 5] local 10.0.3.2 port 38420 connected to 10.0.3.1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 1.40 GBytes 1.21 Gbits/sec
[ 5] 10.00-20.00 sec 1.37 GBytes 1.17 Gbits/sec
[ 5] 20.00-30.00 sec 1.40 GBytes 1.20 Gbits/sec
[ 5] 30.00-40.00 sec 1.39 GBytes 1.19 Gbits/sec
[ 5] 40.00-50.00 sec 1.40 GBytes 1.20 Gbits/sec
[ 5] 50.00-60.00 sec 1.41 GBytes 1.21 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 8.37 GBytes 1.20 Gbits/sec 18 sender
[ 5] 0.00-60.00 sec 8.37 GBytes 1.20 Gbits/sec receiver
iperf Done.
UDP:
Code: [Select]
iperf3 -c 10.0.3.1 -u -t 60 -i 10 -b 10000M
Connecting to host 10.0.3.1, port 5201
[ 5] local 10.0.3.2 port 59369 connected to 10.0.3.1 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-10.00 sec 2.88 GBytes 2.48 Gbits/sec 2138663
[ 5] 10.00-20.00 sec 2.89 GBytes 2.48 Gbits/sec 2143473
[ 5] 20.00-30.00 sec 2.85 GBytes 2.45 Gbits/sec 2110755
[ 5] 30.00-40.00 sec 2.81 GBytes 2.41 Gbits/sec 2081894
[ 5] 40.00-50.00 sec 2.87 GBytes 2.46 Gbits/sec 2126508
[ 5] 50.00-60.00 sec 2.92 GBytes 2.51 Gbits/sec 2167670
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-60.00 sec 17.2 GBytes 2.47 Gbits/sec 0.000 ms 0/12768963 (0%) sender
[ 5] 0.00-60.01 sec 12.5 GBytes 1.79 Gbits/sec 0.001 ms 3471092/12768963 (27%) receiver
iperf Done.
i know that there are some posts going on that on OPNsense 21.1 there is slowdown...
its looks like i am not the only one who doesn't get 10Gb speeds...
i tried a view tuneables but it didn't do anything:
Code: [Select]
kern.ipc.maxsockbuf: 16777216
net.inet.ip.intr_queue_maxlen: 2048
net.inet.tcp.recvspace: 4194304
net.inet.tcp.sendspace: 2097152
net.inet.tcp.recvbuf_max: 16777216
net.inet.tcp.recvbuf_inc: 524288
net.inet.tcp.sendbuf_max: 16777216
net.inet.tcp.sendbuf_inc: 32768
net.route.netisr_maxqlen: 2048
net.link.ifqmaxlen: 2048
need to do more investigation why it won't do 10Gb, maybe its the switch who has wrong settings (it using the defaults settings) or maybe it's the unRAID server...