Suricata IDS/IPS ~56% slower than before update

[klamath]

are you running Suricata and Sensei on the same interface?  It seems that Suricata is crashing and that is causing your gateway monitoring to flap, can you include logs from Suricata?

[carlcedin]

Hi Klamath,

Nope, suricata running on WAN, Sensei are running on LAN with LACP configuration.
here is the suricata log when I started the IPS.

2021-08-25T00:25:18   suricata[97785]   [100262] <Notice> -- all 2 packet processing threads, 4 management threads initialized, engine started.   
2021-08-25T00:25:18   suricata[97785]   [101682] <Notice> -- opened netmap:igb1/T from igb1: 0x68baedfd300   
2021-08-25T00:25:18   suricata[97785]   [101682] <Notice> -- opened netmap:igb1^ from igb1^: 0x68baedfd000   
2021-08-25T00:25:18   suricata[97785]   [100590] <Notice> -- opened netmap:igb1^ from igb1^: 0x68b51534300   
2021-08-25T00:25:17   suricata[97785]   [100590] <Notice> -- opened netmap:igb1/R from igb1: 0x68b51534000   
2021-08-25T00:24:50   suricata[97785]   [100262] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.phpBB3_register_stage2' is checked but not set. Checked in 2010896 and 0 other sigs   
2021-08-25T00:24:50   suricata[97785]   [100262] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.phpBB3_register_stage4' is checked but not set. Checked in 2010897 and 0 other sigs   
2021-08-25T00:24:50   suricata[97785]   [100262] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.phpBB3_test' is checked but not set. Checked in 2010894 and 3 other sigs   
2021-08-25T00:24:50   suricata[97785]   [100262] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ms.rdp.synack' is checked but not set. Checked in 2014384 and 1 other sigs   
2021-08-25T00:24:39   suricata[97785]   [100262] <Warning> -- [ERRCODE: SC_WARN_DEPRECATED(203)] - keyword 'ssh.softwareversion' is deprecated and will be removed soon. Use 'ssh.software' instead. See https://suricata-ids.org/about/deprecation-policy/   
2021-08-25T00:24:39   suricata[97785]   [100262] <Warning> -- [ERRCODE: SC_WARN_DEPRECATED(203)] - keyword 'ssh.softwareversion' is deprecated and will be removed soon. Use 'ssh.software' instead. See https://suricata-ids.org/about/deprecation-policy/   
2021-08-25T00:24:25   suricata[97536]   [100148] <Notice> -- This is Suricata version 5.0.5 RELEASE running in SYSTEM mode   
2021-08-25T00:24:24   suricata[65748]   [100258] <Notice> -- Stats for 'igb1': pkts: 21435809, drop: 1700473 (7.93%), invalid chksum: 0   
2021-08-25T00:24:23   suricata[65748]   [100258] <Notice> -- Signal Received. Stopping engine.

are you running Suricata and Sensei on the same interface?  It seems that Suricata is crashing and that is causing your gateway monitoring to flap, can you include logs from Suricata?

[klamath]

That looks ok, I am wondering if you can include the logs when IDS fails, It seems that it is running successfully.

[carlcedin]

Yes, Its looks good when I started both IPS and sensei.
after few hours usually the problem occur.
When its happened, sensei engine will turn off automatically and I need to turn off IPS first before I can start back IPS and sensei once again.

netstat -ihw 1 also shows no drops.

   packets  errs idrops      bytes    packets  errs      bytes colls
      3.7k     0     0       549K       2.3k     0       404K     0
      3.5k     0     0       671K       2.3k     0       594K     0
      4.5k     0     0       1.1M       3.0k     0       1.5M     0
      3.9k     0     0       782K       2.4k     0       664K     0
      4.3k     0     0       615K       3.0k     0       578K     0
      3.9k     0     0       500K       2.0k     0       352K     0
      3.2k     0     0       741K       2.1k     0       710K     0
      3.3k     0     0       680K       1.5k     0       470K     0
      3.0k     0     0       395K       1.5k     0       245K     0
      4.2k     0     0       771K       2.2k     0       528K     0
      2.4k     0     0       312K       1.3k     0       222K     0
      3.5k     0     0       874K       1.9k     0       689K     0
      2.7k     0     0       317K       1.4k     0       194K     0
      4.2k     0     0       832K       2.2k     0       584K     0
      3.6k     0     0       619K       2.1k     0       410K     0
      4.8k     0     0       1.3M       3.2k     0       1.4M     0
      2.9k     0     0       405K       1.9k     0       301K     0
       22k     0     0       1.7M        20k     0       1.6M     0
      5.3k     0     0       2.4M       3.1k     0       1.2M     0
      5.5k     0     0       1.3M       4.2k     0       1.4M     0
      3.6k     0     0       779K       2.6k     0       854K     0
            input        (Total)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
      4.8k     0     0       1.2M       3.1k     0       1.2M     0
      4.8k     0     0       987K       3.3k     0       807K     0
      4.0k     0     0       736K       1.8k     0       316K     0
      4.1k     0     0       930K       2.5k     0       777K     0
      3.7k     0     0       544K       1.8k     0       313K     0
      2.9k     0     0       478K       1.3k     0       243K     0
      3.8k     0     0       614K       1.8k     0       343K     0
      4.7k     0     0       1.2M       3.3k     0       1.5M     0
      4.3k     0     0       596K       1.8k     0       275K     0
      3.7k     0     0       808K       2.0k     0       725K     0
      3.8k     0     0       736K       2.2k     0       483K     0
      5.1k     0     0       869K       4.0k     0       594K     0
      6.1k     0     0       886K       4.2k     0       1.3M     0
      3.9k     0     0       536K       2.4k     0       354K     0
      3.8k     0     0       580K       1.9k     0       398K     0
      3.3k     0     0       519K       1.7k     0       302K     0
      3.3k     0     0       508K       1.3k     0       236K     0
      2.6k     0     0       413K       1.5k     0       399K     0
      4.0k     0     0       568K       2.0k     0       384K     0
      2.7k     0     0       426K       1.4k     0       340K     0
      3.3k     0     0       730K       1.6k     0       425K     0
            input        (Total)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
      2.8k     0     0       509K       1.5k     0       439K     0
      5.3k     0     0       1.9M       2.9k     0       746K     0
      4.1k     0     0       1.4M       3.1k     0       1.6M     0
      7.8k     0     0       4.4M       3.2k     0       1.7M     0
      2.7k     0     0       679K       1.6k     0       553K     0
      2.7k     0     0       571K       1.3k     0       352K     0
      2.4k     0     0       661K       1.3k     0       348K     0
      3.3k     0     0       500K       1.7k     0       262K     0
      3.1k     0     0       471K       2.1k     0       411K     0

I will let you know, if the problem occur once again.
So far I don't think hardware is the issues here.
I'm running with core i7, 32gb ram. 10g LACP on LAN and the ISP speed just 100mbps.
So it will be enough to handle the process right?

That looks ok, I am wondering if you can include the logs when IDS fails, It seems that it is running successfully.

[h4ck3r]

hi @klamath

OPNsense 21.7.3_3-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
OpenSSL 1.1.1l 24 Aug 2021

Hardware: Dell R720
CPU 1   Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz   Model 62 Stepping 4   2600 MHz 8core
CPU 2   Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz   Model 62 Stepping 4   2600 MHz 8core

Ram : DDR-3   64.00 GB   Presence Detected   Dual Rank   1866 MHz

Ethernet:
NIC Slot 6   Intel(R) Ethernet Converged Network Adapter X540-T2 (WAN,DMZ)
Integrated NIC 1   Intel(R) GbE 4P I350-t rNDC (LAN,MANAGEMENT)

When Suricata is enabled with IDS/IPS protection the max WAN speed is capped at around 650-670Mbps, with IPS mode disabled I can achieve full 827Mb/s down.

I can't say that the ethernet cards we use are not compatible with suricata IPS running on freebsd, because you have witnessed that it works properly in the previous kernel.

At the same time, when I follow the dpinger service, the situation is as follows:

2021-11-12T02:35:16   dpinger[78904]   send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr    
2021-11-11T13:01:05   dpinger[62032]   WAN_GWv4_ X: sendto error: 55   
2021-11-11T02:35:29   dpinger[72741]   GATEWAY ALARM: WAN_GWv4_ (Addr: XAlarm: 0 RTT: 13002us RTTd: 125us Loss: 0%)   
2021-11-11T02:35:29   dpinger[62032]   WAN_GWv4_ X.255.0.37: Clear latency 13002us stddev 125us loss 0%   
2021-11-11T02:35:17   dpinger[38016]   GATEWAY ALARM: WAN_GWv4_ (Addr: X.255.0.37 Alarm: 1 RTT: 12983us RTTd: 102us Loss: 25%)   
2021-11-11T02:35:17   dpinger[62032]   WAN_GWv4_ X.255.0.37: Alarm latency 12983us stddev 102us loss 25%   
2021-11-11T02:35:14   dpinger[62032]   send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr X.255.0.37 bind_addr X.255.0.38 identifier "WAN_GWv4_ "   
2021-11-10T17:00:24   dpinger[89102]   WAN_GWv4_ X.255.0.37: sendto error: 55



It would be great if we could find a solution and suggestion for this problem, thank you for your valuable information sharing.

[klamath]

Hello!

If you have a chance please review https://www.academia.edu/33882347/Suricata_Extreme_Performance_Tuning

I had to disabled most of the intel prefetching options in the BIOS and reduce the TX and RX queues for the nics.  Once I did that I could run IDS/IPS without having any speed issues.

Note that when you start/stop Suricata it will cause dpinger to output errors like you listed.

hi @klamath

OPNsense 21.7.3_3-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
OpenSSL 1.1.1l 24 Aug 2021

Hardware: Dell R720
CPU 1   Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz   Model 62 Stepping 4   2600 MHz 8core
CPU 2   Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz   Model 62 Stepping 4   2600 MHz 8core

Ram : DDR-3   64.00 GB   Presence Detected   Dual Rank   1866 MHz

Ethernet:
NIC Slot 6   Intel(R) Ethernet Converged Network Adapter X540-T2 (WAN,DMZ)
Integrated NIC 1   Intel(R) GbE 4P I350-t rNDC (LAN,MANAGEMENT)

When Suricata is enabled with IDS/IPS protection the max WAN speed is capped at around 650-670Mbps, with IPS mode disabled I can achieve full 827Mb/s down.

I can't say that the ethernet cards we use are not compatible with suricata IPS running on freebsd, because you have witnessed that it works properly in the previous kernel.

At the same time, when I follow the dpinger service, the situation is as follows:

2021-11-12T02:35:16   dpinger[78904]   send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr    
2021-11-11T13:01:05   dpinger[62032]   WAN_GWv4_ X: sendto error: 55   
2021-11-11T02:35:29   dpinger[72741]   GATEWAY ALARM: WAN_GWv4_ (Addr: XAlarm: 0 RTT: 13002us RTTd: 125us Loss: 0%)   
2021-11-11T02:35:29   dpinger[62032]   WAN_GWv4_ X.255.0.37: Clear latency 13002us stddev 125us loss 0%   
2021-11-11T02:35:17   dpinger[38016]   GATEWAY ALARM: WAN_GWv4_ (Addr: X.255.0.37 Alarm: 1 RTT: 12983us RTTd: 102us Loss: 25%)   
2021-11-11T02:35:17   dpinger[62032]   WAN_GWv4_ X.255.0.37: Alarm latency 12983us stddev 102us loss 25%   
2021-11-11T02:35:14   dpinger[62032]   send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr X.255.0.37 bind_addr X.255.0.38 identifier "WAN_GWv4_ "   
2021-11-10T17:00:24   dpinger[89102]   WAN_GWv4_ X.255.0.37: sendto error: 55



It would be great if we could find a solution and suggestion for this problem, thank you for your valuable information sharing.