OPNsense Forum

English Forums => Intrusion Detection and Prevention => Topic started by: dcol on December 08, 2017, 05:13:30 pm

Title: Performance tuning for IPS maximum performance
Post by: dcol on December 08, 2017, 05:13:30 pm
I have researched and tested tunables because I have experienced too many down links and poor performance when using IPS/Inline on the WAN interface that could no longer be ignored. This file, loader.conf.local along with adding some system tunables in the WebGUI, has fixed this for me so I thought I would share with the OPNsense community. Sharing is what makes on open-source project successful. Share your experiences using the info in this post. You may or may not see much performance improvement depending on your hardware, but you will see less dropped connections. If you have any other tunable recommendations, please share and post those experiences here. This thread is for performance tuning ideas.

The biggest impact was from the Flow Control (FC) setting. FC is a level 1 layer adding pause frames before the data is transmitted. My assumption is Netmap has issues with FC which causes the dropped connections. Recommendations from many sources, including Cisco, suggest disabling FC altogether and let the higher levels handle the flow. There are exceptions, but these usually involve ESXi, VMware and other special applications.

I have done all my testing using an Intel i350T4 and i340T4, common NICs used for firewalls, in 4 different systems and, by the way, neither NIC had any performance advantage. I have tested these system for 5 days without any down links experienced after the changes were made. Without these changes every system was plagued with down WAN links and poor performance using the default settings.

Do not use this file if you are not using an igb driver. igb combined with other drivers is ok as long as you have at least one igb NIC, and I recommend you use the igb for all WAN interfaces.

Add the file below in the '/boot' folder and call it 'loader.conf.local' right besides 'loader.conf'. I use WinSCP, in a Windows environment, as a file manager to get easy access to the folders. Don't forget to Enable Secure Shell. I have tried using the 'System Tunables' in the WebGUI to add these settings. Some worked and some didn't using that method. Not sure why. Better to just add this file. If you're a Linux guru, I am not, then use your own methods to add this file.

The two most IMPORTANT things to insure is that power management be disabled in the OPNsense settings and also in the BIOS settings of the system (thanks wefinet). And the second is to disable flow control (IEEE 802.3x) on all ports. It is advisable to not connect an IPS interface to any device which has flow control on. Flow control should be turned off to allow the congestion to be managed higher up in the stack

Please test all tunables in a test environment before you apply to a production system.

# File starts below this line, use Copy/Paste #####################
# Check for interface specific settings and add accordingly.
# These ae tunables to improve network performance on Intel igb driver NICs

# Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
# This tunable must be set according to your configuration. VERY IMPORTANT!
# Set FC to 0 (<x>) on all interfaces
hw.igb.<x>.fc=0 #Also put this in System Tunables hw.igb.<x>.fc: value=0

# Set number of queues to number of cores divided by number of ports. 0 lets FreeBSD decide
hw.igb.num_queues=0

# Increase packet descriptors (set as 1024,2048, or 4096) ONLY!
# Allows a larger number of packets to be processed.
# Use "netstat -ihw 1" in the shell and make sure the idrops are zero
# If the NIC has constant disconnects, lower this value
# if not zero then lower this value.
hw.igb.rxd="4096" # For i340/i350 use 2048
hw.igb.txd="4096" # For i340/i350 use 2048
net.link.ifqmaxlen="8192" # value here equal sum of above values. For i340/i350 use 4096

# Increase Network efficiency
hw.igb.enable_aim=1

# Increase interuppt rate
hw.igb.max_interrupt_rate="64000"

# Network memory buffers
# run "netstat -m" in the shell and if the 'mbufs denied' and 'mbufs delayed' are 0/0/0 then this is not needed
# if not zero then keep adding 400000 until mbufs are zero
kern.ipc.nmbclusters="1000000"

# Fast interrupt handling
# Normally set by default. Use these settings to insure it is on.
# Allows NIC to process packets as fast as they are received
hw.igb.enable_msix=1
hw.pci.enable_msix=1

# Unlimited packet processing
# Use this only if you are sure that the NICs have dedicated IRQs
# View the IRQ assignments by executing this in the shell "vmstat -i"
# A value of "-1" means unlimited packet processing
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
###################################################
# File ends above this line ##################################

##UPDATE 12/12/2017##
After testing I have realized that some of these settings are NOT applied via loader.conf.local and must be added via the WebGUI in System>Settings>Tunables. I have moved these from the file above to this list.
Add to Tunables

Disable Energy Efficiency - set for each igb port in your system
This setting can cause Link flap errors if not disabled
Set for every igb interface in the system as per these examples
dev.igb.0.eee_disabled: value=1
dev.igb.1.eee_disabled: value=1
dev.igb.2.eee_disabled: value=1
dev.igb.3.eee_disabled: value=1

IPv4 Fragments - 0=Do not accept fragments
This is mainly need for security. Fragmentation can be used to evade packet inspection
net.inet.ip.maxfragpackets: value=0
net.inet.ip.maxfragsperpacket: value=0

Set to 0 (<x>) for every port used by IPS
dev.igb.<x>.fc: value=0

##UPDATE 1/16/2018##
Although the tuning in this thread so far just deals with the tunables, there are other settings that can impact IPS performance. Here are a few...

In the Intrusion Detection Settings Tab.

Promiscuous mode- To be used only when multiple interfaces or VLAN's are selected in the Interfaces setting.
This is used so that IPS will capture data on all the selected interfaces. Do not enable if you have just one interface selected. It will help with performance.

Pattern matcher: This setting can select the best  algorithm to use when pattern matching. This setting is best set by testing. Hyperscan seems to work well with Intel NIC's. Try different ones and test the bandwidth with an internet speed test.

Home networks (under advanced menu.
Make sure the interfaces fall within the actual local networks. You may want to change the generic 192.168.0.0/16 to your actual local network ie 192.168.1.1/24

###################################################
USEFUL SHELL COMMANDS
sysctl net.inet.tcp.hostcache.list # View the current host cache stats
vmstat -i # Query total interrupts per queue
top -H -S # Watch CPU usage
dmesg | grep -i msi # Verify MSI-X is being used by the NIC
netstat -ihw 1 # Look for idrops to determine hw.igb.txd and rxd
grep <interface> /var/run/dmesg.boot # Shows useful info like netmap queue/slots
sysctl -A # Shows system variables
###################################################
Title: Re: Performance tuning for IPS maximum performance
Post by: mimugmail on December 08, 2017, 08:19:13 pm
Thanks for sharing!!
What were the results before and after Tuning?
Title: Re: Performance tuning for IPS maximum performance
Post by: dcol on December 08, 2017, 09:35:29 pm
When the flow control was set, I couldn't stay up long enough to get a reading.
Then I tested all the other settings using a line that is rated 300/30 (download/upload)
Using speedtest.net with the settings I get consistent readings @ 311/31 to 315/32 (10 tests)
With the default settings, without changing FC, I get inconsistent readings that varied from 230/20 to 308/28 (also 10 tests). Most tests were below 275/25.

The settings make a difference. Try it.
Title: Re: Performance tuning for IPS maximum performance
Post by: fabian on December 08, 2017, 09:41:06 pm
@dcol Do you want me to make this sticky?
Title: Re: Performance tuning for IPS maximum performance
Post by: dcol on December 08, 2017, 09:41:55 pm
Most definitely! I hope we get some feedback on this with other results

#UPDATE#
I have added some more descriptions and some tests to the original post. Enjoy!
Title: Re: Performance tuning for IPS maximum performance
Post by: mimugmail on December 08, 2017, 10:54:02 pm
Are you really sure the FC values did the trick and not the others?
Normally FC will influence your network badly with TCP .. and most switches doesn't support it either (in both directions).

Would be really interesting, I never did any testing on BSD :)
Title: Re: Performance tuning for IPS maximum performance
Post by: dcol on December 08, 2017, 11:04:01 pm
Actually I am in the process to figure out how to determine if flow control is enabled or not on a device. Unfortunately ethtool is not part of the distro, so I cannot figure it out yet. Would be nice to have ethtool available as an add-on package.

The command 'ethtool --show-pause igb0' would show if RX or TX was off (no FC) or on (FC enabled).

For me, when FC is enabled on the WAN the link crashes a lot. I spoke with the ISP and they confirmed that there is no FC on the bridged connection.

Most modern unmanaged switches do support flow control, 802.3x, and it is selectable on managed switches and most NIC's.

Also, if you look at the netmap documentation it suggests that flow control can negatively affect performance.
https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4#end
Title: Re: Performance tuning for IPS maximum performance
Post by: franco on December 15, 2017, 07:06:18 am
Actually I am in the process to figure out how to determine if flow control is enabled or not on a device. Unfortunately ethtool is not part of the distro, so I cannot figure it out yet. Would be nice to have ethtool available as an add-on package.

Not aware of a FreeBSD sibling here, sorry. :(

Old mailing list threads only suggest sysctl like you found:

https://lists.freebsd.org/pipermail/freebsd-net/2012-July/032868.html


Cheers,
Franco
Title: Re: Performance tuning for IPS maximum performance
Post by: Noctur on December 16, 2017, 04:18:45 pm
Thank you dcol for doing this work and sharing...

Does anyone know or has anyone tried this function with em NICs/drivers? No igb in my box, but I'd like test.

TIA
Title: Re: Performance tuning for IPS maximum performance
Post by: dcol on December 16, 2017, 05:45:07 pm
The following settings will work for the em driver

Put in loader.conf.local
# Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
# This setting must be set according to your configuration. VERY IMPORTANT!
# Set FC to 0(<x>) on every interfaces used by IPS
hw.em.<x>.fc=0 - Also put in System Tunables hw.em.<x>.fc: value=0

hw.em.rx_process_limit=-1
hw.em.enable_msix=1
hw.em.txd=2048
hw.em.rxd=2048
net.link.ifqmaxlen="4096"

Put in Settings>System Tunables
hw.em.eee_setting:  value=0
dev.em.<x>.eee_control: value=0 # replace <x> with interface#, repeat for all installed ports
Title: Re: Performance tuning for IPS maximum performance
Post by: franco on January 17, 2018, 10:54:21 pm
I thought I'd drop by this link regarding previous discussions so that it is not forgotten and can be prodded further. Thanks for your work here. <3

https://github.com/opnsense/core/issues/2083
Title: Re: Performance tuning for IPS maximum performance
Post by: mimugmail on January 29, 2018, 12:32:07 pm
The following settings will work for the em driver

Put in loader.conf.local
# Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
# This setting must be set according to your configuration. VERY IMPORTANT!
# Set FC to 0(<x>) on every interfaces used by IPS
hw.em.<x>.fc=0 - Also put in System Tunables hw.em.<x>.fc: value=0

Is this really hw.driver.number? I only find dev.driver.number is sysctl ..
Title: Re: Performance tuning for IPS maximum performance
Post by: dcol on January 29, 2018, 04:15:43 pm
These were the only em settings in my sysctl
hw.em.eee_setting: 1
hw.em.rx_process_limit: 100
hw.em.enable_msix: 1
hw.em.sbp: 0
hw.em.smart_pwr_down: 0
hw.em.txd: 1024
hw.em.rxd: 1024
hw.em.rx_abs_int_delay: 66
hw.em.tx_abs_int_delay: 66
hw.em.rx_int_delay: 0
hw.em.tx_int_delay: 66
hw.em.disable_crc_stripping: 0

I did see some dev.em settings in the pfsense sysctl but not in OPNsense.
It is possible more settings show up if you have an em driver active. The pfsense did have one active em device.

I would also put these in the tunables
hw.em.eee_setting   value=0
dev.em.<x>.eee_control   value=0 ,<x> being the IPS interface#
Then recheck sysctl and make sure they changed


Tunables are a trial and error thing, but certainly can't hurt to disable any em.eee setting.
Title: Re: Performance tuning for IPS maximum performance
Post by: nines on February 12, 2018, 09:14:15 pm
does anyone know if there are some sort of tunables or fc settings in special for vmxnet3 drivers?

thanks!
Title: Re: Performance tuning for IPS maximum performance
Post by: dcol on February 12, 2018, 09:27:12 pm
Best thing to do is take a look at your sysctl using sysctl -A in a shell
Then see what drivers are in there.
Then you will have a good idea on which drivers you can manipulate.
For example
hw.em.txd is for an Intel driver
hw.igb.txd is for an Intel driver
hw.re.txd is for an Realtek driver
and so on.
Title: Re: Performance tuning for IPS maximum performance
Post by: elektroinside on February 20, 2018, 12:07:24 pm
Took another closer look at these, after analyzing my settings and reconfigured my box.
They might work, I can't really "feel" if they made much of a difference, but kinda feels like faster.
What I did notice though, that my CPU usage dropped considerably after your settings.
When first trying them out, I must have misconfigured something as things got significantly worse, so I think it was my fault, I wasn't paying the necessary attention.

Thanks dcol for your work!

Title: Re: Performance tuning for IPS maximum performance
Post by: elektroinside on February 26, 2018, 10:32:41 am
An important observation:

These settings (if correctly applied) not only enhance IDS/IPS performance, but OpenVPN as well. OpenVPN performs significantly better without any other configured parameters!
Title: Re: Performance tuning for IPS maximum performance
Post by: Evil_Sense on April 10, 2018, 12:57:45 am
I implemented the tuning settings on my apu2c4.

I added the settings to the tunables in the GUI and used the mentioned command's to check before and afterwards, without noticing something odd.

Like elektroinside said, it fells faster, but it seems that cpu usage and memory usage has slightly increased and sometimes the system feels slower than usual.
I'm not sure if this comes from the rx & tx packet descriptor size of 4096 or the high max interrupt rate of 64000 but it seems a bit too heavy for the little apu2c4 :D.

Suggestions are welcome :)
Title: Re: Performance tuning for IPS maximum performance
Post by: dcol on April 10, 2018, 01:02:26 am
Some of the tunables and settings do come with a resource price. Try reducing the interrupt rate. The queue size is a NIC dependent setting and depends of the buffer size in the NIC itself.
Title: Re: Performance tuning for IPS maximum performance
Post by: Evil_Sense on April 10, 2018, 01:15:13 am
Some of the tunables and settings do come with a resource price. Try reducing the interrupt rate. The queue size is a NIC dependent setting and depends of the buffer size in the NIC itself.
Thanks, will try with interrupt value of 42000 and see if it gets a bit better :)
Title: Re: Performance tuning for IPS maximum performance
Post by: jenmonk on April 12, 2018, 05:18:53 am
With IPS/IDS my internet speed drops to 60mbs from 300mbs.
 I want to try your suggestions. Appreciate if you could let me know how to check ports used by IPS
"Set to 0 (<x>) for every port used by IPS
dev.igb.<x>.fc: value=0"

I followed "Fast and easy way to protect your home and/or small office network with OPNsense"  for my Initial setup
Thanks
Title: Re: Performance tuning for IPS maximum performance
Post by: mimugmail on April 12, 2018, 11:42:02 am
I did some testing in a 10G Lab:

#####################################
OPNsense 18.1.6-amd64
FreeBSD 11.1-RELEASE-p9
OpenSSL 1.0.2o 27 Mar 201

Intel(R) Xeon(R) CPU E3-1270 v5 @ 3.60GHz (8 cores)

16GB RAM

Intel X520DA SFP+

Suricata, 11700 Rules enabled

Tests with iperf3:
iperf3 -p 5000 -f m -V -c 10.0.2.10 -t 30 -P 10 -w 12M
#####################################

No IPS enabled: 9400Mbit (30% CPU load)
IDS enabled: 9400Mbit (45% CPU load)
IPS enabled (Default Pattern matcher): 550Mbit (17% CPU load)
IPS enabled (Hyperscan): 1400Mbit (17% CPU load)

Title: Re: Performance tuning for IPS maximum performance
Post by: jenmonk on April 12, 2018, 07:44:13 pm
With IPS/IDS my internet speed drops to 60mbs from 300mbs.
 I want to try your suggestions. Appreciate if you could let me know how to check ports used by IPS
"Set to 0 (<x>) for every port used by IPS
dev.igb.<x>.fc: value=0"

I followed "Fast and easy way to protect your home and/or small office network with OPNsense"  for my Initial setup
Thanks

Appreciate all the help
Title: Re: Performance tuning for IPS maximum performance
Post by: Julien on April 21, 2018, 11:00:17 pm
Some of the tunables and settings do come with a resource price. Try reducing the interrupt rate. The queue size is a NIC dependent setting and depends of the buffer size in the NIC itself.
Thanks, will try with interrupt value of 42000 and see if it gets a bit better :)
Hi EVIL_Sense,
after changing the 42000 value, have you noticed some changes / speed ?
i am willing to get this configured on a production soon as we are from 1024MB when IDS is activated we reach 400MB

Title: Re: Performance tuning for IPS maximum performance
Post by: Evil_Sense on April 21, 2018, 11:29:38 pm
Some of the tunables and settings do come with a resource price. Try reducing the interrupt rate. The queue size is a NIC dependent setting and depends of the buffer size in the NIC itself.
Thanks, will try with interrupt value of 42000 and see if it gets a bit better :)
Hi EVIL_Sense,
after changing the 42000 value, have you noticed some changes / speed ?
i am willing to get this configured on a production soon as we are from 1024MB when IDS is activated we reach 400MB
Well, with 42000 I got a reasonable balance between resource usage and (at least I hope) good/better networking performance.
Title: Re: Performance tuning for IPS maximum performance
Post by: Julien on April 22, 2018, 10:32:13 pm
Some of the tunables and settings do come with a resource price. Try reducing the interrupt rate. The queue size is a NIC dependent setting and depends of the buffer size in the NIC itself.
Thanks, will try with interrupt value of 42000 and see if it gets a bit better :)
Hi EVIL_Sense,
after changing the 42000 value, have you noticed some changes / speed ?
i am willing to get this configured on a production soon as we are from 1024MB when IDS is activated we reach 400MB
Well, with 42000 I got a reasonable balance between resource usage and (at least I hope) good/better networking performance.
Can you share the value ? how much is before and after the IDS is activated ?
i am willing to configure this as the firewall is not near to me, if things missed up i will need to travel like 4 hrs go and 4 hr back.
Title: Re: Performance tuning for IPS maximum performance
Post by: Evil_Sense on April 23, 2018, 12:45:39 am
Some of the tunables and settings do come with a resource price. Try reducing the interrupt rate. The queue size is a NIC dependent setting and depends of the buffer size in the NIC itself.
Thanks, will try with interrupt value of 42000 and see if it gets a bit better :)
Hi EVIL_Sense,
after changing the 42000 value, have you noticed some changes / speed ?
i am willing to get this configured on a production soon as we are from 1024MB when IDS is activated we reach 400MB
Well, with 42000 I got a reasonable balance between resource usage and (at least I hope) good/better networking performance.
Can you share the value ? how much is before and after the IDS is activated ?
i am willing to configure this as the firewall is not near to me, if things missed up i will need to travel like 4 hrs go and 4 hr back.
I don't use IDS, so I can't give a statement on it.
Since I didn't write down the original settings and didn't make speed tests before and after, I'm not really able to provide reliable values. I could however try to remove the settings and measuring against the current state tomorrow.
Title: Re: Performance tuning for IPS maximum performance
Post by: Julien on April 23, 2018, 10:15:40 pm
Some of the tunables and settings do come with a resource price. Try reducing the interrupt rate. The queue size is a NIC dependent setting and depends of the buffer size in the NIC itself.
Thanks, will try with interrupt value of 42000 and see if it gets a bit better :)
Hi EVIL_Sense,
after changing the 42000 value, have you noticed some changes / speed ?
i am willing to get this configured on a production soon as we are from 1024MB when IDS is activated we reach 400MB
Well, with 42000 I got a reasonable balance between resource usage and (at least I hope) good/better networking performance.
Can you share the value ? how much is before and after the IDS is activated ?
i am willing to configure this as the firewall is not near to me, if things missed up i will need to travel like 4 hrs go and 4 hr back.
I don't use IDS, so I can't give a statement on it.
Since I didn't write down the original settings and didn't make speed tests before and after, I'm not really able to provide reliable values. I could however try to remove the settings and measuring against the current state tomorrow.

Thank you,
if you could do that i'll appreciate it.
Title: Re: Performance tuning for IPS maximum performance
Post by: Julien on April 25, 2018, 01:20:33 am
I have IDS enabled using only 1 rule " abuse.ch/SSL IP Blacklist " after testing the speed test its drop simnifically .

hardware is
Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz (4 cores)
Memory 16 % ( 1301/8054 MB )
Title: Re: Performance tuning for IPS maximum performance
Post by: mimugmail on April 25, 2018, 05:53:32 am
IDS or IPS?
Do you use Hyperscan?
Title: Re: Performance tuning for IPS maximum performance
Post by: Julien on April 25, 2018, 01:09:29 pm
IDS or IPS?
Do you use Hyperscan?
yes i am using hyperscan and using Intrusion Detection with IPS mode on see screenshot.
Title: Re: Performance tuning for IPS maximum performance
Post by: Evil_Sense on April 25, 2018, 03:35:17 pm
I finally found time for some tests..

I first tested with the tunables and a system running for couple weeks.

I then removed the tunables, rebooted, waited for 5 minutes and tested again.

Lastly I added the tunables again, rebooted, waited for 5 minutes and tested again.

As you see, the results are within tolerance, could be because my provider connection doesn't saturate the nic capacity of my apu2c4.
Title: Re: Performance tuning for IPS maximum performance
Post by: Evil_Sense on April 25, 2018, 03:36:26 pm
I also attached utilization screenshots, with the tunables it's higher, but since I don't mind using the hardware a bit more I'm ok with that.

(Second post, because only 4 pictures per posts allowed)
Title: Re: Performance tuning for IPS maximum performance
Post by: Julien on April 26, 2018, 03:04:24 pm
Thank you Evil Sens for your answer.
as i understand you didnt really noticed the speed but les on the hardware use.
as i understand i dont mind using the hardware that why we have it there :)
Title: Re: Performance tuning for IPS maximum performance
Post by: neoso on May 30, 2018, 09:22:41 am
Some of the tunables and settings do come with a resource price. Try reducing the interrupt rate. The queue size is a NIC dependent setting and depends of the buffer size in the NIC itself.

Hi,

Is possible put youtr config, in APU2C4?

I read de tuto, bit when insert the config in the loader.conf, when reboot i lost all config.

I hace a FFTH 600MB/600MB .
IPS/IDS activo :  100/100MB
IDS/IPS not active:   300MB/600MB

Is posible that the APU2C4 is poor hardware?

I have ordered a QOTOM on ALLIEXPRESS core i7 8Gb RAM

Do you think that installing PFSENSE will improve this in the APU2C4?
Title: Re: Performance tuning for IPS maximum performance
Post by: xmichielx on August 02, 2018, 10:54:36 am
The config should be in loader.conf.local and some in the tunables.
I tried it for the APU 2C4 but still max ~10/11 MB/s with Suricata inline, Snort with some PF magic (PFSense) gives the full bandwidth.
It's not a true inline IPS but works pretty good for home usage.
Perhaps one day when home hardware (like the APU2c4 which is quad core with 4 GB memory) works nicely with Suricata I will switch, untill then I use Snort since losing 60% of your bandwidth is just not worth it.
Title: Re: Performance tuning for IPS maximum performance
Post by: mimugmail on August 02, 2018, 10:57:02 am
The config should be in loader.conf.local and some in the tunables.
I tried it for the APU 2C4 but still max ~10/11 MB/s with Suricata inline, Snort with some PF magic (PFSense) gives the full bandwidth.
It's not a true inline IPS but works pretty good for home usage.
Perhaps one day when home hardware (like the APU2c4 which is quad core with 4 GB memory) works nicely with Suricata I will switch, untill then I use Snort since losing 60% of your bandwidth is just not worth it.

How many rules do you run on Snort vs Suricata? Can you try changing the Scan engine?
Title: Re: Performance tuning for IPS maximum performance
Post by: dcol on August 02, 2018, 04:07:45 pm
Two point.
OPNsense does not have Snort. OPNsense was built optimizing Suricata.
Some Snort rules are not compatible with Suricata.
Title: Re: Performance tuning for IPS maximum performance
Post by: xmichielx on August 09, 2018, 10:17:17 am
The config should be in loader.conf.local and some in the tunables.
I tried it for the APU 2C4 but still max ~10/11 MB/s with Suricata inline, Snort with some PF magic (PFSense) gives the full bandwidth.
It's not a true inline IPS but works pretty good for home usage.
Perhaps one day when home hardware (like the APU2c4 which is quad core with 4 GB memory) works nicely with Suricata I will switch, untill then I use Snort since losing 60% of your bandwidth is just not worth it.

How many rules do you run on Snort vs Suricata? Can you try changing the Scan engine?

the same ammount; I use the ET Open rules and both work for both Snort and Suricata.
Tried enabling 1 rule to using 15 rules - no difference.
Also tried changing the Scan engine, Hyperscan has the best performance (Intel nic's are used on the APU 2) but no profit there.
Title: Re: Performance tuning for IPS maximum performance
Post by: xmichielx on August 09, 2018, 10:19:59 am
Two point.
OPNsense does not have Snort. OPNsense was built optimizing Suricata.
Some Snort rules are not compatible with Suricata.

I never said that OPNsense have snort that is why I use/used PFsense.
I know that some Snort rules are incompatible with Suricata, I use the supplied ET Open rules and they work for both IDS/IPS.
Still not related to the performance hit on the APU 2, actually I can not find 1 single post where someone says he has 75%-100% of his/hers bandwidth after using Suricata inline (this has nothing to do with OPNsense but is related to Suricata and its scanning engine which caps bandwidth inline when used on 'smaller' hardware for home use).
Title: Re: Performance tuning for IPS maximum performance
Post by: mimugmail on August 09, 2018, 10:55:05 am
I think this is somewhat clocking related which impacts higher on slow hardware.
No idea how Snort on PF works, perhaps it adds an pf rules after match which doesn't require real inline so it might be more performant on smaller hardware, but just a guess.


I'm not against building a snort plugin .. but I'm not sure if it's worth the work since IPS on home use is debatable (my personal opinion)
Title: Re: Performance tuning for IPS maximum performance
Post by: xmichielx on August 09, 2018, 11:36:08 am
I must nuance my 'rant' about Suricata; after enabling just the ones that are the most necessary for me (aka trojan, malware, mobile_malware, explot) and using Hyperscan I get a more reasonable ~14-16 MB/s (where 22 MB/s is my max) which is acceptable for me.
I no have the benefit of using a NIDS/IPS blocking/filtering on the LAN/GUEST_VLAN interfaces and still remain some of my bandwidth.
So big tip for all APU 2 users: use the Hyperscan Scan engine and choose only what is necessary.
I did not use any of the tweaks except above mentioned :)
Title: Re: Performance tuning for IPS maximum performance
Post by: dcol on August 09, 2018, 04:02:23 pm
Using only rules that are 'necessary' is always the proper method. Just takes some homework. If your internal LAN is trusted, then you don't need to use IDS on it. Logic is always the best approach.
Title: Re: Performance tuning for IPS maximum performance
Post by: xmichielx on August 10, 2018, 09:42:58 am
I use the IPS mainly for my LAN/Guest VLAN since I want to detect malware. But I can understand that people also use it on front of their servers etc.
PS changing the networks from 3 private ranges to only 192.168.0.0/16 seems also to effect the bandwith (+/- 1 or 2 MB/s profit!)
Title: Re: Performance tuning for IPS maximum performance
Post by: Julien on November 17, 2018, 02:06:28 am
I use the IPS mainly for my LAN/Guest VLAN since I want to detect malware. But I can understand that people also use it on front of their servers etc.
PS changing the networks from 3 private ranges to only 192.168.0.0/16 seems also to effect the bandwith (+/- 1 or 2 MB/s profit!)
Our internal LAN is trusted as its clean and we know what is running in the internal.
Do you mean we do not need to use IDS for this ? we do have some servers behind and want them to be protect

we keep having one alert from this IP 150.109.50.77 on port 25 in and out and the action is allowed

Code: [Select]
Timestamp 2018-11-17T01:58:28.386557+0100
Alert SURICATA SMTP data command rejected
Alert sid 2220008
Protocol TCP
Source IP 2.51.55.22
Destination IP 150.109.50.77
Source port 25
Destination port 35064
Interface wan
any suggestions how to trade this alert ?
Title: Re: Performance tuning for IPS maximum performance
Post by: massaquah on November 20, 2018, 11:09:00 am
I recently got an upgrade for my internet badwidth from 200/50 mbit to 1000/50 mbit.

Sadly, my initial speed tests only resulted in 160 / 50 mbit.

I quickly identified Suricata with activated IPS as the bottleneck. I tried each combination of  hyperscan vs aho-corasick, activation of Suricata on LAN (igb), LAN+WAN, WAN(em), every performance tuning rule described in the first post of this thread but still I got only around 160 / 50 with IPS enabled.

I also noticed that the Suricata process uses 100% of one CPU core during speed tests whereas the remaining three cores were ideling.
Also, disabling most of the rules resulted in a "successfull" speed test of 950 / 50 mbit.

So my question is, why doesn't Suricata make use of all four cores? Why is the clock speed of a single core the bottleneck here? From what I understood reading about Suricata, it should be capable of multithreading?

Title: Re: Performance tuning for IPS maximum performance
Post by: mimugmail on November 20, 2018, 11:29:14 am
What's your hardware? It always depends on hardware ...
Title: Re: Performance tuning for IPS maximum performance
Post by: massaquah on November 20, 2018, 12:16:17 pm
Intel Pentium G4560T (2 cores, 4 threads) with 2.90 GHZ + 8 GB RAM.

But apart from the clock speed, why is only one core being used by suricata?
Title: Re: Performance tuning for IPS maximum performance
Post by: mimugmail on November 20, 2018, 12:54:17 pm
ps aufxH  (H is important)
Title: Re: Performance tuning for IPS maximum performance
Post by: Sahbi on February 12, 2019, 09:39:03 pm
Had some severe performance issues after enabling IPS mode, like barely saturating 50% of my ISP connection (supposed to be 250/25 Mbps). So I figured I'd chime in with some of my experiences. I'm assuming that since I have an APU4C4 with i211AT NICs, flow control is set to 3 (Full) since it seems to support that according to this here datasheet (https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i211-ethernet-controller-datasheet.pdf). Also I'm using speedtest.net because it's still the most popular one and at least they have decent connected servers close to me, unlike e.g. Google which goes all the way to damn Atlanta. I always used the same server, as well as the relatively new "multi" feature. I'm also running the speedtests from a computer behind OPNSense and not from the box itself. Finally, I have pretty much everything enabled at this point, this includes a transparent HTTPS proxy which requires me to disable hardware offloading for some networking stuff.

First, let's list the rulesets I have in use. Now, I'm not that familiar with OPN nor Suricata yet so I'm not entirely sure if below data is "clean", but should be close enough.
Code: [Select]
root@opn:/usr/local/etc/suricata/rules # ls *.rules
OPNsense.rules emerging-icmp_info.rules
abuse.ch.feodotracker.rules emerging-imap.rules
abuse.ch.sslblacklist.rules emerging-info.rules
abuse.ch.sslipblacklist.rules emerging-malware.rules
abuse.ch.urlhaus.rules emerging-misc.rules
botcc.portgrouped.rules emerging-mobile_malware.rules
botcc.rules emerging-rpc.rules
ciarmy.rules emerging-scan.rules
compromised.rules emerging-shellcode.rules
drop.rules emerging-smtp.rules
dshield.rules emerging-sql.rules
emerging-activex.rules emerging-trojan.rules
emerging-attack_response.rules emerging-user_agents.rules
emerging-current_events.rules emerging-web_client.rules
emerging-deleted.rules emerging-web_server.rules
emerging-dns.rules emerging-web_specific_apps.rules
emerging-dos.rules emerging-worm.rules
emerging-exploit.rules opnsense.test.rules
emerging-ftp.rules opnsense.uncategorized.rules
emerging-icmp.rules

root@opn:/usr/local/etc/suricata/rules # cat *.rules | sed 's/^ *#.*//' | sed '/^ *$/d' | wc -l
   41614

The rules are divided about 50/50 in regards to drop/alert actions, but I don't think that matters for performance because it has to log stuff regardless.

This is before applying any of the tunables mentioned in the OP (at my speeds I don't care about decimals so I'll just round that shit):
I read somewhere on these forums that Hyperscan is preferred in most cases, as such I had that active which caused a significant performance drop compared to A-C. So this was the cause for my issues, at least at the moment. :>

After running sysctl dev.igb.<x>.fc=0 for all interfaces (no need to reboot for these so figured I'd just go ahead and try):
A slight improvement for both algos, with Hyperscan closing the most distance. RAM usage for both tests stayed pretty much the same, there's currently 50% in use after having been a day in full production. Also, after every reboot I waited for the startup beep to go off, then checked with top to see if any startup stuff was still running. Only when everything calmed down will I proceed with the next test.

Now let's try some more tunables:
Code: [Select]
### loader.conf.local

# Flow Control (FC): 0 = Disabled, 1 = Rx Pause, 2 = Tx Pause, 3 = Full FC
hw.igb.0.fc=0
hw.igb.1.fc=0
hw.igb.2.fc=0
hw.igb.3.fc=0

# Set number of queues to number of cores divided by number of ports, 0 lets FreeBSD decide (should be default)
hw.igb.num_queues=0

# Increase packet descriptors (set as 1024, 2048 or 4096 ONLY)
hw.igb.rxd="4096" # Default = 1024
hw.igb.txd="4096"
net.link.ifqmaxlen="8192" # Sum of above two (default = 50)

# Increase network efficiency (Adaptive Interrupt Moderation, should be default)
hw.igb.enable_aim=1

# Increase interrupt rate # Default = 8000
hw.igb.max_interrupt_rate="64000"

# Fast interrupt handling, allows NIC to process packets as fast as they are received (should be default)
hw.igb.enable_msix=1
hw.pci.enable_msix=1

# Unlimited packet processing
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"

### WebGUI > System > Settings > Tunables

# Disable Energy Efficient Ethernet
dev.igb.0.eee_disabled=1
dev.igb.1.eee_disabled=1
dev.igb.2.eee_disabled=1
dev.igb.3.eee_disabled=1

# Set Flow Control
hw.igb.0.fc=0
hw.igb.1.fc=0
hw.igb.2.fc=0
hw.igb.3.fc=0

dev.igb.0.fc=0
dev.igb.1.fc=0
dev.igb.2.fc=0
dev.igb.3.fc=0

# Do not accept IPv4 fragments
net.inet.ip.maxfragpackets=0
net.inet.ip.maxfragsperpacket=0

And reboot. =]

RAM usage is still hovering fine and dandy around 45%.
Now one thing I also noticed while watching top -HS is that Suricata no longer takes an entire core + a bit from the second, but instead distributes its load over 3 cores with the total load being around 180% (out of 400%). It also feels like the web interface is "snappier"; the dashboard page used to take quite some time to load but it's mucho faster now.



So it seems that just disabling flow control brings some slight improvements already, but Hyperscan in particular benefits hugely from adjusting hw.igb.rxd/txd, net.link.ifqmaxlen and hw.igb.max_interrupt_rate. Apparently with newer BSDs (like 10.x onwards) there's a newer driver which reduces the amount of interrupts significantly (https://calomel.org/freebsd_network_tuning.html), so you can probably just set it to 16000 and have the same results. I'm routing a lot of stuff due to a complex homelab setup, so I'll just leave it at 64k for now. =] Probably worth mentioning too, but my lil' APU's CPU temps have never went over 60C so far while after a cold boot it starts at around 59.

Since the difference between A-C and HS at this point is negligible and most likely just the result of tiny factors such as other services happening to check in at the time, I'm satisfied with the current settings and will end my tunables testing here. For shits and giggles I did run an iperf just now, from the same computer behind OPN to a VPS with gigabit in the same country:
Code: [Select]
$ iperf -c vps1 -p 4712 -u -t 60 -i 10 -b 1000M
------------------------------------------------------------
Client connecting to vps1, UDP port 4712
Sending 1470 byte datagrams, IPG target: 11.22 us (kalman adjust)
UDP buffer size: 9.00 KByte (default)
------------------------------------------------------------
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  1.11 GBytes   954 Mbits/sec
[  5] 10.0-20.0 sec  1.11 GBytes   952 Mbits/sec
[  5] 20.0-30.0 sec  1.11 GBytes   954 Mbits/sec
[  5] 30.0-40.0 sec  1.11 GBytes   953 Mbits/sec
[  5] 40.0-50.0 sec  1.11 GBytes   955 Mbits/sec
[  5]  0.0-60.0 sec  6.66 GBytes   953 Mbits/sec
[  5] Sent 4864635 datagrams

Suricata takes a little less than 1 core and the temps are still around 59C. :>