Performance tuning for IPS maximum performance

[dcol]

I have researched and tested tunables because I have experienced too many down links and poor performance when using IPS/Inline on the WAN interface that could no longer be ignored. This file, loader.conf.local along with adding some system tunables in the WebGUI, has fixed this for me so I thought I would share with the OPNsense community. Sharing is what makes on open-source project successful. Share your experiences using the info in this post. You may or may not see much performance improvement depending on your hardware, but you will see less dropped connections. If you have any other tunable recommendations, please share and post those experiences here. This thread is for performance tuning ideas.

The biggest impact was from the Flow Control (FC) setting. FC is a level 1 layer adding pause frames before the data is transmitted. My assumption is Netmap has issues with FC which causes the dropped connections. Recommendations from many sources, including Cisco, suggest disabling FC altogether and let the higher levels handle the flow. There are exceptions, but these usually involve ESXi, VMware and other special applications.

I have done all my testing using an Intel i350T4 and i340T4, common NICs used for firewalls, in 4 different systems and, by the way, neither NIC had any performance advantage. I have tested these system for 5 days without any down links experienced after the changes were made. Without these changes every system was plagued with down WAN links and poor performance using the default settings.

Do not use this file if you are not using an igb driver. igb combined with other drivers is ok as long as you have at least one igb NIC, and I recommend you use the igb for all WAN interfaces.

Add the file below in the '/boot' folder and call it 'loader.conf.local' right besides 'loader.conf'. I use WinSCP, in a Windows environment, as a file manager to get easy access to the folders. Don't forget to Enable Secure Shell. I have tried using the 'System Tunables' in the WebGUI to add these settings. Some worked and some didn't using that method. Not sure why. Better to just add this file. If you're a Linux guru, I am not, then use your own methods to add this file.

The two most IMPORTANT things to insure is that power management be disabled in the OPNsense settings and also in the BIOS settings of the system (thanks wefinet). And the second is to disable flow control (IEEE 802.3x) on all ports. It is advisable to not connect an IPS interface to any device which has flow control on. Flow control should be turned off to allow the congestion to be managed higher up in the stack

Please test all tunables in a test environment before you apply to a production system.

# File starts below this line, use Copy/Paste #####################
# Check for interface specific settings and add accordingly.
# These ae tunables to improve network performance on Intel igb driver NICs

# Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
# This tunable must be set according to your configuration. VERY IMPORTANT!
# Set FC to 0 (<x>) on all interfaces
dev.igb.<x>.fc=0 #Also put this in System Tunables dev.igb.<x>.fc: value=0

# Set number of queues to number of cores divided by number of ports. 0 lets FreeBSD decide
dev.igb.num_queues=0

# Increase packet descriptors (set as 1024,2048, or 4096) ONLY!
# Allows a larger number of packets to be processed.
# Use "netstat -ihw 1" in the shell and make sure the idrops are zero
# If the NIC has constant disconnects, lower this value
# if not zero then lower this value.
dev.igb.rxd="4096" # For i340/i350 use 2048
dev.igb.txd="4096" # For i340/i350 use 2048
net.link.ifqmaxlen="8192" # value here equal sum of above values. For i340/i350 use 4096

# Increase Network efficiency
dev.igb.enable_aim=1

# Increase interuppt rate
dev.igb.max_interrupt_rate="64000"

# Network memory buffers
# run "netstat -m" in the shell and if the 'mbufs denied' and 'mbufs delayed' are 0/0/0 then this is not needed
# if not zero then keep adding 400000 until mbufs are zero
kern.ipc.nmbclusters="1000000"

# Fast interrupt handling
# Normally set by default. Use these settings to insure it is on.
# Allows NIC to process packets as fast as they are received
dev.igb.enable_msix=1
dev.pci.enable_msix=1

# Unlimited packet processing
# Use this only if you are sure that the NICs have dedicated IRQs
# View the IRQ assignments by executing this in the shell "vmstat -i"
# A value of "-1" means unlimited packet processing
dev.igb.rx_process_limit="-1"
dev.igb.tx_process_limit="-1"
###################################################
# File ends above this line ##################################

##UPDATE 12/12/2017##
After testing I have realized that some of these settings are NOT applied via loader.conf.local and must be added via the WebGUI in System>Settings>Tunables. I have moved these from the file above to this list.
Add to Tunables

Disable Energy Efficiency - set for each igb port in your system
This setting can cause Link flap errors if not disabled
Set for every igb interface in the system as per these examples
dev.igb.0.eee_disabled: value=1
dev.igb.1.eee_disabled: value=1
dev.igb.2.eee_disabled: value=1
dev.igb.3.eee_disabled: value=1

IPv4 Fragments - 0=Do not accept fragments
This is mainly need for security. Fragmentation can be used to evade packet inspection
net.inet.ip.maxfragpackets: value=0
net.inet.ip.maxfragsperpacket: value=0

Set to 0 (<x>) for every port used by IPS
dev.igb.<x>.fc: value=0

##UPDATE 1/16/2018##
Although the tuning in this thread so far just deals with the tunables, there are other settings that can impact IPS performance. Here are a few...

In the Intrusion Detection Settings Tab.

Promiscuous mode- To be used only when multiple interfaces or VLAN's are selected in the Interfaces setting.
This is used so that IPS will capture data on all the selected interfaces. Do not enable if you have just one interface selected. It will help with performance.

Pattern matcher: This setting can select the best  algorithm to use when pattern matching. This setting is best set by testing. Hyperscan seems to work well with Intel NIC's. Try different ones and test the bandwidth with an internet speed test.

Home networks (under advanced menu.
Make sure the interfaces fall within the actual local networks. You may want to change the generic 192.168.0.0/16 to your actual local network ie 192.168.1.1/24

###################################################
USEFUL SHELL COMMANDS
sysctl net.inet.tcp.hostcache.list # View the current host cache stats
vmstat -i # Query total interrupts per queue
top -H -S # Watch CPU usage
dmesg | grep -i msi # Verify MSI-X is being used by the NIC
netstat -ihw 1 # Look for idrops to determine hw.igb.txd and rxd
grep <interface> /var/run/dmesg.boot # Shows useful info like netmap queue/slots
sysctl -A # Shows system variables
###################################################

[mimugmail]

Thanks for sharing!!
What were the results before and after Tuning?

[dcol]

When the flow control was set, I couldn't stay up long enough to get a reading.
Then I tested all the other settings using a line that is rated 300/30 (download/upload)
Using speedtest.net with the settings I get consistent readings @ 311/31 to 315/32 (10 tests)
With the default settings, without changing FC, I get inconsistent readings that varied from 230/20 to 308/28 (also 10 tests). Most tests were below 275/25.

The settings make a difference. Try it.

[fabian]

@dcol Do you want me to make this sticky?

[dcol]

Most definitely! I hope we get some feedback on this with other results

#UPDATE#
I have added some more descriptions and some tests to the original post. Enjoy!

[mimugmail]

Are you really sure the FC values did the trick and not the others?
Normally FC will influence your network badly with TCP .. and most switches doesn't support it either (in both directions).

Would be really interesting, I never did any testing on BSD

[dcol]

Actually I am in the process to figure out how to determine if flow control is enabled or not on a device. Unfortunately ethtool is not part of the distro, so I cannot figure it out yet. Would be nice to have ethtool available as an add-on package.

The command 'ethtool --show-pause igb0' would show if RX or TX was off (no FC) or on (FC enabled).

For me, when FC is enabled on the WAN the link crashes a lot. I spoke with the ISP and they confirmed that there is no FC on the bridged connection.

Most modern unmanaged switches do support flow control, 802.3x, and it is selectable on managed switches and most NIC's.

Also, if you look at the netmap documentation it suggests that flow control can negatively affect performance.
https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4#end

[franco]

Actually I am in the process to figure out how to determine if flow control is enabled or not on a device. Unfortunately ethtool is not part of the distro, so I cannot figure it out yet. Would be nice to have ethtool available as an add-on package.

Not aware of a FreeBSD sibling here, sorry.

Old mailing list threads only suggest sysctl like you found:

https://lists.freebsd.org/pipermail/freebsd-net/2012-July/032868.html


Cheers,
Franco

[Noctur]

Thank you dcol for doing this work and sharing...

Does anyone know or has anyone tried this function with em NICs/drivers? No igb in my box, but I'd like test.

TIA

[dcol]

The following settings will work for the em driver

Put in loader.conf.local
# Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
# This setting must be set according to your configuration. VERY IMPORTANT!
# Set FC to 0(<x>) on every interfaces used by IPS
hw.em.<x>.fc=0 - Also put in System Tunables hw.em.<x>.fc: value=0

hw.em.rx_process_limit=-1
hw.em.enable_msix=1
hw.em.txd=2048
hw.em.rxd=2048
net.link.ifqmaxlen="4096"

Put in Settings>System Tunables
hw.em.eee_setting:  value=0
dev.em.<x>.eee_control: value=0 # replace <x> with interface#, repeat for all installed ports

[franco]

I thought I'd drop by this link regarding previous discussions so that it is not forgotten and can be prodded further. Thanks for your work here. <3

https://github.com/opnsense/core/issues/2083

[mimugmail]

The following settings will work for the em driver

Put in loader.conf.local
# Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
# This setting must be set according to your configuration. VERY IMPORTANT!
# Set FC to 0(<x>) on every interfaces used by IPS
hw.em.<x>.fc=0 - Also put in System Tunables hw.em.<x>.fc: value=0

Is this really hw.driver.number? I only find dev.driver.number is sysctl ..

[dcol]

These were the only em settings in my sysctl
hw.em.eee_setting: 1
hw.em.rx_process_limit: 100
hw.em.enable_msix: 1
hw.em.sbp: 0
hw.em.smart_pwr_down: 0
hw.em.txd: 1024
hw.em.rxd: 1024
hw.em.rx_abs_int_delay: 66
hw.em.tx_abs_int_delay: 66
hw.em.rx_int_delay: 0
hw.em.tx_int_delay: 66
hw.em.disable_crc_stripping: 0

I did see some dev.em settings in the pfsense sysctl but not in OPNsense.
It is possible more settings show up if you have an em driver active. The pfsense did have one active em device.

I would also put these in the tunables
hw.em.eee_setting   value=0
dev.em.<x>.eee_control   value=0 ,<x> being the IPS interface#
Then recheck sysctl and make sure they changed


Tunables are a trial and error thing, but certainly can't hurt to disable any em.eee setting.

[nines]

does anyone know if there are some sort of tunables or fc settings in special for vmxnet3 drivers?

thanks!

[dcol]

Best thing to do is take a look at your sysctl using sysctl -A in a shell
Then see what drivers are in there.
Then you will have a good idea on which drivers you can manipulate.
For example
hw.em.txd is for an Intel driver
hw.igb.txd is for an Intel driver
hw.re.txd is for an Realtek driver
and so on.