Messages

556

17.1 Legacy Series / Re: WAN link gone sometimes (igb driver, I211 nics), ifconfig d/u fixes it

« on: January 01, 2018, 08:07:07 pm »

That link is for development and simulation tools. probably a good place if you were going to hack the drivers or firmware.
This is what you want
https://downloadcenter.intel.com/download/19186

557

17.1 Legacy Series / Re: WAN link gone sometimes (igb driver, I211 nics), ifconfig d/u fixes it

« on: January 01, 2018, 06:43:45 pm »

That Intel download link is for drivers, not firmware. In the FreeBSD environment we have no control over the drivers that are used. The firmware is included as part of the bootutil software. But, would be nice to have a hacked driver with no PM at all that could be compiled into the FreeBSD OS.

558

17.1 Legacy Series / Re: WAN link gone sometimes (igb driver, I211 nics), ifconfig d/u fixes it

« on: December 30, 2017, 09:29:15 pm »

There are a few items of concern here.
First, the num_queues setting has to do with the number of cores available divided by the number of ports. There should never be more ports than cores or the queues will overrun and could cause a reset of the port. The value of num_queues should be less than or equal to the cores/ports number. This is automatically calculated by the OS if not overridden by the settings. As an example, if you have 4 cores and 3 ports, the num_queues should be 4/3=1.33 which should be set to 1.

Secondly, the eee setting must be done in the tunables section, as the eee setting does not work in the loader.conf.local. Also, all power management settings in the BIOS should be disabled.
You can use the command 'sysctrl -A' in the shell to see the actual settings in use.

559

17.1 Legacy Series / Re: [FIXED] WAN link gone sometimes (igb driver, I211 nics), ifconfig d/u fixes it

« on: December 18, 2017, 05:05:24 pm »

You could also use Intel's Bootutil to disable power management on the NIC using the following command
BootUtil --WOLD

Works with all Intel NIC's

Get the tool here
https://downloadcenter.intel.com/downloads/eula/19186/Intel-Ethernet-Connections-Boot-Utility-Preboot-Images-and-EFI-Drivers?httpDown=https%3A%2F%2Fdownloadmirror.intel.com%2F19186%2Feng%2FPREBOOT.EXE

Intel's webpage
https://downloadcenter.intel.com/download/19186/Intel-Ethernet-Connections-Boot-Utility-Preboot-Images-and-EFI-Drivers

560

Intrusion Detection and Prevention / Re: Performance tuning for IPS maximum performance

« on: December 16, 2017, 05:45:07 pm »

The following settings will work for the em driver

Put in loader.conf.local
# Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
# This setting must be set according to your configuration. VERY IMPORTANT!
# Set FC to 0(<x>) on every interfaces used by IPS
hw.em.<x>.fc=0 - Also put in System Tunables hw.em.<x>.fc: value=0

hw.em.rx_process_limit=-1
hw.em.enable_msix=1
hw.em.txd=2048
hw.em.rxd=2048
net.link.ifqmaxlen="4096"

Put in Settings>System Tunables
hw.em.eee_setting:  value=0
dev.em.<x>.eee_control: value=0 # replace <x> with interface#, repeat for all installed ports

561

Intrusion Detection and Prevention / Re: Comment on Inline mode

« on: December 13, 2017, 12:29:38 am »

Funny, never got it to work on the other system without packet errors on any computer using Inline. I think that my bridged pass through WAN connection was just too active for that system. probably too much overhead for netmap. But no issues on OPNsense, same computer so the simplicity paid off. Thanks.

I never used an e1000 or emulator mode so I didn't see any issues while on FreeBSD 11.0.  Now using 11.1

562

Intrusion Detection and Prevention / Re: IPS blocked, then allowed

« on: December 12, 2017, 10:48:51 pm »

Just tried it on a pfs box with the same results.

563

Intrusion Detection and Prevention / Re: link goes down on WAN using IDS/IPS

« on: December 12, 2017, 03:56:26 pm »

I stand corrected, sort of. I do suggest at least 4 cores and 2Ghz processing power as well as a minimum 8GB memory. The sticky in this topic has more to do with the issue.

564

Intrusion Detection and Prevention / Re: IPS blocked, then allowed

« on: December 12, 2017, 03:53:22 pm »

This issue is not an OPNsense issue since the same thing happens in other firewalls using Suricata.

565

Intrusion Detection and Prevention / Comment on Inline mode

« on: December 12, 2017, 03:51:56 pm »

I find it strange that Inline mode works without issues in OPNsense while the other solution has nothing but trouble with it and tells everyone not to use it because it is too buggy. Am I missing something? Are they missing something?

If OPNsense is not masking the netmap bad packet errors then the only conclusion I can come to is the simplicity of OPNsense is what makes it better and more efficient in packet handling with netmap.

I would love to hear some responses to this from the devs of OPNsense. Great job guys, you truly have your finger on how to develop a superior product by learning from others mistakes. Its no wonder this product forked. Thanks!

566

Intrusion Detection and Prevention / Re: Performance tuning for IPS maximum performance

« on: December 08, 2017, 11:04:01 pm »

Actually I am in the process to figure out how to determine if flow control is enabled or not on a device. Unfortunately ethtool is not part of the distro, so I cannot figure it out yet. Would be nice to have ethtool available as an add-on package.

The command 'ethtool --show-pause igb0' would show if RX or TX was off (no FC) or on (FC enabled).

For me, when FC is enabled on the WAN the link crashes a lot. I spoke with the ISP and they confirmed that there is no FC on the bridged connection.

Most modern unmanaged switches do support flow control, 802.3x, and it is selectable on managed switches and most NIC's.

Also, if you look at the netmap documentation it suggests that flow control can negatively affect performance.
https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4#end

567

Intrusion Detection and Prevention / Re: Performance tuning for IPS maximum performance

« on: December 08, 2017, 09:41:55 pm »

Most definitely! I hope we get some feedback on this with other results

#UPDATE#
I have added some more descriptions and some tests to the original post. Enjoy!

568

Intrusion Detection and Prevention / Re: Performance tuning for IPS maximum performance

« on: December 08, 2017, 09:35:29 pm »

When the flow control was set, I couldn't stay up long enough to get a reading.
Then I tested all the other settings using a line that is rated 300/30 (download/upload)
Using speedtest.net with the settings I get consistent readings @ 311/31 to 315/32 (10 tests)
With the default settings, without changing FC, I get inconsistent readings that varied from 230/20 to 308/28 (also 10 tests). Most tests were below 275/25.

The settings make a difference. Try it.

569

Intrusion Detection and Prevention / Re: IPS blocked, then allowed

« on: December 08, 2017, 05:39:45 pm »

I submitted a bug report to OPNsense git-hub and Suricata. Neither has responded, so far.

According to the Suricata manual, a drop is definitely suppose to end the chain, but in my case it does not. This has happened with many other matching rules as well. Looks like the rules engine processes all packets against every rule in its algorithm. This has to impede performance and stress to netmap.

At this point I am not sure if it is an adjustable setting in the Suricata configuration or a bug.

570

Intrusion Detection and Prevention / Performance tuning for IPS maximum performance

« on: December 08, 2017, 05:13:30 pm »

I have researched and tested tunables because I have experienced too many down links and poor performance when using IPS/Inline on the WAN interface that could no longer be ignored. This file, loader.conf.local along with adding some system tunables in the WebGUI, has fixed this for me so I thought I would share with the OPNsense community. Sharing is what makes on open-source project successful. Share your experiences using the info in this post. You may or may not see much performance improvement depending on your hardware, but you will see less dropped connections. If you have any other tunable recommendations, please share and post those experiences here. This thread is for performance tuning ideas.

The biggest impact was from the Flow Control (FC) setting. FC is a level 1 layer adding pause frames before the data is transmitted. My assumption is Netmap has issues with FC which causes the dropped connections. Recommendations from many sources, including Cisco, suggest disabling FC altogether and let the higher levels handle the flow. There are exceptions, but these usually involve ESXi, VMware and other special applications.

I have done all my testing using an Intel i350T4 and i340T4, common NICs used for firewalls, in 4 different systems and, by the way, neither NIC had any performance advantage. I have tested these system for 5 days without any down links experienced after the changes were made. Without these changes every system was plagued with down WAN links and poor performance using the default settings.

Do not use this file if you are not using an igb driver. igb combined with other drivers is ok as long as you have at least one igb NIC, and I recommend you use the igb for all WAN interfaces.

Add the file below in the '/boot' folder and call it 'loader.conf.local' right besides 'loader.conf'. I use WinSCP, in a Windows environment, as a file manager to get easy access to the folders. Don't forget to Enable Secure Shell. I have tried using the 'System Tunables' in the WebGUI to add these settings. Some worked and some didn't using that method. Not sure why. Better to just add this file. If you're a Linux guru, I am not, then use your own methods to add this file.

The two most IMPORTANT things to insure is that power management be disabled in the OPNsense settings and also in the BIOS settings of the system (thanks wefinet). And the second is to disable flow control (IEEE 802.3x) on all ports. It is advisable to not connect an IPS interface to any device which has flow control on. Flow control should be turned off to allow the congestion to be managed higher up in the stack

Please test all tunables in a test environment before you apply to a production system.

# File starts below this line, use Copy/Paste #####################
# Check for interface specific settings and add accordingly.
# These ae tunables to improve network performance on Intel igb driver NICs

# Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
# This tunable must be set according to your configuration. VERY IMPORTANT!
# Set FC to 0 (<x>) on all interfaces
dev.igb.<x>.fc=0 #Also put this in System Tunables dev.igb.<x>.fc: value=0

# Set number of queues to number of cores divided by number of ports. 0 lets FreeBSD decide
dev.igb.num_queues=0

# Increase packet descriptors (set as 1024,2048, or 4096) ONLY!
# Allows a larger number of packets to be processed.
# Use "netstat -ihw 1" in the shell and make sure the idrops are zero
# If the NIC has constant disconnects, lower this value
# if not zero then lower this value.
dev.igb.rxd="4096" # For i340/i350 use 2048
dev.igb.txd="4096" # For i340/i350 use 2048
net.link.ifqmaxlen="8192" # value here equal sum of above values. For i340/i350 use 4096

# Increase Network efficiency
dev.igb.enable_aim=1

# Increase interuppt rate
dev.igb.max_interrupt_rate="64000"

# Network memory buffers
# run "netstat -m" in the shell and if the 'mbufs denied' and 'mbufs delayed' are 0/0/0 then this is not needed
# if not zero then keep adding 400000 until mbufs are zero
kern.ipc.nmbclusters="1000000"

# Fast interrupt handling
# Normally set by default. Use these settings to insure it is on.
# Allows NIC to process packets as fast as they are received
dev.igb.enable_msix=1
dev.pci.enable_msix=1

# Unlimited packet processing
# Use this only if you are sure that the NICs have dedicated IRQs
# View the IRQ assignments by executing this in the shell "vmstat -i"
# A value of "-1" means unlimited packet processing
dev.igb.rx_process_limit="-1"
dev.igb.tx_process_limit="-1"
###################################################
# File ends above this line ##################################

##UPDATE 12/12/2017##
After testing I have realized that some of these settings are NOT applied via loader.conf.local and must be added via the WebGUI in System>Settings>Tunables. I have moved these from the file above to this list.
Add to Tunables

Disable Energy Efficiency - set for each igb port in your system
This setting can cause Link flap errors if not disabled
Set for every igb interface in the system as per these examples
dev.igb.0.eee_disabled: value=1
dev.igb.1.eee_disabled: value=1
dev.igb.2.eee_disabled: value=1
dev.igb.3.eee_disabled: value=1

IPv4 Fragments - 0=Do not accept fragments
This is mainly need for security. Fragmentation can be used to evade packet inspection
net.inet.ip.maxfragpackets: value=0
net.inet.ip.maxfragsperpacket: value=0

Set to 0 (<x>) for every port used by IPS
dev.igb.<x>.fc: value=0

##UPDATE 1/16/2018##
Although the tuning in this thread so far just deals with the tunables, there are other settings that can impact IPS performance. Here are a few...

In the Intrusion Detection Settings Tab.

Promiscuous mode- To be used only when multiple interfaces or VLAN's are selected in the Interfaces setting.
This is used so that IPS will capture data on all the selected interfaces. Do not enable if you have just one interface selected. It will help with performance.

Pattern matcher: This setting can select the best  algorithm to use when pattern matching. This setting is best set by testing. Hyperscan seems to work well with Intel NIC's. Try different ones and test the bandwidth with an internet speed test.

Home networks (under advanced menu.
Make sure the interfaces fall within the actual local networks. You may want to change the generic 192.168.0.0/16 to your actual local network ie 192.168.1.1/24

###################################################
USEFUL SHELL COMMANDS
sysctl net.inet.tcp.hostcache.list # View the current host cache stats
vmstat -i # Query total interrupts per queue
top -H -S # Watch CPU usage
dmesg | grep -i msi # Verify MSI-X is being used by the NIC
netstat -ihw 1 # Look for idrops to determine hw.igb.txd and rxd
grep <interface> /var/run/dmesg.boot # Shows useful info like netmap queue/slots
sysctl -A # Shows system variables
###################################################