IPS and throughput performance

[pubare]

Sorry, I know there are a lof of threads related to this.  I'm new to opnsense, but have done a fair bit of research and looking through the forum threads as well.
I have a Protectli FW6B with an Intel  i3-7100U (2-core / HT, 2.4GHz) with 16GB of RAM and (6) Intel 82583V (I210) NICs.  Internet pipe is 300Mbps fiber-to-the-home.  Opnsense initial install was 20.7, now at 21.1.2.

With IPS enabled and only using a single-ruleset (emerging web client, 859 rules) online speedtests show ~190Mbps down / ~290Mbps up.  I thought it might be related to the Intel NICs, as there are a lot of comments about them, so I went through all the standard "tunables" and made certain the em and dev.em.X.iflib settings werre the "recommended" for 1Gb Intel NICs.  This made no difference what-so-ever.

Disabling IPS, online speedtests immediately change to ~580Mbps down / ~290MBps up (fyi, this is a little better than the results I got with my old consumer router - which I would expect to be the case).  This is fully repeatable.  There also doesn't appear to be ANY change in CPU utilization / Memory consumption during the tests with IPS on or off...  I've also tried with different rulesets (malware, ciarmy, botcc.portgroup) and get identical results.

To me this doesn't seem like expected behavior and even though it's only an I3 it "should" be able to keep up (based on experience with commericial firewalls that have UTM).  It seems very strange to me as well that the CPU utilization doesn't appear to change with IPS enabled.  But again, I am new to opnsense, and it has been quite a while since I've played with *nix firewalls / routers of any type.  Are my expectations off?  Or am I missing something?

em0 is the WAN port, em1 (LAN) configured identically.  Main references for settings were various forum posts and https://calomel.org/freebsd_network_tuning.html
 ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=802008<VLAN_MTU,WOL_MAGIC>
        media: Ethernet autoselect (1000baseT <full-duplex>)

sysctl hw.em
hw.em.max_interrupt_rate: 8000
hw.em.eee_setting: 1 (per man this is the disabled value)
hw.em.rx_process_limit: 1000 (default is 100, without MSIX hesitant to try "-1")
hw.em.smart_pwr_down: 0

 sysctl dev.em.0
dev.em.0.eee_control: 1 (per man this is the disabled value)
dev.em.0.fc: 0
dev.em.0.iflib.disable_msix: 1 (this appears to be default, several reports of traffic failure with MSIX enabled)

dmesg | grep em0
em0: <Intel(R) PRO/1000 Network Connection> port 0x2000-0x201f mem 0x7e400000-0x7e41ffff,0x7e420000-0x7e423fff irq 16 at device 0.0 on pci1
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:e0:67:21:c4:36
em0: netmap queues/slots: TX 1/1024, RX 1/1024

sysctl kern.ipc
kern.ipc.maxsockbuf: 16777216
kern.ipc.nmbclusters: 492680

sysctl net.inet.tcp.tso
net.inet.tcp.tso: 0

Edit: forgot to include the changed entropy pool (shouldn't matter at less than 10Gbps anyway)
sysctl kern.random
kern.random.harvest.mask_symbolic: PURE_RDRAND,[UMA],[FS_ATIME],SWI,[INTERRUPT],NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
kern.random.harvest.mask: 65887

[LOTRouter]

I have the exact same Protectli FW6B (i3-7100U) with 8G RAM running Suricata with all rules enabled and on my Xfinity gigabit service I get 1.2G down 35M up.  All this without tweaking any NIC settings.

[pubare]

LOTRouter, out of curiousity - is your unit using CoreBios?

[pubare]

Very strange...  I was about to flash back from coreboot to the standard AMI BIOS as a (very) out-side possibility, decided to test a few other things first.

1) I'm using a firewall group for "Inside Networks"
2) I have the main LAN interface assigned as a fail-over LAG (POS Netgear smart-switch can't do LACP and doesn't behave properly with static or LB modes - wish I would have thought to look for EOS / EOL cisco switches on amazon / e-bay)

I dropped the firewall group after moving the rules to the individual interfaces - and the problem largely disappeared.  Used the same testing methodolgy as before (enable/disable rule-set, download and update rules, restart service, go to http://www.dslreports.com/speedtest) and definitely getting different results without the firewall group.

Rulesets for botcc.portgroupd, ciarmy, emerging-malware, and emerging-mobile_malware now have no impact on throughput.  Attempting to use emerging-web_client still tanks throughput though - guessing that is a ruleset issue.  I don't want to tear-down lagg0 just to check this rule-set as it is a PITA to reassign everything, wish there was a simple way to "reassign" an interface (but get why there isn't).

This doesn't make much sense to me, other than a weird possibility that ipfw has some sort of "translation" issue talking to pf if a group is in use.  Can't sort out in my head how that would happen though...

Between the Unbound DNS SBLs, a firewall drop alias for https://sslbl.abuse.ch/blacklist, http://rules.emergingthreats.net/blockrules/compromised-ips.txt, http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt, and the above IPS rulesets that do work I'm pretty happy with the additional level of protection.

Thank you to the devs for an excellent product.

[LOTRouter]

LOTRouter, out of curiousity - is your unit using CoreBios?

Yes it is

[QuaCKeReD]

Attempting to use emerging-web_client still tanks throughput though - guessing that is a ruleset issue. 

Wow, I had same issue with throughput (1Gbps line with 146Mbps download). Tried removing this rule and instantly shot up to 934Mbps down! Good spot, thank you!

Between the Unbound DNS SBLs, a firewall drop alias for https://sslbl.abuse.ch/blacklist, http://rules.emergingthreats.net/blockrules/compromised-ips.txt, http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt

And thanks for this too, I had the DNS blocks, but the lists are useful too 😀

EDIT: aren't those lists already in the ETPro Telemetry rulesets?
EDIT2: speed dropped shortly afterwards, so not necessarily down to that for me 😟

[QuaCKeReD]

Found https://forum.opnsense.org/index.php?topic=6930.msg44740#msg44740 in my travels.

It does seem to cover the lists you provided, and more, in a smaller number of entries!