Messages

try PCIe passthru for the NIC - my guess is you've got issues somewhere in the unraid emulated NIC implementation

the bigger issue is the increased availability of 2.5gbs and higher connections - it's very hard to find a cpu with sufficient single-core speed to handle those kind of speeds in zenarmor - you generally end up with a bottleneck

VLAN tagged ppoe is common for ftth installs in NZ/Aus/UK and parts of Europe - lots of people ( myself ) included are happily running opnsense on those connections

one thing to check is whether your ISP allows 'third party' routers - a few look at the MAC of the device connecting and reject it if it's not their device ( thatr's where MAC cloning comes in).

Other thing worth trying if you have a managed switch is to set the switch to handle the tagging/untagging to the ISP hardware and present an untagged WAN feed to your opnsense appliance. Shouldn't matter - but it can eliminate one possible issue ( incorrect tagging setup in opnsense )

can you switch out the 82574 for something newer?  I had stability issues with the em driver and an 82576 - went away going to something which is more current ( igb, ixgbe etc )

found a block diagram for a 'typical' h77/z77 board - the 2 pci slots are off a pci/pcie bridge chip and the upstream link is pcie3x1 - that should be heaps for 4 x gbe links ( presuming the bridge chip isn't utter rubbish )

I'd check core loading on the firewall when running a speed test from a lan client to wan and see if you're saturating a single core ( ssh in to opnsense, run 'top -P' and watch the 'interrupt and 'idle' column ).

I dug up the pro/1000 mt dual specs pdf - no mention of rss on it at all that I can see, freedBSD used to have a separate 'emx' driver with rss support for some of those older cards but it doesn't list the 82546, just newer variants ( https://man.dragonflybsd.org/?command=emx&section=4 ). Suspect you need a new NIC

PCI is very bandwidth constrained compared to PCIe - and with 4 gbe nics on PCI you're almost certainly running into those constraints

I'd replace those old pro1000 cards with either a couple of iintel i21x series cards (if you only need a couple of ports ) or if you really need 4 ports something like an i350-t4 - again well supported NICs for freeBSD.

you'll also probably want to look at enabling RSS once you've got the cards replaced - the i7-3770 has pretty low single-core perf by modern standards and without RSS you can end up limited by that 

https://forum.opnsense.org/index.php?topic=24409.0

Hi,

With regards to Zenarmor, we have identified two steps that can help improve its performance. Firstly, we will be implementing a feature that pins a random core to the system. Currently, it is pinned to CPU 1 when you check the box. Secondly, we are working on providing multicore support to the engine, which is expected to be shipped in November."

Let me know if you need any further assistance.

the original req for the 'do not pin' option ( which was made by me ) was to increase perf on modern multi-core cpus ( and espec under esxi ) - and up to the latest release it did the 'right' thing ( didn't pin to any core, let the scheduler assign the process at will )

there was a regression in the new version where 'do not pin' stopped doing that and instead right now pins the eastpect process to cpu0 ( which is an even WORSE option than the default of pining cpu1 if you haven't got rss enabled as cpu0 tends to get hammered by interrupts )

Again I have an open req to revert to the previous behaviour open - how come your devs are instead deciding 'pin to a random cpu' is a better fix ( rule #1 - don't fight the kernel's scheudler - it generally knows more than you )?

Anyway - very much looking forward to full multicore support

for the op - if yout want to revert to previous ( and what I consider 'correct' behaviour) save the script below and create a cron job to run it every 10 or 30mins ( eastpect doesn't get restarted often ).

Also consider turning on RSS as it definitely helps performance in vm's with a lot of vcpus assigned

#!/bin/sh

eastpect_instance_0=`ps -ax | awk '$5 ~ "eastpect" && $0 ~ "Instance 0"  { print $1 }'`

echo "Eastspect Instance 0 pid=" $eastpect_instance_0

#echo "current cpu affinity"
cpuset -g -p $eastpect_instance_0

# change affinity to all cores
cpuset -l ALL -p $eastpect_instance_0

#echo "new cpu affinity"
cpuset -g -p $eastpect_instance_0

exit 0

logs etc sent

note that I re-enabled the cron job I used to have prior to the GUI option to un-pin, so if the logs include the current cpuset output for eastspect it'll show it using 0-7 due to the cron job 'un-pinning' the process - without the cron job easpect pins to cpu 0 if 'do not pin' is selected, and defaults back to cpu 1 if 'do not pin' is unticked

despite having the option 'do not pin engine packet processors to dedicated CPU cores' selected I can see that eastpect is running ONLY on cpu 0 ( and watching core loading with htop whilst stress testing confirms this )

cpuset -g on the easptect pid gives this :

Eastspect Instance 0 pid= 61127
current cpu affinity
pid 61127 mask: 0
pid 61127 domain policy: first-touch mask: 0

if I manually set the cpu mask to all cores using cpuset I see what I would expect and much more even cpu utilisation at high loads

Eastspect Instance 0 pid= 61127
current cpu affinity
pid 61127 mask: 0, 1, 2, 3, 4, 5, 6, 7
pid 61127 domain policy: first-touch mask: 0

If I un-tick 'do not pin' it sets affinity to cpu 1 as previously, so looks like in the new version there's been some sort of regression in the 'do not pin' code ( ticking it now basically pins the eastpect process to cpu 0 - which is obviously NOT what is intended )

...next time buy an optiplex SFF and you can add PCI NICs, problem 100% solved ;-)

Is there a recommendation for a micro or USFF device than can take a single or dual nic?

Intel do NUC models with dual lan ports ( latest ones are dual 2.5gbe ) - not as cheap as the topton etal boxes, but reliable, have proper support with BIOS updates and are rated for 24/7 operation by Intel

if it's a sg330v2 it's got an i5-6500 in it - which is easily double the single core perf of the xeon 2420 ( whilst puilling a fraction of the power ) - plus of course it's got faster RAM which will again help ( ddr4 vs ddr3 )

first thing I'd try is a different known good NIC

Also the pro/1000s are pretty ancient - ideally you want something newish and with good driver support ( pair of i210/i219s or maybe an i350-t2/t4 or even an x540-t2/x550-t2)

presumably those are 2420 (v0) in the g8 - like most of those old xeons that have terrible single-core perf, and zenarmor is currently single threaded - you'd need something with about double the single core perf to not bottleneck at <1gbps ( if you want to run zenarmor at full 10gbps that's going to be an even bigger problem - until it's multi-threaded I finding a cpu with high enough single core perf is an issue )

first thing to try would be running opnsense bare-metal rather than virtualised - it's the only way to tell if the issue is the hypervisor and your use of vmnics, or you're actually running into hardware limits ( I presume you've already watched cpu load whilst running tests to check you're not bottlenecking on a single core)