Throughput drops massively with option selected (even in bypass)

[Ishuk0ne]

Not sure if this is a bug or not but I thought I'd highlight this as I was getting massive throughput problems and was working with my ISP to resolve it as I had disabled Zenarmor and still had issues so assumed it wasn't my config. Found this was the issue completely by accident.

If you tick the box that says "Do not pin engine packet processors to dedicated CPU cores" then I found gigabit throughput went from 1000mbit a sec to 250 - 300mbit.

Putting the packet engine into bypass makes no difference, which I found strange.
Untick the option and save and full gigabit passes through again.

OPNSense config (Runs as a VM in ESXi 8.0 ):
6x vCores
12GB RAM
100GB Storage allocated
WAN - dedicated gigabit
LAN - LAGG consisting of 4 gigabit connections
Several VLANs with LAN as parent interface

Hardware config:
Asus Z97-Pro
Intel Core i7-4790K
Corsair 4x 8GB DDR3
Onboard Intel Gigabit NIC
2x Dual port Intel Gigabit NIC PCIe cards
Samsung 840 SSD's for storage

I am fine with the option off, I thought spreading the load would improve performance but hey :)

Ish

[meyergru]

Sometimes, depending on the design of drivers, hardware and subsystems, coordinating / locking between threads (or CPU cores) can be more complex than to use one of them to work on the packets sequentially.

Context switching is a heavy process when it is applied to gigabit connections at rates of several hundreds or thousands of pps - and you use 4 of them in parallel. This is even more true when that context switching must be emulated in part because of virtualisation. BTW: Did you enable all virtualisation enhancements in your BIOS?

[sy]

Hi,

With regards to Zenarmor, we have identified two steps that can help improve its performance. Firstly, we will be implementing a feature that pins a random core to the system. Currently, it is pinned to CPU 1 when you check the box. Secondly, we are working on providing multicore support to the engine, which is expected to be shipped in November."

Let me know if you need any further assistance.

[johndchch]

Hi,

With regards to Zenarmor, we have identified two steps that can help improve its performance. Firstly, we will be implementing a feature that pins a random core to the system. Currently, it is pinned to CPU 1 when you check the box. Secondly, we are working on providing multicore support to the engine, which is expected to be shipped in November."

Let me know if you need any further assistance.

the original req for the 'do not pin' option ( which was made by me ) was to increase perf on modern multi-core cpus ( and espec under esxi ) - and up to the latest release it did the 'right' thing ( didn't pin to any core, let the scheduler assign the process at will )

there was a regression in the new version where 'do not pin' stopped doing that and instead right now pins the eastpect process to cpu0 ( which is an even WORSE option than the default of pining cpu1 if you haven't got rss enabled as cpu0 tends to get hammered by interrupts )

Again I have an open req to revert to the previous behaviour open - how come your devs are instead deciding 'pin to a random cpu' is a better fix ( rule #1 - don't fight the kernel's scheudler - it generally knows more than you )?

Anyway - very much looking forward to full multicore support

for the op - if yout want to revert to previous ( and what I consider 'correct' behaviour) save the script below and create a cron job to run it every 10 or 30mins ( eastpect doesn't get restarted often ).

Also consider turning on RSS as it definitely helps performance in vm's with a lot of vcpus assigned

#!/bin/sh

eastpect_instance_0=`ps -ax | awk '$5 ~ "eastpect" && $0 ~ "Instance 0"  { print $1 }'`

echo "Eastspect Instance 0 pid=" $eastpect_instance_0

#echo "current cpu affinity"
cpuset -g -p $eastpect_instance_0

# change affinity to all cores
cpuset -l ALL -p $eastpect_instance_0

#echo "new cpu affinity"
cpuset -g -p $eastpect_instance_0

exit 0

[Ishuk0ne]

How do you switch on RSS?

[johndchch]

https://forum.opnsense.org/index.php?topic=24409.0

[Ishuk0ne]

So they haven't just made a radio button to switch it on and off yet then?