Performance troubleshooting - 1 core used out of 2

[JayST]

i'm looking into a performance issue with  OPNsense 21.7.7 running on a VMware VM on a relatively powerfull Dell R640 server. The VM uses vmxnet3 virtual adapters.
the OPNsense VM running with 2 cores and CPU core 2 maxes out on 400Mbit WAN throughput.
I'm monitoring (SNMP) the CPU usage during this load, and it seems one of the 2 cores (CPU2) assigned to the VM is maxed out and the other is idling away.
I'd really appreciate some help / pointers about where to look next to find out how i can determine where the bottleneck is and what can/cant be done about it.

Some questions i have:
How would i best monitor this to determine the bottlneck?
Would upgrading the VM with more cores help? Perhaps 2 is too low to multithread?

thanks in advance.

some more information.

Code: [Select]

# vmstat -i
interrupt                          total       rate
irq1: atkbd0                           2          0
irq18: uhci0                     2247772          2
cpu0:timer                     137996878        107
cpu1:timer                     198828337        154
irq256: ahci0                     858812          1
irq258: mpt0                     9736309          8
irq260: vmx0:irq0             1148453361        891
irq268: vmx1:irq0             1803966729       1400
irq277: vmx2:irq0              162289982        126
irq286: vmx3:irq0              823455667        639
Total                         4287833849       3328
#

Edit:
I found some tunables of possible interest

net.isr.maxthreads="-1"
net.isr.bindthreads="1"
hw.pci.honor_msi_blacklist="0"
I'm not sure if i would need to set all 3 of these or just the last one. Also the first two are mentioned together wit the need for setting net.isr.dispatch , but unclear for me if that only counts when the first maxthreads tunable would actually be set to a positive integer.

and then possible playing arround with values for:
hw.vmx.txnqueue (to be equal to the core-count?)
hw.vmx.rxnqueue (to be equal to the core count?)
hw.vmx.txndesc (not sure)
hw.vmx.rxndesc (not sure)

edit2:
below the vmstat output after setting
net.isr.maxthreads="-1"
net.isr.bindthreads="1"
hw.pci.honor_msi_blacklist="0"

AND rebooting. Had to do the reboot to see the change happen.

Code: [Select]

# vmstat -i
interrupt                          total       rate
irq1: atkbd0                           2          0
irq18: uhci0                        1370          2
cpu0:timer                         94246        129
cpu1:timer                        102043        140
irq256: ahci0                        524          1
irq258: mpt0                       62743         86
irq260: vmx0:rxq0                  22699         31
irq261: vmx0:rxq1                  49248         67
irq270: vmx1:rxq0                  15597         21
irq271: vmx1:rxq1                  26109         36
irq281: vmx2:rxq0                  58672         80
irq282: vmx2:rxq1                  11467         16
irq292: vmx3:rxq0                   5856          8
irq293: vmx3:rxq1                   9690         13
Total                             460266        630



edit3:
so after adding all three tunables above (and reboot), both virtual cores were used more even. But under load (more sessions), the firewall would not respond well and was overloaded. 2 cores was not enough. I increased the core count to 6 and that gave it room to keep breathing AND reach throughput over 1Gbit/s instead of 400Mbit

I think i'm OK now with this, but i'm still unsure if i need these two tunables in addition to the hw.pci.honor_msi_blacklist="0" :
net.isr.maxthreads="-1"
net.isr.bindthreads="1"

comments welcome