ESXI/vSphere: Slow iPerf3 performance with more than 1 core

[ohne]

Can anybody give me a hint why on ESXI, but also on Proxmox, iPerf3 speeds are getting more than halved, when using more than one core. Everybody who is facing slow VM(opnSense) - VM/Host performance should give it a try and use only one core for the opnSense VM. This nearly doubles the performance.

Is there a reason?
Host is a i7-8700 with 32GB RAM and around 10 VMs.
opnSense VM got, well 1 core and 2GB RAM
NIC: 1x e1000
ZFS
Tunables + loader.conf.local:

vm.pmap.pti="0"
hw.ibrs_disable="1"
net.inet.tcp.tso="0"

Results with 1 core: 5,88 Gbit's
Results with 4 cores: 1,78 Gbit's

[Georges]

Hello,

I found some performance issue too.
I'm on OPNsense 21.10.2-amd64

Results with 1 core: 0.365 Gbit's
Results with 4 cores: 0.327 Gbit's

And without the modification

Results with 1 core: 0.142 Gbit's
Results with 4 cores: 0.125 Gbit's

[Georges]

I have done the test with ipfire i got 750Mbps/s and opnsense 350 Mbps/s

same Esxi 6.7
ram 8go
vCPU 4
NVMe