Fiber finally arrived in my neighborhood! I've been using opnsense for years as a VM in my vmware homelab with both LAN and WAN as vmxnet adapters - never noticed any problems with slow DSL. Now however, i'm seeing that my throughput is being limited to around 150mbps. If I plug my d-link consumer grade router in i can get gig speeds, but thru opnsense i can only get the 150 or so. Is that to be expected, is there anything to do? Currently everything going to my hypervisors is over 40gb trunked links between my ICX 6610 switch, but each machine has a couple of extra gigabit links on it that i suppose I could passthru directly to the VM to speed things up - but i'm not a fan of pass thru though because sometimes it is flaky and more importantly, it precludes using vmotion to live migrate the VM from one hypervisor to another when doing maintenance, which is important for a router.
May be time to go for a standalone opnsense appliance - what's best for routing symmetric gig these days?
Have you tried replacing vmxnet with e1000? In the TrueNAS community there is consensus that this is the more reliable and better performing driver.
I don't have such complex setup as you, but this post helped me:
https://forum.opnsense.org/index.php?topic=18754.msg90576#msg90576 (https://forum.opnsense.org/index.php?topic=18754.msg90576#msg90576)
Update:
Here for example the topic w.r.t. VMXNET tuning that confirms it works https://forum.opnsense.org/index.php?topic=25076.0 (https://forum.opnsense.org/index.php?topic=25076.0) .
And my test VM shows 10Gbps for 8 vCPU (Xeon @2.4GHz) and 'iperf3 -P 5 -t60' and following tunables:
net.isr.maxthreads = -1
net.isr.bindthreads = 1
hw.pci.honor_msi_blacklist = 0
hw.ibrs_disable = 1
dev.vmx.0.iflib.override_ntxds="0,4096"
dev.vmx.0.iflib.override_nrxds="0,2048,0"
dev.vmx.1.iflib.override_ntxds="0,4096"
dev.vmx.1.iflib.override_nrxds="0,2048,0"
T.