Virtualized OpnSense Best Practices to Improve Performance w/Multi-Gig Service?

[theprez1980]

Hey All -

Had no problem with performance until I upgraded to 5G fiber - now I'm lucky to get 3G in both directions.  Directly connecting a PC to the ISP router shows approx 5100/5100 so there seems to be a configuration issue or bottleneck somewhere in my setup.   

Here's my setup:

Dell PowerEdge T620 with Dual CPU (E5-2643) with 512GB ECC RAM running Proxmox.
Intel X550-T2 passed thru directly to OpnSense for WAN connection to ISP's 5GB port - link status confirmed to be 5000
Intel X540-T2 passed thru directly to OpnSense for LAN connection to Juniper 48 port EX3300 switch- link confirmed to be 10000

In Proxmox:
OpnSense has been allocated 8 cores, and has the highest CPU priority with 16GB dedicated RAM
The CPU is set to Host
The CPU setting also passes the AES instruction set to the VM

In OpnSense:
Hardware offloading is disabled (boxes are checked)
In Tunables - disabled the Spectre and Meltdown mitigation are disabled and system was rebooted

What else am I missing here?   There's no fancy rules yet defined, no VLANs - just a vanilla setup with WAN DHCP, and LAN DHCP with OpnSense providing DHCP services currently.

Thanks

[fgerardi]

Hi,
I had a similar issue few months ago. In my case, I upgraded wan connection from 1GB to 2.5GB.
My proxmox server is connected to a switch at 10GB.

I solved my problem enabling flow control on the switch ports involved.
In your case I suggest to check and try to enable flow control on both Intel network devices (via some tunables?).

In my server I don't use passthrough, I use virtio driver combined with Multiqueue configuration.

I hope this could help.

Best regards