Performance issue with xeon D-2733NT

[c0nnected]

Hi,
Starting with a little background: OPNsense is running as VM on proxmox, cpu 8threads, model set as host and all NICs (i350 (igb), x550(ix), and e823-c (ice (thats 25gb sfp28 ports)) are passed as PCI device so no virtualization on them. To the opnsense there is one more network device connected vtnet0 (linux bridge set in VirtIO).
My problem is when I set CPU governor for proxmox to power save mode I'm unable to cross 1.2Gb/s on iperf3 between ix port and vtenet despite cpu is around 20-25% of usage (it's purely routing only, no fw, no ips etc.), however when I set CPU governor to performance I'm able to max out my 2.5Gb/s desktop port and also CPU usage is lower(?) around 10%. I tried few different settings in tunatable unfortunately without any luck, is there any reason why 8 threads of modern cpu in powersave mode (it still draws around 35w at idle) is unable to achieve more than 1.2Gb/s? Also why would cpu usage drop despite pushing more data in performance mode?

[yourfriendarmando]

In earlier days of virtualization, I would leave any governors off. These days, I leave my Linux-based hosts on their default governor of OnDemand. In guests, I leave them on the same or equivalent (Windows: Balanced, FreeBSD: Adaptive).

On all the bare metal I set up for myself and clients, I set to HiAdaptive, although Adaptive seems to perform just as well. I do prefer it keeps the CPU on the ready to reduce latency.

You're running a server with high end SFP28 ports, thus power saving isn't your priority. I think for 10Gb and faster PHYs, your priorities are to keep the transceivers cool to prevent them from slowing down to protect themselves. Try different fiber, make sure they are not snagged, kinked, nicked, or tightly wound. You want to avoid reflection and refraction. Even the polishing of the fiber tips could matter at 25Gb and above. Try some single mode fiber with an APC finish with your chosen connector.

[134]

In my test i had Opnsense VM within promox running on i3-13100 and X710-DA2. Inter-VLAN routing traffic reached 18-19 Gbps, while raw speed between VMs in same VLAN reached 29Gbps. Your CPU doesn't have same single-thread performance but it should do much better job than 1.2Gbps.

I used SR-IOV passthrough, VMs and Opnsense each had its own VF. Traffic between VFs aren't constrained by PHY speed of 10Gbps. All your NICs are capable of SR-IOV so this is recommended way instead of whole NIC passthrough or VirtIO.

I'd like to read more about your detailed setup, switch, VLANs ... Something must be wrong.