Internet speeds reduced set to 2.5gb

[manki_09]

I have an issue when I have my WAN connection set to 2.5gb link speed to my Xfinity XB8 gateway, my download speeds are limited to around 300-350mbps. When I change my link speed on the WAN side back to 1gb I get around 940mbps (max 1gb link will do).

See attached image for network diagram.

Some additional information
This occurs on all 3 VLANs connected to my switch.
IPerf3 from a LAN 10gb interface to OPNSense easily gets over 9000 mbps.
Running a server hosting openspeedtest and iperf3, all PCs get their correct speed for their link speed across all vlans.

WAN set to 1gb
Running speedtest from OPNSense CLI gets around 1500 mbps when link is set to 2.5gbps, but get very high latency (log attached).
Running speedtest from 10gb and 1gb PCs get 940mpbs

WAN set to 10gb
Running speedtest OPNSense CLI at 1gb link I get 940mbps
Running speedtest from 10gb PCs get 1800mbps+
Running speedtest from 1gb PCs get 300-350mbps

I've tried just about everything on the Aruba switch to see if was doing something funky but nothing improved or depreciated the performance.
I've also tried other internet speedtest services like fast.com and I get the same results.
Cables used are CAT6 and TwinAx. All cables are shorter than 1m, with Gateway to Router being 6 inches. All manufactured cables.

Any ideas would be great.

P.S.
Forgot to mention. I haven't seen CPU usage above ~15% on OPNSense and htop shows multiple cores being used during these speedtests with none close to peaking out. The system is bare metal running an Intel i5-10600t and 16gb on RAM.

[meyergru]

Did you try RSS?

[manki_09]

Did you try RSS?

I thought I had already done that. Some of the tunable items were complete except net.inet.rss.enabled = 1. I did change that and rebooted. However, my results have not changed.

I do have a USB 3 2.5gb adapter coming today I'm going to tryout incase the NIC is causing something. I use 1gb versions of these adapters for work all the time so no harm buying another one.

[passeri]

You are shaping the WAN-side speed? What was the preceding position, when using auto-negotiation?

[manki_09]

You are shaping the WAN-side speed? What was the preceding position, when using auto-negotiation?

I currently have shaping turned off. I tried shaping as a troubleshooting step to limit the speed to 1gb but nothing changed.

The intel x550 NICs will not auto negotiate to 2.5gbps. Which is programmed into the firmware. Manual selection is required. This is why I have a 2.5gb usb nic order so I can test if the NIC is at fault.