I have an issue when I have my WAN connection set to 2.5gb link speed to my Xfinity XB8 gateway, my download speeds are limited to around 300-350mbps. When I change my link speed on the WAN side back to 1gb I get around 940mbps (max 1gb link will do).
See attached image for network diagram.
Some additional information
This occurs on all 3 VLANs connected to my switch.
IPerf3 from a LAN 10gb interface to OPNSense easily gets over 9000 mbps.
Running a server hosting openspeedtest and iperf3, all PCs get their correct speed for their link speed across all vlans.
WAN set to 1gb
Running speedtest from OPNSense CLI gets around 1500 mbps when link is set to 2.5gbps, but get very high latency (log attached).
Running speedtest from 10gb and 1gb PCs get 940mpbs
WAN set to 10gb
Running speedtest OPNSense CLI at 1gb link I get 940mbps
Running speedtest from 10gb PCs get 1800mbps+
Running speedtest from 1gb PCs get 300-350mbps
I've tried just about everything on the Aruba switch to see if was doing something funky but nothing improved or depreciated the performance.
I've also tried other internet speedtest services like fast.com and I get the same results.
Cables used are CAT6 and TwinAx. All cables are shorter than 1m, with Gateway to Router being 6 inches. All manufactured cables.
Any ideas would be great.
P.S.
Forgot to mention. I haven't seen CPU usage above ~15% on OPNSense and htop shows multiple cores being used during these speedtests with none close to peaking out. The system is bare metal running an Intel i5-10600t and 16gb on RAM.
Did you try RSS (https://docs.opnsense.org/troubleshooting/performance.html)?