Internet speeds reduced set to 2.5gb

[manki_09]

I have an issue when I have my WAN connection set to 2.5gb link speed to my Xfinity XB8 gateway, my download speeds are limited to around 300-350mbps. When I change my link speed on the WAN side back to 1gb I get around 940mbps (max 1gb link will do).

See attached image for network diagram.

Some additional information
This occurs on all 3 VLANs connected to my switch.
IPerf3 from a LAN 10gb interface to OPNSense easily gets over 9000 mbps.
Running a server hosting openspeedtest and iperf3, all PCs get their correct speed for their link speed across all vlans.

WAN set to 1gb
Running speedtest from OPNSense CLI gets around 1500 mbps when link is set to 2.5gbps, but get very high latency (log attached).
Running speedtest from 10gb and 1gb PCs get 940mpbs

WAN set to 10gb
Running speedtest OPNSense CLI at 1gb link I get 940mbps
Running speedtest from 10gb PCs get 1800mbps+
Running speedtest from 1gb PCs get 300-350mbps

I've tried just about everything on the Aruba switch to see if was doing something funky but nothing improved or depreciated the performance.
I've also tried other internet speedtest services like fast.com and I get the same results.
Cables used are CAT6 and TwinAx. All cables are shorter than 1m, with Gateway to Router being 6 inches. All manufactured cables.

Any ideas would be great.

P.S.
Forgot to mention. I haven't seen CPU usage above ~15% on OPNSense and htop shows multiple cores being used during these speedtests with none close to peaking out. The system is bare metal running an Intel i5-10600t and 16gb on RAM.

[meyergru]

Did you try RSS?

[manki_09]

Did you try RSS?

I thought I had already done that. Some of the tunable items were complete except net.inet.rss.enabled = 1. I did change that and rebooted. However, my results have not changed.

I do have a USB 3 2.5gb adapter coming today I'm going to tryout incase the NIC is causing something. I use 1gb versions of these adapters for work all the time so no harm buying another one.

[passeri]

You are shaping the WAN-side speed? What was the preceding position, when using auto-negotiation?

[manki_09]

You are shaping the WAN-side speed? What was the preceding position, when using auto-negotiation?

I currently have shaping turned off. I tried shaping as a troubleshooting step to limit the speed to 1gb but nothing changed.

The intel x550 NICs will not auto negotiate to 2.5gbps. Which is programmed into the firmware. Manual selection is required. This is why I have a 2.5gb usb nic order so I can test if the NIC is at fault.

[passeri]

I currently have shaping turned off. I tried shaping as a troubleshooting step to limit the speed to 1gb but nothing changed.

The intel x550 NICs will not auto negotiate to 2.5gbps. Which is programmed into the firmware. Manual selection is required. This is why I have a 2.5gb usb nic order so I can test if the NIC is at fault.

I see. You mean like this comment which I found on the Intel site here?

The autonegotiation for 2.5 and 5Gb speeds for the X550 was changed in 2020.

Default autonegotiation excludes the 2.5 and 5Gb speeds.

If 2.5 or 5Gb is chosen in the dropdown, it will change autonegotiation to only advertise that speed. So it is not forcing to 2.5Gb or 5Gb when those options are chosen, it changes the advertised speed.

That may be an issue if the switch is configured as forced to 2.5Gb instead of autonegotiate.

If that still does not help, please make sure the ethernet updated to the latest NVM and drivers.

This comment and the prior discussion on the Intel site imply to me that the problem may lie with NIC configuration rather than with Opnsense config. Your proposed test may be informative ("may" because I lack complete confidence in USB-Ethernet adapters even though I sometimes use them in testing).