Messages

I don't think it's an issue per say with shaping, but the fact that the switch didn't actually apply flow control when the interface was enabled.
Then changing some settings in shaping (enabling and disabling) seemed to have forced flow control to enable even though it should have been already.

Well I solved the issue. It believe OPNsense was not the issue.
Flow Control on the Aruba switch wasn't working even though it was enabled. I was able to do some more testing with IPerf and found that when sending data from a higher link.
Did testing between 10gb links and 1 gig links and found the same 300mbps when sending data from the 10gb links to 1gb but fine in the opposite direction.

I think there was a glitch in the Aruba Switch (3.3.2) that seemed to not apply flow control. I finally got it to work after testing Aruba shaping setting and when removed the shaping settings it seems to work just fine. Speeds on 1gb links are good and also good on 2.5 and 10gb links.

So, I did a little more testing and replaced the Xfinity XB8 gateway with a Hitron CODA56. I'm not a fan of running ISP equipment especially Xfinity.

Before I swapped the modem out I tried using the 2.5gb USB adapter for the WAN interface. Which yielded the same results.

I continued with the modem swap and connected a laptop with built in 2.5gb nic and saw the full 2gbps/300mbps plus some. I rebooted the modem and connected it up to OPNSense as it was all originally connected.... unfortunately the symptoms remain unchanged.

My next planned test will be to use the 2.5gbe usb adapter on the OPNSense router to bypass the rest of my network and connect my 2.5gb enabled laptop directly to it.

FWIW, I have an XB7 gateway connected to my OPNsense via a 2.5GbE link.  I'm not seeing this, but IIRC it was a little sensitive to which modem port I used.  Have you tried switching the XB8 port?

Yes I did attempt to switch. The others are only 1GB

This is a stab in the dark, but trying turning EEE off on OPNSense.

In the UK, we had issues with the 2.5G port on the DOCSIS 3.1 modem with specific firmware which were related to Energy Efficient Ethernet. We've since had a firmware upgrade on the modem to fix it, but hey...worth a try?

Looks like Intel's X550 chips don't support EEE. Good idea though.

[Modem]

I've done a bit more testing that could point towards the Xfinity Gateway (in bridge mode) being the issue. I have a cheap managed switch with 2 SFP+ ports and 4x 2.5 gb ports. Put in between the Router and the Gateway, this gives me the ability to change port speeds on each device.

Here are my test results after running a speedtest a few times. I cycled through the ports a couple time to ensure nothing changed.

Modem	Router	Speed Result
2.5GB	10GB	~325mbps
2.5GB	2.5GB	~325mpbs
2.5GB	1GB	~650mpbs
1GB	10GB	940mpbs
1GB	2.5GB	940mpbs
1GB	1GB	940mpbs

It seems to run pretty consistent when the modem's link is set to 1GB at 900-940mpbs as a 1GB link should. 

However, when the modem's link is set to 2.5GB I get pretty inconsistent speeds, sometimes better than the speeds above but usually only 50-100mbps better. Never reaching within a 1GB link should be.


I have not tested my 2.5GB USB adapter yet since I have services being hosted on this and requires a gateway reboot to connect a new device/MAC. I may try it out tomorrow morning, but for now things seem to point towards the modem.

I currently have shaping turned off. I tried shaping as a troubleshooting step to limit the speed to 1gb but nothing changed.
The intel x550 NICs will not auto negotiate to 2.5gbps. Which is programmed into the firmware. Manual selection is required. This is why I have a 2.5gb usb nic order so I can test if the NIC is at fault.

I see. You mean like this comment which I found on the Intel site here?

The autonegotiation for 2.5 and 5Gb speeds for the X550 was changed in 2020.
Default autonegotiation excludes the 2.5 and 5Gb speeds.
If 2.5 or 5Gb is chosen in the dropdown, it will change autonegotiation to only advertise that speed. So it is not forcing to 2.5Gb or 5Gb when those options are chosen, it changes the advertised speed.
That may be an issue if the switch is configured as forced to 2.5Gb instead of autonegotiate.
If that still does not help, please make sure the ethernet updated to the latest NVM and drivers.

This comment and the prior discussion on the Intel site imply to me that the problem may lie with NIC configuration rather than with Opnsense config. Your proposed test may be informative ("may" because I lack complete confidence in USB-Ethernet adapters even though I sometimes use them in testing).

Yes that's the post I learned that NIC needs to be manually set to get 2.5gb/5gb links by default. I don't have the latest firmware (3.70) for the NIC, but I do have 3.50 and looking at the change log I don't appear to be exhibiting the bugs that were fixed.

Last night I did do a bit of research and found this on the forums.
Adding Speed Parameters to X550 Config

I added a tunable option of dev.ix.1.advertise_speed with a value of 23 which now allows my NIC to auto negotiate to either 100, 1g, 2.5g or 10g. This seems to be working now. 

You are shaping the WAN-side speed? What was the preceding position, when using auto-negotiation?

I currently have shaping turned off. I tried shaping as a troubleshooting step to limit the speed to 1gb but nothing changed.

The intel x550 NICs will not auto negotiate to 2.5gbps. Which is programmed into the firmware. Manual selection is required. This is why I have a 2.5gb usb nic order so I can test if the NIC is at fault.

[net.inet.rss.enabled = 1]

Did you try RSS?

I thought I had already done that. Some of the tunable items were complete except net.inet.rss.enabled = 1. I did change that and rebooted. However, my results have not changed.

I do have a USB 3 2.5gb adapter coming today I'm going to tryout incase the NIC is causing something. I use 1gb versions of these adapters for work all the time so no harm buying another one.

I have an issue when I have my WAN connection set to 2.5gb link speed to my Xfinity XB8 gateway, my download speeds are limited to around 300-350mbps. When I change my link speed on the WAN side back to 1gb I get around 940mbps (max 1gb link will do).

See attached image for network diagram.

Some additional information
This occurs on all 3 VLANs connected to my switch.
IPerf3 from a LAN 10gb interface to OPNSense easily gets over 9000 mbps.
Running a server hosting openspeedtest and iperf3, all PCs get their correct speed for their link speed across all vlans.

WAN set to 1gb
Running speedtest from OPNSense CLI gets around 1500 mbps when link is set to 2.5gbps, but get very high latency (log attached).
Running speedtest from 10gb and 1gb PCs get 940mpbs

WAN set to 10gb
Running speedtest OPNSense CLI at 1gb link I get 940mbps
Running speedtest from 10gb PCs get 1800mbps+
Running speedtest from 1gb PCs get 300-350mbps

I've tried just about everything on the Aruba switch to see if was doing something funky but nothing improved or depreciated the performance.
I've also tried other internet speedtest services like fast.com and I get the same results.
Cables used are CAT6 and TwinAx. All cables are shorter than 1m, with Gateway to Router being 6 inches. All manufactured cables.

Any ideas would be great.

P.S.
Forgot to mention. I haven't seen CPU usage above ~15% on OPNSense and htop shows multiple cores being used during these speedtests with none close to peaking out. The system is bare metal running an Intel i5-10600t and 16gb on RAM.