WAN speed degradation for all 1Gbps ethernet ports while 10G SFP+ client is fine

[Noobstradamus]

I currently am using a SFP+ DAC cable to connect the switch to my firewall. They both negotiate at 10G. I have a 10G capable client connected to another SFP+ port on my Aruba Instant on switch and this client receives full WAN bandwidth >1400 Mbps. The problem though is when using the SFP+ DAC cable to my firewall literally all regular 1 Gbps ethernet clients have neutered or heavily reduced WAN speeds less than 300 Mbps with other speed oddities. The other odd thing is that inter-vlan/internal traffic seems to be full speed e.g. when I use my 10G client to my NAS I get full 1Gbps speed transfer (NAS has 1G port).

How is it possible though that the 10G client is getting full speeds using the same DAC cable firewall connection while the other 1G clients cannot get their full 1G WAN download speed? Same VLAN and all the same firewall settings as well. If I move WAN to the firewalls 1G ethernet port then 1G clients get full speed, but of course the 10G client does not get full WAN then. So not sure what could be causing this behavior.



Firewall Sophos 330 Rev 2 running latest OPNsense 22.1.3 (has 2 Intel x520 SFP+ ports):

->SFP+ Port 1: WAN 2.5Gbps Microtik S+RJ10 transceiver -> S33 cable modem  (get full 1400 Mbps for only 10G client so the 2.5Gbps to the modem is negotiating properly)
                                                             
 ->SFP+ Port 2: 10G SFP+ DAC cable to switch (1G clients degraded WAN speed <300Mbps, LAN speed normal 1Gbps)



This issue has been driving me up a wall with trying to figure out if it is some kind of weird buffer issue. I have tried all the tunables such as disable flow control, enabling and disabling hardware offload etc... Any help would be greatly appreciated.

[Vesalius]

Did you assign the parent interface of VLAN interfaces and enable each of them, even if it is not used directly? New requirement that has commonly been overlooked with 22.1 and above.

[Noobstradamus]

Thanks for the reply. Yes all VLAN are assigned to an interface and enabled. Please see attached.

[Noobstradamus]

Turns out it was a problem with flow control. Once I enabled flow control for only my LAN SFP+ connection on my firewall to the switch and left flow control off for the WAN interface on the firewall both 1G and 10G clients were getting proper WAN speeds on the switch.

Remember ethernet speed mismatch can cause problems, use flow control to resolve them!