LAN speeds are slower on card SFP+ port compared to motherboard RJ45

[sinayion]

(please let me know what other info I can provide to help diagnose this issue)

Hardware
Computer: Lenovo M75s Gen 2 Desktop (ThinkCentre) - Type 11R8
Network card: 10Gtek X710-10G-2S-X8
Switch: Omada SG3428X v1.30
Connection between opnsense and modem: CAT8 with transceiver
Connection between opnsense and switch: Active SFP+ DAC, ACC Cable with CDR

Previously posted on Reddit: https://www.reddit.com/r/opnsense/comments/1h14vx0/lan_speeds_are_slower_on_card_sfp_port_compared/

I have a 10gtek card with 2 SFP+ ports (X710-10G-2S-X8). I have been using one port for WAN (CAT 8 with transceiver connected to modem) and the other for LAN (Active SFP+ DAC, ACC Cable with CDR to Omada managed switch). Speedtests have normally been close to my actual connection (1200Mbps/35Mbps) for two years now, no issues.

I noticed in the last month that my speeds were terrible compared to usual, usually ~120/~30. I changed nothing in my opnsense system other than regular opnsense, BIOS and NVM updates. I actually reinstalled opnsense from scratch to test this, but had the same issue. What fixed it: I switched my LAN interface to the regular ethernet interface on the motherboard (named re0) via the opensense command line, rebooted, and my old fast speeds came back.

Any idea on how I can figure out why the old LAN SFP+ port is being slow? Below is my ifconfig -a output.

ixl0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 80:61:5f:14:f7:cf
inet 67.188.154.177 netmask 0xfffff800 broadcast 255.255.255.255
inet6 fe80::8261:5fff:fe14:f7cf%ixl0 prefixlen 64 scopeid 0x1
inet6 2001:558:6045:1f:b861:31fd:dda6:f10f prefixlen 128 pltime 301639 vltime 301639
media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

ixl1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: OPT1 (opt1)
options=4900028<VLAN_MTU,JUMBO_MTU,NETMAP,HWSTATS,MEXTPG>
ether 80:61:5f:14:f7:d0
inet 10.1.0.1 netmask 0xffff0000 broadcast 10.1.255.255
inet6 fe80::8261:5fff:fe14:f7d0%ixl1 prefixlen 64 scopeid 0x2
inet6 2601:640:cc00:f321:8261:5fff:fe14:f7d0 prefixlen 64
groups: CommonDNS
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

re0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN (lan)
options=102008<VLAN_MTU,WOL_MAGIC,NETMAP>
ether 88:ae:dd:12:b4:cd
inet 10.0.0.1 netmask 0xffff0000 broadcast 10.0.255.255
inet6 fe80::8aae:ddff:fe12:b4cd%re0 prefixlen 64 scopeid 0x3
inet6 2601:640:cc00:f320:8aae:ddff:fe12:b4cd prefixlen 64
groups: CommonDNS
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

[meyergru]

You have quite some special setup there - for starters, you use netmap (Zenarmor?). Try disabling that.

Also, when you compare the speeds and you could swap re0 and ixl0, your SFP+ cages could do 10 Gbit/s, but may have transceivers that only use or support 1 Gbit/s. There is a speed mismatch and currently your ixl seems to be configured to do pause frames (rxpause, txpause).

A third point that comes to mind are the jumbo frames which may or may not be supported by your transceivers and/or counterparts.

Sometimes, it also helps to select the media explicitely (10Gbase-SR, 10Gbase-Twinax).

[sinayion]

You have quite some special setup there - for starters, you use netmap (Zenarmor?). Try disabling that.

Also, when you compare the speeds and you could swap re0 and ixl0, your SFP+ cages could do 10 Gbit/s, but may have transceivers that only use or support 1 Gbit/s. There is a speed mismatch and currently your ixl seems to be configured to do pause frames (rxpause, txpause).

A third point that comes to mind are the jumbo frames which may or may not be supported by your transceivers and/or counterparts.

Sometimes, it also helps to select the media explicitely (10Gbase-SR, 10Gbase-Twinax).

Thanks, I like my little setup. Opsense has been awesome for the past two years.

I already tried with Zenarmor disabled; I also tried the speeds after immediately reinstalling opnsense to make sure it wasnt some weird config/plugin I had added.

I know the transceivers for the two desktops are fine (connected to the Omada switch) because they give correct speeds when I use re0 as LAN. I wish I had another SFP+ DAC cable to test if this thing somehow has deteriorated after two years. Are the pause frames something I would have explicitly configured somewhere? I don't recall setting those up.

On the Omada switch I have the default value of 1518 under Jumbo Frames (the field allows even values between the range 1518-9216). I haven't changed the MTU fields under Interfaces in opnsense.

I'll try forcing the speed in opnsense later tonight after everyone else doesn't need the internet.