Hi everybody
In my configuration I observe a strange behaviour, I can't understand. My setup is quite simple: I installed OPNSense on a R86S with 3x2.5G and 2x10G SFP+ ports. OPNSense is configured as transparent bridge with all ports. If I connect the bridge to my provider router from a 2.5G port I get 2.3Gbit/s for up and down, measured with speedtest from Ookla on the box. My provider router has a multi port and can deliver up to 10GBit/s. So this connection is limited by the line rate.
If I change the cable from the 2.5G port to the 10G SFP+ port (via a RJ45 module) I measure 7.7GBit/s as down and 1.1GBit/s as up. What I don't understand: Why is up lower with the faster port? Do I have to tweak settings on the Mellanox connectx-3 to increase uploads? I would be very happy about any help or hints.
Hi legoingschenoer,
That sounds like the Mellanox NIC is running in half-duplex, i.e. the 10 GBit/s are shared among send and receive.
What does "ifconfig -a" say?
Thanks for the fast answer!
Here my output from ifconfig -a:
igc0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: ETH1 (opt2)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 00:f0:cb:fe:b0:df
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
...
mlxen0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN (wan)
options=8c00a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
ether 00:02:c9:c1:8c:b8
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
mlxen1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN (lan)
options=8c00a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
ether 00:02:c9:c1:8c:b9
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: OPT (opt1)
ether 58:9c:fc:10:82:4b
inet 192.168.xx.2 netmask 0xffffff00 broadcast 192.168.xx.255
inet6 aaaa:bbbb:cccc:dddd:5a9c:fcff:fe10:824b prefixlen 64
inet6 fe80::5a9c:fcff:fe10:824b%bridge0 prefixlen 64 scopeid 0xa
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: mlxen0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000
member: mlxen1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000
member: igc2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 2000000
member: igc1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 2000000
member: igc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000000
groups: bridge
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
On mlxen0 I read ,,full-duplex". Do you mean this?
According to the output both Mellanox ports run as full-duplex, so this can't be the issue.
But it's strange that the ports run as 10GBASE-CX4, which is an old standard with SFF-8470 connector.
Maybe that's just the transceiver. Which SFP+-RJ45 transceivers are you using? Maybe they're the problem
Thanks for the reply!
Didn't know about the CX4. Just saw it and read something about a twinax cable. I expect someting like 10GBase-T because I am using a RJ45 CAT7 cable. Tryed to change the ,,Speed" but the GUI has only the following options: CR4, CX4, SR and 1000base-T. Even with 1000base-T and a reboot it falls back to CX4.
I'am using a tranceiver from ONTi ONT-C1TE-R01. Ordered it from Aliexpress: https://www.aliexpress.com/i/4000340325163.html. Do you think I should try another tranceiver? Do you know a compatible tranceiver?
I searched in this forum and other users also get 10GBASE-CX4 reported, so this probably is normal for Mellanox cards, although it should be 10GBASE-CR4 for DACs.
With "ifconfig -v mlxen0" you can show information about the transceiver.
But it could also be the other side that is connected to the cable. Did you test the modem with another device?
What is your line speed and is the CAT7 cable shielded and how long is it?
The device itself doesn't seem to be the problem, with the version GW-R86S-G3 it reached about 9.3 GBit/s in iperf3 on OPNsense 22.7:
https://www.servethehome.com/the-gowin-r86s-revolution-low-power-2-5gbe-and-10gbe-intel-nvidia/3/
Some Gowin Solution developer tested it with Ookla speedtest and got 7527.79 Mbit/s down and 7669.46 Mbit/s up with version GW-R86S-G4 on OPNsense (23.1?):
https://www.reddit.com/r/R86SNetworking/comments/12j6r5m/ama_for_the_gowin_r86su_firewall_router_first/
This is my output from "ifconfig -v mlxen0":
mlxen0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN (wan)
options=8c00a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
ether 00:02:c9:c1:8c:b8
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: PN: SN: 2306210059 DATE: 2023-06-24
module temperature: 159.00 C voltage: 0.45 Volts
lane 1: RX power: 3.28 mW (5.15 dBm) TX bias: 108.96 mA
Some strange values ... but i do not expect much.
I connected a quite old Win10 PC with an Intel X550-T2 NIC to the same cable and got 7.8 GB/s (down) and 3.7GB/s (up). My line speed is 10GB/s, the cable is shielded and only half a meter in length. But I did the test with a server on the internet.
It looks like a tranceiver problem... Contacted ONTi, but they do not know of any problems: just plug and play. Maybe I should try an other tranceiver.
So I suppose the line is 10GBit/s symmetrical.
Because your X550-T2 also doesn't get the full upload speed this would suggest a problem with the line or with the modem.
Can you test another speed test, e.g. speedof.me?
There are also iperf3 binaries for Windows:
https://files.budman.pw/ (yes, it looks shady, but the guy seems to have a good reputation: https://www.neowin.net/forum/topic/1234695-iperf-314-windows-build/)
https://github.com/ar51an/iperf3-win-builds
(current version: 3.14)
Create a .bat file and do e.g.:
iperf3 -c a210.speedtest.wobcom.de
iperf3 -c a210.speedtest.wobcom.de -R
iperf3 -c a210.speedtest.wobcom.de --bidir
pause
This will test the upload (default), then test the download (-R) then do a concurrent download and upload (--bidir).