Mellanox Connect-X4 25Gbit/s not Working (working in pfsense)

[wondaah]

Hi,

I recently purchased a R86S-N305C which has two 25G interfaces from a MLX Connectx-4 card.

The issue that I'm facing is that it does not seem to negotioate correctly, I worked at my own ISP so I know what they have configured and I know it is correct.

Normally to get 25Gbit/s WAN Speeds we did set the Autoneg to off, Speed to 25Gbit/s and FEC to ON. I cannot see any documentations on how to do that except this thread with no answers on it.  https://forum.opnsense.org/index.php?topic=30467.msg146919#msg146919

Could anyone help me config wise to turn FEC on? Or maybe something more temporary than described in the above post?

Best regards

Drivers getting loaded:

Code Select Expand

root@OPNsense:~ # dmesg | grep mlx
mlx5_core0: <mlx5_core> mem 0x60e2000000-0x60e3ffffff at device 0.0 on pci4
mlx5: Mellanox Core driver 3.7.1 (November 2021)uhub1: 2 ports with 2 removable, self powered
mlx5_core0: INFO: mlx5_port_module_event:710:(pid 12): Module 0, status: unplugged
mlx5_core0: WARN: mlx5_vsc_set_space:125:(pid 0): Space 0x7 is not supported.
mlx5_core0: WARN: mlx5_fwdump_prep:102:(pid 0): VSC scan space is not supported
mlx5_core: INFO: (mlx5_core0): E-Switch: Total vports 9, l2 table size(65536), per vport: max uc(1024) max mc(16384)
mlx5_core1: <mlx5_core> mem 0x60e0000000-0x60e1ffffff at device 0.1 on pci4
mlx5_core1: INFO: mlx5_port_module_event:705:(pid 12): Module 1, status: plugged and enabled
mlx5_core1: WARN: mlx5_vsc_set_space:125:(pid 0): Space 0x7 is not supported.
mlx5_core1: WARN: mlx5_fwdump_prep:102:(pid 0): VSC scan space is not supported
mlx5_core: INFO: (mlx5_core1): E-Switch: Total vports 9, l2 table size(65536), per vport: max uc(1024) max mc(16384)
mlx5_core0: ERR: mlx5_cmd_check:710:(pid 0): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x6c4d48)
mlx5_core1: ERR: mlx5_cmd_check:710:(pid 0): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x6c4d48)
mce0: ERR: mlx5e_ioctl:3514:(pid 40442): tso4 disabled due to -txcsum.
mce0: ERR: mlx5e_ioctl:3527:(pid 41160): tso6 disabled due to -txcsum6.
mce1: ERR: mlx5e_ioctl:3514:(pid 43036): tso4 disabled due to -txcsum.
mce1: ERR: mlx5e_ioctl:3527:(pid 43090): tso6 disabled due to -txcsum6.

IFCONFIG

Code Select Expand

mce0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN (wan)
        options=7e8800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO>
        ether 24:8a:07:95:d9:18
        inet6 fe80::268a:7ff:fe95:d918%mce0 prefixlen 64 tentative scopeid 0x8
        media: Ethernet autoselect <rxpause,txpause>
        status: no carrier (Cable is unplugged)
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Also mlxen5 yes is active in /boot/loader

[wondaah]

I just installed pfsense to test...

Its working there, without enabling anything...

[LOTRouter]

Have you tried this?

https://www.reddit.com/r/opnsense/comments/16q3e76/fixing_mellanoxcard/

[nghappiness]

Can you try follow this PDF file section 3.9?

https://network.nvidia.com/pdf/prod_software/Mellanox_FreeBSD_User_Manual_v3.5.2.pdf

hope it helps.