R86S - Mellanox

[ProximusAl]

Hi all,

Looking for a bit of advice.

I just bought an R86S from AliExpress which has 3 x 2.5Gbe and 2 SFP+.

I've just installed OPNSense from scratch, and updated to 22.7.10_2

I was going to play with having the LAN interface on SFP+ number 1, but noticed that neither adapter was showing under Interface Assignments.

dmesg | grep mlx shows the following:

mlx4_core0: <mlx4_core> mem 0x7fd00000-0x7fdfffff,0x6000000000-0x60007fffff at device 0.0 on pci5
mlx4_core: Mellanox ConnectX core driver v3.7.1 (November 2021)
mlx4_core: Initializing mlx4_core
mlx4_core0: Unable to determine PCI device chain minimum BW

So after a bit of googling, I found that I can run the command:

kldload mlx4en

And the interfaces then show up fine.

My question is, is this normal?

I know I could add this into tunables to load, but it feels wrong.  It feels like it's trying to load already as per dmesg, so loading it again doesn't feel right.

Does anyone have any experience of these Mellanox SFP+ with OPNSense, and could possibly guide me to the correct resolution.  I should add, currently the SFP+ slot is empty, but I plan to add a 10G ethernet SFP+ adapter for testing.

Thanks

[ProximusAl]

OK, I seem to have got around this.

Adding to tunables didn't work.  The driver still doesn't load.

I found this post from early in the year:

https://github.com/opnsense/core/issues/5491

Seems to talk about using rc.syshook early to load the driver.

So a quick creation of 25-mlxload in /usr/local/etc/rc/syshook.d/early with content:

#!/bin/sh

kldload mlx4en

and it now seems to work after a reboot.

One thing for others to be aware of, on the R86S, the ports go left to right, igc0,1,2 but the SFP+ ports are reversed, so left to right is mlxen1, mlxen0.  That caught me out.

I now have a 10GBaseT adapter working in mlxen1 :)

[dd2594opn]

I have the same equipment. Do the following and both sfp+ work at 10G:

Add a file to the /boot directory. Name: loader.conf.local  -- this enables the keeping of the file across updates, etc.

Contents of the file:

mlx4en_load="YES"

--- save and reboot. Both adapters then are detected as expected.

[FraLem]

Great Info.
Do you have the means of running some sort of stress/troughput  test on the device?
Rgds

[ProximusAl]

I have the same equipment. Do the following and both sfp+ work at 10G:

Add a file to the /boot directory. Name: loader.conf.local  -- this enables the keeping of the file across updates, etc.

Contents of the file:

mlx4en_load="YES"

--- save and reboot. Both adapters then are detected as expected.

Yes, this seems to work great.

Thanks

[dd2594opn]

FraLem -- I have not been able to fully stress test them.

You can look at a review of the device here though: https://youtu.be/Z-YLy-RRZnM

[cygni]

Not sure if I should be piggy backing off this thread but I also recently received an R86S.
From googling I found the
loader.conf.local:
mlx4en_load="YES"

I can see the interfaces and assign them. I ended up buying two different 10gTek DACs (one listed as compatible with Ubiquti and one Mellanox). When I connect them to the switch I can see the details of the DAC in the Unifi Controller but the port is greyed out (disconnected). In the dmesg, I can see link DOWN but I am not sure where else to look to troubleshoot this.

root@OPNsense:" # dmesg | grep mlx
m1x4_core: Mellanox ConnectX core driver v3.7.1 (November 2021)
m1x4_core: Initializing m1x4_core
mlx4_core0: Unable to determine PCI device chain minimum BW
m1x4_en m1x4_core: Activating port:1
mlxen0: Ethernet address: xx
m1x4_en: mlx4_core0: Port 1: Using 4 TX rings
mlxen0: link state changed to DOWN
mlx4_en: mlx4_core@: Port 1: Using 8 RX rings m1x4_en: mlxeno: Using 4 TX rings
m1x4_en: mlxen0: Using 8 RX rings
m1x4_en: mlxen0: Initializing port mlx4_en mlx4_core: Activating port:2
mlxen1: Ethernet address: xx
m1x4_en: mlx4_core@: Port 2: Using 4 TX rings
mlxen1: link state changed to DOWN
m1x4_en: m1x4_core0: Port 2: Using 8 RX rings
m1x4_en: mlxen1: Using 4 TX rings
m1x4_en: mlxen1: Using 8 RX rings
m1x4_en: mlxen1: Initializing port
mlxen0: tso4 disabled due to -txcsum.
mlxen0: tso6 disabled due to -txcsum6.
mlxen1: ts04 disabled due to -txcsum.
mlxen1: tso6 disabled due to -txcsum6.

[ProximusAl]

I can't really help with that, but I've ordered the 10GTek DAC also, and I am awaiting a QNAP QSW-M1208-8C switch that they will ultimately plug into. The DAC I ordered is generic compatibility, so I have fingers crossed.

[ProximusAl]

Oh No.  Looks like I have the same problem as you.

The R86S is spewing this out with a DAC connected between SFP+ port and SFP+ port on new QNAP switch.

2023-01-06T11:20:54   Notice   configd.py   [0309ab3b-ec2a-45d3-826a-6fee54839855] Linkup stopping mlxen1   
2023-01-06T11:20:52   Notice   configd.py   [1d98ba56-93fe-4a55-b940-9c2cdcc15fba] New IPv4 on mlxen1   
2023-01-06T11:20:52   Notice   configd.py   [a128a906-a8ab-41f8-881e-543551341074] Linkup starting mlxen1

mlx4_en: mlxen1: Link Up
mlxen1: link state changed to UP
mlx4_en: mlxen1: Link Down
mlxen1: link state changed to DOWN
mlx4_en: mlxen1: Link Up
mlxen1: link state changed to UP
mlx4_en: mlxen1: Link Down
mlxen1: link state changed to DOWN

It's only a 1Gbps DAC (10GTek), and I've tried fixing the port speed, but I cannot get it to work :(

[ProximusAl]

Further to this.

If I replace the DAC with a 10Gtek 10GBase-TX ethernet adapter it all works beautifully.
My QNAP has combo ports, so this was easy to test.

So, I'm thinking that the Mellanox SFP+ on the R86S isn't suitable for a DAC, but is happy with a 10GBase-TX adapter.  Unfortunately, this scuppers my plans massively, as I *must* plug a DAC into it, as that is what is provided by my ISP, unless we can find some combination and go full fibre.

[ProximusAl]

So as a test, I've replaced the DAC with:

10GTek SFP-LX transceivers at both ends, with OS2 LC to LC fibre patch cable in between.

It works, and link stays up no problems.

So I'm pretty confident now that the issues is with the DAC and the R86S.

Not sure where to go from here..

[dd2594opn]

It sounds like an incompatible DAC. Had the same kind of issue with some optical SFP+ on a Mikrotik switch.

I have two Cisco DAC (SFP-H10GB-CU1M) and have no issues with the connection between the R86S and Mikrotik CRS317 it is connected to.

[ProximusAl]

I'm hoping this post helps at least save someone money.

I still have issues with the R86S and SFP+ adapters between my QNAP switch and R86S.

Here's the outcome:

Working Equipment
-----------------
2 x SFP-LX 1310nm (ASF13-24-10-D) with OS2 fibre Patch cable (This is 1Gb only)
1 x 10Gtek 10GBase-T RJ-45 (ASF-10G-T)

Non Working
-----------
1 x 1Gbase-CU DAC (CAB-1GSFP-P1M)
1 x 10Gbase-CU DAC (CAB-10GSFP-P1M)
2 x 10Gbase-LR 1310nm SFP+ (AXS13-192-10) with OS2 fibre Patch cable

What has annoyed me, is the 10Gbase-CU DAC was clearly listed on Amazon as SFP-H10GB-CU1M. but that doesn't appear to be the case.

The 1Gbase-CU DAC (CAB-1GSFP-P1M) that doesn't work above between the R86S and the QNAP, DOES work between the R86S and my ISP ADVA device.

So I'm pouring money down the drain currently, trying to find a working combo.

I can only get 10G between the two using the RJ-45 adapter, which gets too hot for my liking, and I don't really want to be using CAT6 cables.

[Patrick M. Hausen]

Why not order Mellanox coded optical transceivers from e.g. fs.com?

https://www.fs.com/de/c/10g-modules-56

[ProximusAl]

I'm not sure it's the Mellanox that's causing the issue.

I had the 1Gb DAC not work between the R86S and the QNAP, but it does work between the R86S and my ADVA (ISP NTE)

Also, a previous poster bought a Mellanox coded DAC and that didn't work for him either.

I'm not sure now what end is causing the issue.

I need an SFP+ that works between the R86S and my QNAP switch, and then from the QNAP downstream to other devices, including Watchguard and Ubiquiti.

I may need to buy optics of the same wavelength all branded correctly, so Mellanox for the R86S, QNAP for the QNAP, Ubiquiti for the Unifi equipment.

This is my first venture into optics (aside from the optics coming from the ISP), and so far not enjoying it :)