OPNsense Forum
Archive => 22.7 Legacy Series => Topic started by: ProximusAl on January 04, 2023, 01:47:05 pm
-
Hi all,
Looking for a bit of advice.
I just bought an R86S from AliExpress which has 3 x 2.5Gbe and 2 SFP+.
I've just installed OPNSense from scratch, and updated to 22.7.10_2
I was going to play with having the LAN interface on SFP+ number 1, but noticed that neither adapter was showing under Interface Assignments.
dmesg | grep mlx shows the following:
mlx4_core0: <mlx4_core> mem 0x7fd00000-0x7fdfffff,0x6000000000-0x60007fffff at device 0.0 on pci5
mlx4_core: Mellanox ConnectX core driver v3.7.1 (November 2021)
mlx4_core: Initializing mlx4_core
mlx4_core0: Unable to determine PCI device chain minimum BW
So after a bit of googling, I found that I can run the command:
kldload mlx4en
And the interfaces then show up fine.
My question is, is this normal?
I know I could add this into tunables to load, but it feels wrong. It feels like it's trying to load already as per dmesg, so loading it again doesn't feel right.
Does anyone have any experience of these Mellanox SFP+ with OPNSense, and could possibly guide me to the correct resolution. I should add, currently the SFP+ slot is empty, but I plan to add a 10G ethernet SFP+ adapter for testing.
Thanks
-
OK, I seem to have got around this.
Adding to tunables didn't work. The driver still doesn't load.
I found this post from early in the year:
https://github.com/opnsense/core/issues/5491
Seems to talk about using rc.syshook early to load the driver.
So a quick creation of 25-mlxload in /usr/local/etc/rc/syshook.d/early with content:
#!/bin/sh
kldload mlx4en
and it now seems to work after a reboot.
One thing for others to be aware of, on the R86S, the ports go left to right, igc0,1,2 but the SFP+ ports are reversed, so left to right is mlxen1, mlxen0. That caught me out.
I now have a 10GBaseT adapter working in mlxen1 :)
-
I have the same equipment. Do the following and both sfp+ work at 10G:
Add a file to the /boot directory. Name: loader.conf.local -- this enables the keeping of the file across updates, etc.
Contents of the file:
mlx4en_load="YES"
--- save and reboot. Both adapters then are detected as expected.
-
Great Info.
Do you have the means of running some sort of stress/troughput test on the device?
Rgds
-
I have the same equipment. Do the following and both sfp+ work at 10G:
Add a file to the /boot directory. Name: loader.conf.local -- this enables the keeping of the file across updates, etc.
Contents of the file:
mlx4en_load="YES"
--- save and reboot. Both adapters then are detected as expected.
Yes, this seems to work great.
Thanks
-
FraLem -- I have not been able to fully stress test them.
You can look at a review of the device here though: https://youtu.be/Z-YLy-RRZnM
-
Not sure if I should be piggy backing off this thread but I also recently received an R86S.
From googling I found the
loader.conf.local:
mlx4en_load="YES"
I can see the interfaces and assign them. I ended up buying two different 10gTek DACs (one listed as compatible with Ubiquti and one Mellanox). When I connect them to the switch I can see the details of the DAC in the Unifi Controller but the port is greyed out (disconnected). In the dmesg, I can see link DOWN but I am not sure where else to look to troubleshoot this.
root@OPNsense:" # dmesg | grep mlx
m1x4_core: Mellanox ConnectX core driver v3.7.1 (November 2021)
m1x4_core: Initializing m1x4_core
mlx4_core0: Unable to determine PCI device chain minimum BW
m1x4_en m1x4_core: Activating port:1
mlxen0: Ethernet address: xx
m1x4_en: mlx4_core0: Port 1: Using 4 TX rings
mlxen0: link state changed to DOWN
mlx4_en: mlx4_core@: Port 1: Using 8 RX rings m1x4_en: mlxeno: Using 4 TX rings
m1x4_en: mlxen0: Using 8 RX rings
m1x4_en: mlxen0: Initializing port mlx4_en mlx4_core: Activating port:2
mlxen1: Ethernet address: xx
m1x4_en: mlx4_core@: Port 2: Using 4 TX rings
mlxen1: link state changed to DOWN
m1x4_en: m1x4_core0: Port 2: Using 8 RX rings
m1x4_en: mlxen1: Using 4 TX rings
m1x4_en: mlxen1: Using 8 RX rings
m1x4_en: mlxen1: Initializing port
mlxen0: tso4 disabled due to -txcsum.
mlxen0: tso6 disabled due to -txcsum6.
mlxen1: ts04 disabled due to -txcsum.
mlxen1: tso6 disabled due to -txcsum6.
-
I can’t really help with that, but I’ve ordered the 10GTek DAC also, and I am awaiting a QNAP QSW-M1208-8C switch that they will ultimately plug into. The DAC I ordered is generic compatibility, so I have fingers crossed.
-
Oh No. Looks like I have the same problem as you.
The R86S is spewing this out with a DAC connected between SFP+ port and SFP+ port on new QNAP switch.
2023-01-06T11:20:54 Notice configd.py [0309ab3b-ec2a-45d3-826a-6fee54839855] Linkup stopping mlxen1
2023-01-06T11:20:52 Notice configd.py [1d98ba56-93fe-4a55-b940-9c2cdcc15fba] New IPv4 on mlxen1
2023-01-06T11:20:52 Notice configd.py [a128a906-a8ab-41f8-881e-543551341074] Linkup starting mlxen1
mlx4_en: mlxen1: Link Up
mlxen1: link state changed to UP
mlx4_en: mlxen1: Link Down
mlxen1: link state changed to DOWN
mlx4_en: mlxen1: Link Up
mlxen1: link state changed to UP
mlx4_en: mlxen1: Link Down
mlxen1: link state changed to DOWN
It's only a 1Gbps DAC (10GTek), and I've tried fixing the port speed, but I cannot get it to work :(
-
Further to this.
If I replace the DAC with a 10Gtek 10GBase-TX ethernet adapter it all works beautifully.
My QNAP has combo ports, so this was easy to test.
So, I'm thinking that the Mellanox SFP+ on the R86S isn't suitable for a DAC, but is happy with a 10GBase-TX adapter. Unfortunately, this scuppers my plans massively, as I *must* plug a DAC into it, as that is what is provided by my ISP, unless we can find some combination and go full fibre.
-
So as a test, I've replaced the DAC with:
10GTek SFP-LX transceivers at both ends, with OS2 LC to LC fibre patch cable in between.
It works, and link stays up no problems.
So I'm pretty confident now that the issues is with the DAC and the R86S.
Not sure where to go from here..
-
It sounds like an incompatible DAC. Had the same kind of issue with some optical SFP+ on a Mikrotik switch.
I have two Cisco DAC (SFP-H10GB-CU1M) and have no issues with the connection between the R86S and Mikrotik CRS317 it is connected to.
-
I'm hoping this post helps at least save someone money.
I still have issues with the R86S and SFP+ adapters between my QNAP switch and R86S.
Here's the outcome:
Working Equipment
-----------------
2 x SFP-LX 1310nm (ASF13-24-10-D) with OS2 fibre Patch cable (This is 1Gb only)
1 x 10Gtek 10GBase-T RJ-45 (ASF-10G-T)
Non Working
-----------
1 x 1Gbase-CU DAC (CAB-1GSFP-P1M)
1 x 10Gbase-CU DAC (CAB-10GSFP-P1M)
2 x 10Gbase-LR 1310nm SFP+ (AXS13-192-10) with OS2 fibre Patch cable
What has annoyed me, is the 10Gbase-CU DAC was clearly listed on Amazon as SFP-H10GB-CU1M. but that doesn't appear to be the case.
The 1Gbase-CU DAC (CAB-1GSFP-P1M) that doesn't work above between the R86S and the QNAP, DOES work between the R86S and my ISP ADVA device.
So I'm pouring money down the drain currently, trying to find a working combo.
I can only get 10G between the two using the RJ-45 adapter, which gets too hot for my liking, and I don't really want to be using CAT6 cables.
-
Why not order Mellanox coded optical transceivers from e.g. fs.com?
https://www.fs.com/de/c/10g-modules-56
-
I'm not sure it's the Mellanox that's causing the issue.
I had the 1Gb DAC not work between the R86S and the QNAP, but it does work between the R86S and my ADVA (ISP NTE)
Also, a previous poster bought a Mellanox coded DAC and that didn't work for him either.
I'm not sure now what end is causing the issue.
I need an SFP+ that works between the R86S and my QNAP switch, and then from the QNAP downstream to other devices, including Watchguard and Ubiquiti.
I may need to buy optics of the same wavelength all branded correctly, so Mellanox for the R86S, QNAP for the QNAP, Ubiquiti for the Unifi equipment.
This is my first venture into optics (aside from the optics coming from the ISP), and so far not enjoying it :)
-
Regularly you buy optics matching the device. You cannot expect optics to be interchangeable between devices.
fs.com are high quality at good prices and they code to match your device. They even sell you coding equipment if you absolutely must run one SFP+ e.g. in a Cisco one day and a Juniper device the next day.
But generally buy an optic matching the device, plug it in, leave it there.
HTH,
Patrick
-
Thanks Patrick,
OK, so as a test, I've only gone between the R86S and a MikroTik SFP+ switch.
Everything stays up whether I use a DAC or optics, so clearly the QNAP is extremely picky.
Might need to buy some QNAP optics :(
-
@pmhausen.
So I've found a compatibility chart for my QNAP switch.
https://www.qnap.com/en-uk/compatibility/?model=472
It lists various DACs that are so say compatible, but as I've invested already in optics and fibre I'd like to use that.
Most of the optics on the chart seem to be MultiMode fibre, but I've got a lot of SingleMode optics and cable already.
I can see one optic listed as compatible with the same wavelength as my other optics I already own.
10G Ethernet, 1310nm, 10GBASE-LR, SFP+ Transceiver for 10km SMF Links
Avago AFCT-739ASMZ
If I ordered this:
https://www.fs.com/uk/products/14914.html
do you think it would be feasible to then use my other optics known to work with the R86S and existing fibre patch cables. Trying to not waste money.
I've read that MultiMode should be used for short distances like I want, but as I say I have a *lot* of SingleMode optics already with fibre patch cables.
Apologies for asking you to educate me, but I'd like to learn......rather than waste money guessing what might work.
-
Ok, so some updates for those following this thread:
First, @cygni, you said you had trouble between the R86S and Ubiquiti, but you didn't say what Unifi device.
I can confirm that a generic 10GTek DAC (CAB-10GSFP-P1M) 10GBase-CU works perfectly well between the R86S and a UDM Pro.
It's the only Unifi device I have with SFP+ to test, but it works.
I also tested a Cisco SFP-10G-SR with some MM fibre and that also worked between R86S and UDM Pro.
Now to my situation....
After a lot of testing I finally know what my issue is, and it's a bit complicated.
My QNAP QSW-M1208-8C switch has 12 SFP+ ports, and 8 10GBaseT ports, of which the 8 are combo ports with 8 other SFP+.
Ports 1-4 are standalone SFP+, and 5-12 are combo ports.
I can use pretty much anything between the R86S SFP+ and the QNAP ports 1-4, whether that be a DAC, optics SM or MM, and the link always comes up.
OPNSense is happy, and that's great.
If I use *any* of the other combo SFP+ ports on the QNAP, the link flaps up and down.
I've raised a support issue with QNAP on this but I don't think it's simple, because...
If I add a MikroTik CRS305 switch *between* the R86S and the QNAP SFP+ combo ports.....yep...the links stay up.
I've tested the R86S SFP+ port with a generic DAC between an ADVA ISP device, a Watchguard Firewall SFP+, a Unifi UDM SFP+ and link is always up.
I've tested the QNAP combo SFP+ port on Watchguard Firewall SFP+ and Unifi UDM SFP+ and link is always up.
But, if you go R86S to QNAP Combo SFP+....nada
Only R86S to QNAP non combo works.
As I said, mind boggling.
So to resolve my conundrum, I have VLANd the QNAP untagged, so port based, in the following way:
SFP+ Port 2 goes to my R86S, but VLAN with ports 9-12 (The combos) for my downstream devices.
SFP+ Port 1 goes to my *other* R86S, but VLAN with ports 5-8 (The combos) for my downstream devices.
Port 3 and 4 on seperate management VLAN.
That supports both my internet connections.
What have I learned from this.....well, don't go spending hundreds on a QNAP QSW-M1208-8C if you want to use it with an R86S, find another VLAN capable switch.
I hope that QNAP action my support ticket, but I don't hold out much hope, and at least I have a workaround.
I spent most of my time blaming the *cheap* R86S device, but I'm squarely pointing the finger now at QNAP, although it is super strange how the R86S and QNAP combo ports just dont seem to be compatible, but the non combo ports are perfect.
Hope this helps someone in the future....
P.S. Love OPNSense :)
-
Finally finished my config.
Here is the fruits of my labour.
2 x R86S on separate 1Gb Leased Lines, converged into MultiWan on my Firebox, and serving a UDM Pro.
All links are 10Gb downstream of the R86s’s save for the Ethernet in the management ports at 1Gb.
These little machines are for routing only, running OPNSense 23.1.
(https://i.ibb.co/jw5DLmR/98-A43-F50-113-E-4-BF0-A086-292-C0-D642-CC7.jpg) (https://ibb.co/qgnpRK5)
-
Nice setup! I am looking to purchase the same unit. How has it been so far?
-
It's been brilliant.
Been running in production now for a few months with no issues.