R86S - Mellanox

[Patrick M. Hausen]

Regularly you buy optics matching the device. You cannot expect optics to be interchangeable between devices.

fs.com are high quality at good prices and they code to match your device. They even sell you coding equipment if you absolutely must run one SFP+ e.g. in a Cisco one day and a Juniper device the next day.

But generally buy an optic matching the device, plug it in, leave it there.

HTH,
Patrick

[ProximusAl]

Thanks Patrick,

OK, so as a test, I've only gone between the R86S and a MikroTik SFP+ switch.

Everything stays up whether I use a DAC or optics, so clearly the QNAP is extremely picky.

Might need to buy some QNAP optics

[ProximusAl]

@pmhausen.

So I've found a compatibility chart for my QNAP switch.

https://www.qnap.com/en-uk/compatibility/?model=472

It lists various DACs that are so say compatible, but as I've invested already in optics and fibre I'd like to use that.

Most of the optics on the chart seem to be MultiMode fibre, but I've got a lot of SingleMode optics and cable already.

I can see one optic listed as compatible with the same wavelength as my other optics I already own.

10G Ethernet, 1310nm, 10GBASE-LR, SFP+ Transceiver for 10km SMF Links
Avago   AFCT-739ASMZ

If I ordered this:

https://www.fs.com/uk/products/14914.html

do you think it would be feasible to then use my other optics known to work with the R86S and existing fibre patch cables. Trying to not waste money.

I've read that MultiMode should be used for short distances like I want, but as I say I have a *lot* of SingleMode optics already with fibre patch cables.

Apologies for asking you to educate me, but I'd like to learn......rather than waste money guessing what might work.

[ProximusAl]

Ok, so some updates for those following this thread:

First, @cygni, you said you had trouble between the R86S and Ubiquiti, but you didn't say what Unifi device.
I can confirm that a generic 10GTek DAC (CAB-10GSFP-P1M) 10GBase-CU works perfectly well between the R86S and a UDM Pro.
It's the only Unifi device I have with SFP+ to test, but it works.
I also tested a Cisco SFP-10G-SR with some MM fibre and that also worked between R86S and UDM Pro.

Now to my situation....

After a lot of testing I finally know what my issue is, and it's a bit complicated.

My QNAP QSW-M1208-8C switch has 12 SFP+ ports, and 8 10GBaseT ports, of which the 8 are combo ports with 8 other SFP+.
Ports 1-4 are standalone SFP+, and 5-12 are combo ports.

I can use pretty much anything between the R86S SFP+ and the QNAP ports 1-4, whether that be a DAC, optics SM or MM, and the link always comes up.
OPNSense is happy, and that's great.
If I use *any* of the other combo SFP+ ports on the QNAP, the link flaps up and down.
I've raised a support issue with QNAP on this but I don't think it's simple, because...
If I add a MikroTik CRS305 switch *between* the R86S and the QNAP SFP+ combo ports.....yep...the links stay up.

I've tested the R86S SFP+ port with a generic DAC between an ADVA ISP device, a Watchguard Firewall SFP+, a Unifi UDM SFP+ and link is always up.
I've tested the QNAP combo SFP+ port on Watchguard Firewall SFP+ and Unifi UDM SFP+ and link is always up.
But, if you go R86S to QNAP Combo SFP+....nada
Only R86S to QNAP non combo works.

As I said, mind boggling.

So to resolve my conundrum, I have VLANd the QNAP untagged, so port based, in the following way:
SFP+ Port 2 goes to my R86S, but VLAN with ports 9-12 (The combos) for my downstream devices.
SFP+ Port 1 goes to my *other* R86S, but VLAN with ports 5-8 (The combos) for my downstream devices.
Port 3 and 4 on seperate management VLAN.

That supports both my internet connections.

What have I learned from this.....well, don't go spending hundreds on a QNAP QSW-M1208-8C if you want to use it with an R86S, find another VLAN capable switch.
I hope that QNAP action my support ticket, but I don't hold out much hope, and at least I have a workaround.

I spent most of my time blaming the *cheap* R86S device, but I'm squarely pointing the finger now at QNAP, although it is super strange how the R86S and QNAP combo ports just dont seem to be compatible, but the non combo ports are perfect.

Hope this helps someone in the future....

P.S. Love OPNSense

[ProximusAl]

Finally finished my config.

Here is the fruits of my labour.

2 x R86S on separate 1Gb Leased Lines, converged into MultiWan on my Firebox, and serving a UDM Pro.

All links are 10Gb downstream of the R86s’s save for the Ethernet in the management ports at 1Gb.

These little machines are for routing only, running OPNSense 23.1.

[zhuoerh]

Nice setup! I am looking to purchase the same unit. How has it been so far?

[ProximusAl]

It's been brilliant.

Been running in production now for a few months with no issues.