Help with Intel X553 and 1G SFP

[shor0814]

I recently purchased a  Qotom 20331G9 and am having some issues with the SFP+ when attempting to connect my 1G fiber SFP transceiver into the X553 SFP+ port.

For background, I have fiber internet that uses a SFP 1G module connected to a Calix modem.  To avoid double NAT I had unplugged the SFP and plugged it into my Ubiquiti Dream Machine Pro (UDMP) and it lights right up and connects immediately when using the correct MAC address.

Fast forward to this month when the UDM upgrade changed something that blocked the GRE connections on my network, combined with poor IPV6 support I decided to try OPNsense.  The Qotom was configured, and imagine my disappointment when there are no lights and no connection shown (using ifconfig) on the SFP+ port when I plug my fiber module into the X553 port.

To troubleshoot, I verified that I get lights and connectivity when plugging in a SFP+ 10G twinax cable connected to a 10G switch.  I also made sure that "hw.ix.unsupported_sfp=1" is in the loader.conf.local. 

I also don't see the settings in the UI for speed and duplexing like described in the docs, but that is probably because the supported media in ifconfig is only media autoselect.  I was trying to force 1G speeds to see if that would help but can't do that easily.

Is there any additional tests I can do?  Any additional settings I can try?  Maybe I am not waiting long enough for a connection?   I was going to install Linux and see what happens.  Anyone able to find actual support docs for the X553?  They seem to be nonexistent and the 550 series is also a bit vague.

For reference, the UDMP has an Ethernet controller: Annapurna Labs Ltd. SFP+ 10G Ethernet Adapter (rev 03)
The SFP adapter for the modem is a Calix with the following on the label: 1GE SC BiDi 1310nm "U" 20km  RT.

It just feels "wrong" that my UDMP works with this module but OPNsense doesn't.  In the meantime, I am asking my ISP if there is a way to put the modem in transparent bridge mode or if they have a different transceiver. 

Help is much appreciated.  I do have another server with similar hardware but does have an Intel x722 (which seems to do the same thing) but I can easily replace the PCI card in that one if there is a known supported card.

[Greg_E]

It may be an issue with the firmware or model of NIC chip that was used in the qotom device.

This is wasteful, but can you connect the 1gb SFP into a switch, and then connect the 10gb SFP+ into the same switch? Essentially using the switch to "buffer" between the two connections.

I think you are on the right track to where you need to force the speed down to 1gbps, and you'll probably also need to spoof the MAC address of the original modem too. Since the interface isn't coming up, I'm not sure how you will be able to enter the MAC address. I also don't have a 10gb card I can use for testing.

I would probably through a Windows evaluation version on there, try to do a firmware update on that device, and then go back to OPNsense. Alternate if you grab all the correct drivers and utilities, you can set up a Hiren's Boot CD https://www.hirensbootcd.org/ , drop the drivers into the DRIVER folder, and the utility somewhere on the bootable device. You should then be able to update things if there is a firmware update available.

Last thought is that the SFP module just isn't programmed to work with that interface, usually this is the interface blocking things. Not sure how to fix this unless you can find some generic firmware or a utility that allows you to "open it up" to all modules.

[shor0814]

It may be an issue with the firmware or model of NIC chip that was used in the qotom device.

This is wasteful, but can you connect the 1gb SFP into a switch, and then connect the 10gb SFP+ into the same switch? Essentially using the switch to "buffer" between the two connections.

I think you are on the right track to where you need to force the speed down to 1gbps, and you'll probably also need to spoof the MAC address of the original modem too. Since the interface isn't coming up, I'm not sure how you will be able to enter the MAC address. I also don't have a 10gb card I can use for testing.

I would probably through a Windows evaluation version on there, try to do a firmware update on that device, and then go back to OPNsense. Alternate if you grab all the correct drivers and utilities, you can set up a Hiren's Boot CD https://www.hirensbootcd.org/ , drop the drivers into the DRIVER folder, and the utility somewhere on the bootable device. You should then be able to update things if there is a firmware update available.

Last thought is that the SFP module just isn't programmed to work with that interface, usually this is the interface blocking things. Not sure how to fix this unless you can find some generic firmware or a utility that allows you to "open it up" to all modules.

You know, using a switch isn't such a bad idea, it just acts as a distribution layer.  In fact, I can probably plug the SFP into my Unifi switch that has a SFP port and just see if it activates.  Then, just have to find the right switch (Mikrotik maybe?) to try with.  Basically a transparent interface between the fiber and the router. 

It is funny, the default install on the Qotom is Windows and I could have just finished the install and ran an update on all the firmware.  I will have to dig around for a firmware update for that particular chipset.

I was able to spoof the MAC address already, I can get into OPNsense easily enough on the LAN interface and change settings, which is why I found that the speed/duplex box is missing on this particular hardware.

The first time I tried this, I got an error about an unsupported device on the SFP port, and was able to fix it with "hw.ix.unsupported_sfp=1" is in the loader.conf.local file.  That pretty much opened up to any module, but, there may be more to it as well.

Now, time to go testing while my wife isn't using the internet!!!

[netnut]

It just feels "wrong" that my UDMP works with this module but OPNsense doesn't.

The first important thing, OPNsense doesn't (or does) support your SFP module, your NIC does. OPNsense (or FreeBSD) perfectly supports the Intel X553 network adapter, so if your module doesn't work, look at the adapter, not OPNsense.

Now your using the X5xx series from Intel, which is known to only support genuine Intel SFP's (hence the "hw.ix.unsupported_sfp=1" option), which is only found in the in-tree FreeBSD and Linux drivers. Third party SFP's shouldn't be a problem with this setting, but there are some specials like 10Gb RJ45 adapters (out-of-spec) and BiDi variants for AON Fiber like yours.

The other challenge is the Qotom device, got my hands on two of these (1U version) units a few weeks ago and both have problems with one of the SFP cages. The mini-pc and 1U versions have swapped fronts (so up&down is reverse for the two different models, see pictures on Qotom website for reference) and the SFP cages are _not_ numbered on the chassis (the copper ones are, but don't match OPNsense/FreeBSD assigment). If looking at the front of the 1U chassis the assignment in OPNsense 24.7 is this:

Code: [Select]

igc0 igc2 ix3 ix1

igc4 igc1 igc3 ix2 ix0

The ix3 port looks problematic for _any_ SFP with both the units I touched (supported or not), the interface is detected but the port won't power the module and so completly useless. I did read in several forums on the Interwebs (also this one), more people experiencing "one SFP isn't working" but lacks any details or orientation.

So back to your original question:

Is there any additional tests I can do?  Any additional settings I can try?  Maybe I am not waiting long enough for a connection? 

There're are a few things to check here: the SFP module, Qotom device (ix3 cage) and the Intel X553 NIC.

You're saying the module _does_ work in another device, so assume that's not your problem.

For the Qotom device, try and swap your SFP module in each and every SFP cage. Connect to OPNsense over SSH, select the "Shell" option and from the console type for every cage (interface) with the module inserted:

Code: [Select]

ifconfig -v "interface name"
With the -v option ifconfig on OPNsense shows you the module information (if correctly detected), something like this:

Code: [Select]

ix0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9216
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,HWSTATS,MEXTPG>
ether a1:b2:c3:d4:e5:f6
media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
drivername: ix0
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: Intel Corp PN: ABC123 SN: XYZ789 DATE: yyyy-mm-dd
module temperature: 0 C voltage: 0 Volts
lane 1: RX power: 0.00 mW (-0.00 dBm) TX bias: 0.00 mA

If you see the module correctly in the ifconfig output but it still doesn't work or you don't see the module info at all (which explains why it doesn't work right away) use your Joker -> the Intel X722 NIC. I'm more familiar with the Intel E810 series, but there's a bigger change it's less picky and/or will support more (third party) modules than your X533.

At this stage these are the most important task you can try to get to the root of your problem.

[shor0814]

Well, not sure what to think now.  Plugged the sfp into my Arista switch and no lights, I wasn't surprised.  Plugged into my 24 port Ubiquiti switch and lights.  Didn't have time to check for actual connectivity because the wife started asking "where is the internet" so I plugged back into the WAN port and.....nothing, at all.  I might have broke something.  Either burned out the sfp or maybe got the fiber dirty.  No go when plugging back into the modem either. So, technician scheduled for Monday and if he is the same guy as the install I might bend his ear a bit about solutions.

On the positive-ish side I might have the perfect switch, a Mikrotik CRS305.  4x10G SFP+ and 1 GBE management.  I know Mikeotik and Ubiquiti have both used the Annapurna Labs hardware.

I probably ought to ask about buying a couple of sfp's if it did burn out just in case.

[shor0814]

It just feels "wrong" that my UDMP works with this module but OPNsense doesn't.

The first important thing, OPNsense doesn't (or does) support your SFP module, your NIC does. OPNsense (or FreeBSD) perfectly supports the Intel X553 network adapter, so if your module doesn't work, look at the adapter, not OPNsense.

Now your using the X5xx series from Intel, which is known to only support genuine Intel SFP's (hence the "hw.ix.unsupported_sfp=1" option), which is only found in the in-tree FreeBSD and Linux drivers. Third party SFP's shouldn't be a problem with this setting, but there are some specials like 10Gb RJ45 adapters (out-of-spec) and BiDi variants for AON Fiber like yours.

The other challenge is the Qotom device, got my hands on two of these (1U version) units a few weeks ago and both have problems with one of the SFP cages. The mini-pc and 1U versions have swapped fronts (so up&down is reverse for the two different models, see pictures on Qotom website for reference) and the SFP cages are _not_ numbered on the chassis (the copper ones are, but don't match OPNsense/FreeBSD assigment). If looking at the front of the 1U chassis the assignment in OPNsense 24.7 is this:

Code: [Select]

igc0 igc2 ix3 ix1

igc4 igc1 igc3 ix2 ix0

The ix3 port looks problematic for _any_ SFP with both the units I touched (supported or not), the interface is detected but the port won't power the module and so completly useless. I did read in several forums on the Interwebs (also this one), more people experiencing "one SFP isn't working" but lacks any details or orientation.

So back to your original question:

Is there any additional tests I can do?  Any additional settings I can try?  Maybe I am not waiting long enough for a connection? 

There're are a few things to check here: the SFP module, Qotom device (ix3 cage) and the Intel X553 NIC.

You're saying the module _does_ work in another device, so assume that's not your problem.

For the Qotom device, try and swap your SFP module in each and every SFP cage. Connect to OPNsense over SSH, select the "Shell" option and from the console type for every cage (interface) with the module inserted:

Code: [Select]

ifconfig -v "interface name"
With the -v option ifconfig on OPNsense shows you the module information (if correctly detected), something like this:

Code: [Select]

ix0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9216
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,HWSTATS,MEXTPG>
ether a1:b2:c3:d4:e5:f6
media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
drivername: ix0
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: Intel Corp PN: ABC123 SN: XYZ789 DATE: yyyy-mm-dd
module temperature: 0 C voltage: 0 Volts
lane 1: RX power: 0.00 mW (-0.00 dBm) TX bias: 0.00 mA

If you see the module correctly in the ifconfig output but it still doesn't work or you don't see the module info at all (which explains why it doesn't work right away) use your Joker -> the Intel X722 NIC. I'm more familiar with the Intel E810 series, but there's a bigger change it's less picky and/or will support more (third party) modules than your X533.

At this stage these are the most important task you can try to get to the root of your problem.

Wasn't really blaming OPNsense, I know it is ultimately the NIC, just spouted off out of frustration.

Will try your suggestions, when I get a new module. 

I was aware of  mislabeled ports, and in order tonfigure out which was which, I connected my twinax sfp to each one to figure out the order.  Using the -v option will be a next step.

Speaking of ix3, I did notice at one point the dmesg output specifically said that ix3 had an unsupported module even with the unsupported_sfp setting and nothing actually plugged in.  Went away on a reboot.

The other 1U server is probably overkill, but, I can pick any pci based NIC. 

I will give it a shot soon.

[Greg_E]

The Microtik switch is probably pretty good, Serve The Home said good things about it.

[shor0814]

Tech came in today, changed the router out (now a Calix U6XW instead of U6X) and looks like he changed the module as well.  He wasn't very knowledgeable about networking in the general sense but got it up and working.

I saw him working on a setting or two when he was finalizing setup and I should have paid more attention to it since I think he put me on a VLAN.  So now, when I plug into my UDMPro, I get lights but no IP address and I am going to bet, I need a VLAN.  On the positive side, he left a few SFP's for my testing purposes.

I did talk to sales, specifically the business side, they indicate that it should be able to plug in to the my hardware just don't expect any support.  They are going to have an L2 tech give me a call just to see if we can make it work on the residential service.  I know it works, just not sure if they will give me the VLAN info.  Would suck to try 4000+ VLANs.

I get the new Mikrotik tomorrow, so I should at least be able to look for lights.  I will happily accept fiber->switch->OPNsense to avoid all issues with the Intel cards.  I did reach out to someone at FS.com to see what they have for Intel coded modules.

At least now, I know what modules I have:
100-05609............. GPON ONT SFP module so if anyone has suggestions I am all ears.

I have heard of GPON ONU on a stick but at this point tonight, I am too fried to research.  But if I am on a VLAN, I still will be stuck.

[Greg_E]

But, is your service tied to the MAC address of that specific module? Yes you can spoof the address, but this happens after the OS loads, so funny things could happen in this process.

[shor0814]

But, is your service tied to the MAC address of that specific module? Yes you can spoof the address, but this happens after the OS loads, so funny things could happen in this process.

Previously, before the new modem (and I assume VLAN) I was able to spoof my MAC address in my UDMPro and everything worked like a charm.  From what I could see, watching the tech, he was typing the new router's MAC address into the laptop to provision is (watched him looking at the box on the table), I don't think anything else changed.  The MAC he was reading was the MAC of the modem as verified in the admin section.

[shor0814]

So here is where I am now:

TLDR;  I have internet but not a working solution, and can a SFP+ port fry an SFP module?  Because I have a dead one after plugging into the Mikrotik CRS305.

Longer version

Tech was way behind schedule and was absolutely willing to work on helping with my direct connection but his first concern was that my modem was one of the original Calix that they don't use anymore and my install wasn't documented the way they want because it was one of the first installs.

He changed out the router to a newer model (Calix u6xw), which included a new SFP module.  This was fine as I figured I would just change the MAC address and at least be back to square 1.  The tech also handed me some extra SFP modules which was awesome of him.

He spent some time fighting with his software to update the MAC address to reflect the new modem and was struggling to provision the new one that he started to get even further behind than before he got there.  He suggested that I live with what I have for a few days and call support to help out.  I understand his situation and was perfectly fine with it.

After he left I gave the direct connection a quick shot, lights came on but never got an IP address.  It dawned on me that one thing I noticed, but didn't ask about was that in addition to the MAC address he also entered one more item which my guess right now is a VLAN.  I didn't catch the number unfortunately.

I am in touch with the tech support, their level 1 can't tell me anything about VLANs, but did verify the MAC address.  This was a Sunday so I have to wait until Monday or Tuesday to hear back.  She got a bit confused when I was talking about trying to avoid double NAT and bridge mode.

In the meantime, I got my Mikritok CRS305 and plugged one of the spare SFP modules in and the switchOS recognized it.  So this afternoon I plugged my fiber in and nothing, no lights etc. so I plugged it into the UDM and nothing, not lights or anything.  I realize now that maybe I need to drop the speed down to 1G in the SwitchOS?  I have read that you can fry an SFP via an SFP+ port due to laser power?

Thought better of it and put the fiber back in the ISP modem and waiting for their team to call me.

On the other side, I am working with FS.com to find a module.  They have a module that I can use and we have found some documentation that the SFP is 1310nm-TX and 1490nm-RX so they have a module that we can code as Intel.  They also suggest that I might try their GPON stick module with the MAC address coded in.  I may buy both the SFP and the GPON and a few cables that I need anyway.  Any thoughts on the GPON and the SFP?  I'm concerned that I have destroyed 2 SFP modules (or at least they seem destroyed) just by plugging into the SFP+ port.  At $10 each I won't cry over it but seems odd that they just stop working.

Thoughts are appreciated.

[Greg_E]

If the Mikrotik hardware is defective/damaged, then yes it could damage the SFP. Have you looked at the contacts on the SHP to see if any have been worn or lifted? Could be a connector issue too.

[shor0814]

If the Mikrotik hardware is defective/damaged, then yes it could damage the SFP. Have you looked at the contacts on the SHP to see if any have been worn or lifted? Could be a connector issue too.

I will go check that today.  This would be the 2nd module that went dead, the first was after I plugged it into either my Arista switch or the Ubiquiti switch (not dream machine).  I will double check the contacts to make sure there wasn't something hitting them, not sure I trust anything.

Not plugging anything in until I have time to talk to my ISP.

[Greg_E]

Have you looked at the user guide for the module? Sometimes these modules require a reboot before they are active. And some may require a power down before insertion. Might be worth putting one of the "dead" modules in and giving the hardware a reboot. I have had this a couple times with some older Cisco stuff, but newer hardware generally knows how to handle these modules.

[shor0814]

There doesn't seem to be a user guide for the Calix modules.  How do you mean a reboot for modules, never heard of doing that.  Unless you mean reboot the server with or without the SFP inserted?  I am open to anything because I find it hard to imagine that plugging into something that is supposed to be "backwards compatible" destroys it, that sounds wrong across the board.