Problem with the OPNsense DEC750 and SFP

[Gilad]

Hi,

It seems like something is not working when I'm using SFP adapters with the DEC750...

The DEC750 (latest software version) is connected with SFP (x0) to a Cisco L2 switch.

The WAN interface is coming up, but I can't send or receive any traffic. The only way to get it working is to change something in the WAN interface settings (like MTU), and then the connection is back to life.

Before the MTU change I can't even see the MAC address entry on the Cisco...

I'm using a standard Cisco RJ45 SFTP (I've tried several). The following is the log on the console when inserting the FSP:

Code Select Expand

ax0: xgbe_phy_sfp_signals: port_sfp_inputs: 0x0
ax0: SFP detected:
ax0:   vendor:   CISCO-AVAGO     
ax0:   part number:    ABCU-5710RZ-CS4
ax0:   revision level: B2 
ax0:   serial number:  AGM165220FH     
miibus0: <MII bus> on ax0
e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 0 on miibus0
e1000phy0:  none, 1000baseSX, 1000baseSX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
ax0: Link is UP - 1Gbps/Full - flow control off


Any ideas?

[Gilad]

Update: I've tried another Cisco switch (different model), but still the same issue.

Everything working as expected when I'm using the DEC750 built-in Ethernet port.

Is it possible that the problem is just compatibility between the SFP+ port and the SFP transmitter?...
I've ordered a SFP+ 1G/2.5G/5G/10G transmitter that should arrive today, I'll update tomorrow.

[lilsense]

I believe 10Gig SFP+ ports only work with 10Gig SFP+ and not SFP. you should use the copper 1Gig ports.

[meyergru]

I had a SFP module running, however, it needed to set the speed manually.

[fre4ki]

I think so too. Maybe you have to set speed and duplex to Gigabit and Full-Duplex.

If you have Multimode you can also check the red light of the transceiver (incoming and outgoing).

[olmari]

I use DEC750 with 1gbps SPF-DAC cable to connect onto HP/Aruba 1gbps switch with "programmable" flexoptics cable. 1gbps works just fine with DEV750, but indeed I had to, in this particular case, "program" both ends as 1gpbs modules, leaving either/both end as 10gpbs did not work even when manually selecting speeds.

That in mind, plus what is said in https://shop.opnsense.com/product/dec750-opnsense-desktop-security-appliance/ directly too, generally SFP/1gbps modules should work, but just neds to be same in both ends.

[Greg_E]

SFP28, SFP+ are backwards compatible to a certain extent. As mentioned, you may need to turn auto negotiation off and set the speed manually, I need to do this on non-Cisco modules in 2960s switches and also on my Mikrotik. In the case of the Mikrotik, it is a 10gbps per port switch and I'm running several 1gbps modules. Just needed to set the speed to 1gbps manually and the interfaces popped to life.

[Splodge]

Ive had no end of issues with DEC750 and 1gb SFP / DAC cables late 2023/24, and ended up engaging support as it was still under warranty.
Issues seem the same as this - SFP L2 comes up but no traffic flows; stop/starting the interface will work until the next power cycle, when it dropped again.
Happened with the known working FS 1g SFP on the product page, as well as various Cisco SFP & DAC cables.
Affected both WAN (1gb FS SFP to 1gb ONT) and LAN 1gb DAC to switch SFP - both Cisco and Ubiquity.

Strange variation is if you factory reset the device into default config, it works as expected, its only once you create your own config the issue arises...
Support were going to discuss internally as they think its something with the timings on SFP initialisation.

Workaround they gave me is to create a file in /usr/local/etc/rc.syshook.d/start

called 80-axgbe

containing

#!/bin/sh
ifconfig ax0 down && ifconfig ax0 up
ifconfig ax1 down && ifconfig ax1 up
configctl interface reconfigure wan

make sure its executable with chmod +x 80-axgbe

then every reboot, the interfaces get forced down/up to reset the SFP.

Seems to work, but not sure I totally trust it to come back up in all circumstances, so I'm using it at home now...

[oklahoma]

Hi!
I got the same problem with a brand new DEC2752 machine. As advised in your solution, I put the 80-axgbe file in the folder. But the problem still persists. If I call the file manually, it solves the problem. The problem is also gone if I release the cable, plug it in again and run option 11 "Reload all services". Or another solution after a reboot is to run option 1 "Assign interfaces". But all of those is not really solving the problem. I hope there is a final solution....

[swb]

Any resolution on this one?

Dual DEC2750 with compatible FS SFP-GB-GE-T, both unable to communicate, was working when initially set up

[Patrick M. Hausen]

SFP copper is notoriously unreliable. With two DEC2752 I recommend getting fiber SFP+ and a fiber patch cable. Why are you using 1G SFP in these ports, anyway?

[swb]

For a backup ether WAN connection.

[Greg_E]

SFP copper is notoriously unreliable.

Ugg... I have a Windows device that has a 10gbps copper connection and no where to replace this with fiber or a DAC cable, the device is literally a foot of cable away from the switch port mounted in a rolling rack. We haven't used it enough to know if the SFP+ module I bought to go in the switch is going to be worth using. This might be fun (not). I do know that I can get 3 cameras at 140mbps each into this device without causing issues, so maybe I'm OK. Yes it's a video production system in a box that uses (among other things) NDI to move the streams in and out.

[Patrick M. Hausen]

Main problem is they get awfully hot. Make sure you have proper airflow in that location. And if the switch management allows that check the SFP+ temperature.

[Greg_E]

Heat I have noticed, hotter than optical and of course way hotter than DAC.