OpnSense SFP+ connection to 2g fiber ONT is limited to 1g.

[cologuy]

I have a Watchguard M470 with a 4 port SFP+ module running OpnSense 25.7.6 and it's been running great for more than a year but I recently upgraded to 2g fiber (was 1g before) and I'm capped at 1g.

I'm able to connect from a PC with a 2.5g NIC to the fiber ONT and get 2.1+g download/upload speeds but connecting from my SFP+ port on the M470 to the ONT I seem to be capped at 1g (950mbs).

I have the interface speed set to 10G-BASE-SR and ifconfig from the opnsense shell shows 10Gbase-SR.

I'm using a 10GTek ASF-10G2-T optic to connect from the M470 SFP+ port to the 10g RJ-45 port on the ONT through a 12in CAT7 cable. The 10gTek does allow 1g/2.5/5/10g connections but everything seems to be linked at 10g.

I checked my PC to switch link speed and it's 10g full-duplex. What else can I look for?

Here is the ifconfig from OpnSense shell:

ix3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WAN (opt2) options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,HWSTATS,MEXTPG> inet 38.xx.xx.xx netmask 0xffffff00 broadcast 38..xx.xx.255 media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

[meyergru]

Did you try RSS as explained here, point #10?

Also, is any type of IDS active? The Watchguard M470 only has an Intel G4400, which is fairly slow by today's standards. A simple N100 beats it by more than a factor of two.

[cologuy]

Interesting. I have a Intel Xeon E3-1260L v5 that I could try with it. Roughly 30% faster than the N100.

The M470 is rated at 450 users and 19.6G throughput but it's worth a try. Watchguard M470

It's also worth noting that the download is consistent around 950Mbs or the 1g limit.

[cologuy]

Just upgraded my M470 to the Xeon E3-1260L V5 and got the same results. CPU usage was less than 25% on the G4400 and about 19% on the 1260L during the
speed test.

[ProximusAl]

Not that this helps you much, but I also have a Watchguard M470 with a 4 port SFP+ module and I'm using it as designed, so its running Fireware 12.11.4.

It's easily pushing 5Gbps through the SFP+ without a sweat, and I use the Cisco SFP-10G-SR Compatible 10GBASE-SR SFP+ 850nm 300m DOM Duplex LC/UPC MMF Optical Transceiver Module from FS.com.

So the device itself is capable of those speeds.

[meyergru]

If the cap is exactly at 1 Gbps, it is more likely that the SFP+ module connects only at 1 Gbps to the ONT. It happens often that SFP+ modules connect at their highest specified speeds to the host (in your case 10 Gbps), yet use lower speeds on the real link. For example, some SFP+ slots only support 1/10 Gbps, so only those can be reported. The real link speed is up to the SPF+ module and you cannot choose it from the host.

That is true of the ax and ix drivers.

[cologuy]

Thanks, that is actually really helpful. It's good to know it's capable, I just need to get all the details right.

So the M470 running the Firmware 12.11.4 Watchguard software?

Does this module look like the same part as yours?

Not that this helps you much, but I also have a Watchguard M470 with a 4 port SFP+ module and I'm using it as designed, so its running Fireware 12.11.4.

It's easily pushing 5Gbps through the SFP+ without a sweat, and I use the Cisco SFP-10G-SR Compatible 10GBASE-SR SFP+ 850nm 300m DOM Duplex LC/UPC MMF Optical Transceiver Module from FS.com.

So the device itself is capable of those speeds.

[meyergru]

Even if it was, @ProximusAl is using a pure optic SFP+ module which always has a 10 Gbps link speed.

Your problem is different: You use a transceiver that can theoretically handle different Ethernet speeds on the link side, yet always reports 10 Gbps to your host. Also, you cannot force a link speed on your side because of this, so you are stuck with whatever link speed is automatically negotiated.

Even if both sides can theoretically do 2.5 Gbps over ethernet, sometimes it is not auto-negotiated. This is even true for some non-SFP+ ethernet adapters, like the Intel X550, where 2.5 Gbps must be forced.

Alas, you probably cannot force the specific speed (2.5 Gbps) from the ONT side of the link, either.

[lunch]

If the cap is exactly at 1 Gbps, it is more likely that the SFP+ module connects only at 1 Gbps to the ONT. It happens often that SFP+ modules connect at their highest specified speeds to the host (in your case 10 Gbps), yet use lower speeds on the real link. For example, some SFP+ slots only support 1/10 Gbps, so only those can be reported. The real link speed is up to the SPF+ module and you cannot choose it from the host.

That is true of the ax and ix drivers.

I'm chiming in as I have been experiencing a similar issue (however, I am able to fallback on hardware that poses no issues, for now). I received a new modem from my ISP, a Sagemcom Fast 5697, and my connection from my hardware to the modem's 10g port caps my speed to a strange 5-600mbps down and ~1gbps up. This same connection to the old modem, a Sagemcom Fast 5689e, resulted in my full connection speed: 1.5 down/1 up.

Connecting via Intel x520-DA2, 10Gtek 1.25/2.5/5/10G-T SFP+ to RJ45 CAT.6a transceiver and CAT6 cable.

Is it possible the 10g port on this new device is for some reason incompatible with my transceiver? And if so, is that something that can be fixed via firmware or do such hardware incompatibilites exist that it can't be remedied outside of a hardware change?

[ProximusAl]

Does this module look like the same part as yours?

It is indeed that exact module that I use.....
I use MMF Fibre between the Firebox and my 10G Unifi switches. 
My use of OPNSense is actually *above* the M470, using the exact same modules and fibre leads.

Dont know if this will work, but here is a picture:
https://ibb.co/B5m6xxHG
If that doesnt work try this:
https://postimg.cc/wyc7sZJH

[meyergru]

At these speeds, problems can occur for a multitude of reasons, bad cabling being one of them. As I said, some devices cannot auto-negotiate, thus they may fall back to 1 Gbps.

But no, I do not know if anything can be done via firmware. I found that especially ethernet SFP+ transceivers are problematic - for this reason and also, because they get very hot with higher speeds. I use SFP+ only with DAC, or if I needed longer cabling, I would use optics transveivers.

I could not even use GPON SFP transceivers in a meaningful way, because most host adapters do not support HSGMII mode with 2.5 Gbps (or they cannot mix 10 and 2.5 Gbps speeds, like the two ax port on the DEC750, so I was capped at 1 Gbps there, too.

[lunch]

At these speeds, problems can occur for a multitude of reasons, bad cabling being one of them. As I said, some devices cannot auto-negotiate, thus they may fall back to 1 Gbps.

But no, I do not know if anything can be done via firmware. I found that especially ethernet SFP+ transceivers are problematic - for this reason and also, because they get very hot with higher speeds. I use SFP+ only with DAC, or if I needed longer cabling, I would use optics transveivers.

I could not even use GPON SFP transceivers in a meaningful way, because most host adapters do not support HSGMII mode with 2.5 Gbps (or they cannot mix 10 and 2.5 Gbps speeds, like the two ax port on the DEC750, so I was capped at 1 Gbps there, too.

Ah, okay, well that's unfortunate. I cannot bypass my ISP's hardware, so I have no ability to use a DAC between my OPNsense machine and the modem--I can only use a SFP+ RJ45 transceiver and ethernet. I know they can be problematic, but I unfortunately have no other choice here. And the problem is that this new modem is their new hardware, so should my current model die (I just replaced a dead one) then I'll be facing this problem again in the future as it will be the only replacement offered.

It feels like whack-a-mole having to buy and hope, but is it possible another transceiver could play nicer? Just trying to understand my options for the future.

[meyergru]

If you have a managed switch with both DAC and 2.5 GbE ports, you can use a DAC to connect the switch and your OpnSense and use the 2.5 GbE port to connect to your ONT. By using a VLAN you can separate the WAN traffic from your other (V)LANs. That is even possible with just one DAC connection between the switch and OpnSense, because you can use it for all VLANs.

Kind of a "router on a stick" configuration.

On the switch, you often can set a fixed speed for ethernet ports and the DAC connection is 10 Gbps anyway.

An alternative to this is an OpnSense device with 2.5 Gbps enthernet ports (and potentially, SFP+ for future-proofing), like I use one.

[lunch]

Okay, great--will keep this in mind! I don't currently have a managed switch with an SFP+ port (managed switch without, unfortunately). But it seems like the less finnicky option for the future. Appreciate it.

[cologuy]

Nice network rack! Here is my network closet rack setup with a fail over multi-WAN setup using 2 M470's. It's a homelab but I work from home so it's more business than homelab.

I have a 40g MMF link via the Brocade 7450-24p switch to my server rack in another part of the house so I can have multiple 10g streams between my office workstations and servers and the server rack. I also have a 10g switch on my desk to connect 10g workstation and any servers that I'm provisioning.

I ordered a 10g only RJ45 module from Amazon. I'm hoping that will take care of it but I have the FS module bookmarked as a backup.

>>My use of OPNSense is actually *above* the M470, using the exact same modules and fibre leads.

Can you expand on this? Do you mean *above* physically in the picture?

Thanks for the input.

Does this module look like the same part as yours?

It is indeed that exact module that I use.....
I use MMF Fibre between the Firebox and my 10G Unifi switches. 
My use of OPNSense is actually *above* the M470, using the exact same modules and fibre leads.

Dont know if this will work, but here is a picture:
https://ibb.co/B5m6xxHG
If that doesnt work try this:
https://postimg.cc/wyc7sZJH