Adding Speed Parameters to X550 Config

[spetrillo]

Hello all,

My firewall is running with a 2 port X550. I have these ports connected to a 2.5 gig switch. When I look at the switch these ports are running at 1 gig. In doing some research here I found that I need to augment the config of the X550 ports, to support 2.5 gig. Out of the gate the OS supports 100M, 1 gig, and 10 gig. I want to add support for 2.5 gig. I added sysctl dev.ix.0.advertise_speed=23 to /boot/loader.conf but it looks to get overwritten at OS boot time. The file's documentation tells me to add this to the tunables section in the GUI, so I did that and rebooted but my switch is still showing that the ports are running at 1 gig.

The tunables section has a type selection, which is boot-time, runtime, or environment. I have no ability to change this so my additions are running as environment. This could be the problem. Has anyone had success in get the X550 ports to support 2.5 gig?

Thanks,
Steve

[meyergru]

Those adapters cannot reliably auto-negotiate the intermediate speeds, and they cannot do this on other OSes, either.

This has been the case since 23.1 - potentially because of the underlying FreeBSD version, see https://github.com/opnsense/core/issues/6526

Thus, you can only do as much as advertise the capability on OpnSense and force the 2.5 Gpbs speed from the other side.

See also:

https://forum.opnsense.org/index.php?topic=33154.0

(and other topics in this forum)

[BrandyWine]

1st thing to try is what meyergru mentions, hard set both sides and see what you get.
After that, go see if there's some new firmware available from the Intel 34.0 bundle download.

I forget what driver is used for x550, but maybe try "dmesg |grep x550" and lets see where its at on nvm version.

[pfry]

Those adapters cannot reliably auto-negotiate the intermediate speeds, and they cannot do this on other OSes, either. [...]

Interestingly, Intel indicates "NBASE-T support in Linux Only", and while multi-gig/NBASE-T rates are listed in some places ("Data Rate Per Port 10GbE/5GbE/2.5GbE/1GbE/100Mb"), they are not in others (e.g. the adapter product brief).

[BrandyWine]

Interestingly, Intel indicates "NBASE-T support in Linux Only"

It's more weird than interesting. Probably has to do with linux driver, which gets more attention than any other.

Really need to go back and understand 802.3bz, because this spec has in it ability for "learning" best speed based on cable quality.
Is freeBSD driver as robust? Not sure.

Could it be switch firmware? Are there any updates available for that item?

Is your cable a quality twisted pair than is certified for 802.3bz? If not then get a certified cable is and see if that's better.

Hard to know for sure where/what is causing the issue. Try new cable and auto, try hard set, do you get your desired 2.5Gb?

Did you read this Intel doc https://cdrdv2.intel.com/v1/dl/getContent/335253
freeBSD 14.x is a supported OS

[pfry]

[...]
Really need to go back and understand 802.3bz, because this spec has in it ability for "learning" best speed based on cable quality.[...]

I'm not aware of any cable testing or training as part of any Ethernet link establishment - the link is simply negotiated to the highest common advertised speed (and not always successfully). Cable quality (assuming a cable that allows minimal communication between the endpoints) should not affect autonegotiation. Any testing and link speed reduction would be manual (as in the rest of your post). Heck, the only relevant testing I know of (without specialized equipment) is "looking at counters" (on both ends). (I suppose TDR can be useful if you don't know how long your cable is.)

I don't know of any good resources that cover later 802 standards (for less than IEEE prices).

[meyergru]

Folks, note, that I wrote: "It cannot handle auto-negotiation of NBASE-T", not: "it cannot handle them it all".

AFAIK, it can handle those speeds in any OS. With autonegotiation, it normally will only use 100, 1 or 10000 Mbps and that pertains also to Linux. That is even in the Intel docs:

# ethtool <ethX>

By default, devices based on the Intel(R) Ethernet Controller x550 do not
advertise 2.5 Gbps or 5 Gbps. To have your device advertise these speeds, use
the following:

# ethtool -s <ethX> advertise N

Where N is a combination of the following.
100baseTFull   0x008
1000baseTFull  0x020
2500baseTFull  0x800000000000
5000baseTFull  0x1000000000000
10000baseTFull 0x1000

For example, to turn on all modes:
# ethtool -s <ethX> advertise 0x1800000001028

For more details please refer to the ethtool man page.

NOTE: On Linux systems with INTERFACES(5), this can be specified as a pre-up
command in /etc/network/interfaces so that the interface is always brought up
with NBASE-T support. For example:

# iface <ethX> inet dhcp
    pre-up ethtool -s <ethX> advertise 0x1800000001028 || true

Caution: Only experienced network administrators should force speed and duplex
or change autonegotiation advertising manually. The settings at the switch must
always match the adapter settings. Adapter performance may suffer or your
adapter may not operate if you configure the adapter differently from your
switch.

In Linux, you can ask the driver to advertise those speeds, though. So, it is either that or disable autonegotitation on either side.

[meyergru]

BTW: The Intel ix driver has a section in it that reads:

if (hw->mac.type == ixgbe_mac_X550 &&
            hw->phy.autoneg_advertised == 0) {
            /*
            * 2.5G and 5G autonegotiation speeds on X550
            * are disabled by default due to reported
            * interoperability issues with some switches.
            *
            * The second condition checks if any operations
            * involving setting autonegotiation speeds have
            * been performed prior to this ixgbe_config_link()
            * call.
            *
            * If hw->phy.autoneg_advertised does not
            * equal 0, this means that the user might have
            * set autonegotiation speeds via the sysctl
            * before bringing the interface up. In this
            * case, we should not disable 2.5G and 5G
            * since that speeds might be selected by the
            * user.
            *
            * Otherwise (i.e. if hw->phy.autoneg_advertised
            * is set to 0), it is the first time we set
            * autonegotiation preferences and the default
            * set of speeds should exclude 2.5G and 5G.
            */
            autoneg &= ~(IXGBE_LINK_SPEED_2_5GB_FULL |
                IXGBE_LINK_SPEED_5GB_FULL);
        }

The FreeBSD ix docs are quite unhelpful as to what is to be specified to get the correct advertisements, so:

Code Select Expand

#define IXGBE_LINK_SPEED_UNKNOWN 0
#define IXGBE_LINK_SPEED_10_FULL 0x0002
#define IXGBE_LINK_SPEED_100_FULL 0x0008
#define IXGBE_LINK_SPEED_1GB_FULL 0x0020
#define IXGBE_LINK_SPEED_2_5GB_FULL 0x0400
#define IXGBE_LINK_SPEED_5GB_FULL 0x0800
#define IXGBE_LINK_SPEED_10GB_FULL 0x0080
#define IXGBE_LINK_SPEED_82598_AUTONEG (IXGBE_LINK_SPEED_1GB_FULL | \
IXGBE_LINK_SPEED_10GB_FULL)
#define IXGBE_LINK_SPEED_82599_AUTONEG (IXGBE_LINK_SPEED_100_FULL | \
IXGBE_LINK_SPEED_1GB_FULL | \
IXGBE_LINK_SPEED_10GB_FULL)

Thus, I assume that "hw.ix.advertise_speed = 0x0CAA" or "hw.ix.advertise_speed = 3242" might do the trick, but I haven't tried.