Adding Speed Parameters to X550 Config

[spetrillo]

Hello all,

My firewall is running with a 2 port X550. I have these ports connected to a 2.5 gig switch. When I look at the switch these ports are running at 1 gig. In doing some research here I found that I need to augment the config of the X550 ports, to support 2.5 gig. Out of the gate the OS supports 100M, 1 gig, and 10 gig. I want to add support for 2.5 gig. I added sysctl dev.ix.0.advertise_speed=23 to /boot/loader.conf but it looks to get overwritten at OS boot time. The file's documentation tells me to add this to the tunables section in the GUI, so I did that and rebooted but my switch is still showing that the ports are running at 1 gig.

The tunables section has a type selection, which is boot-time, runtime, or environment. I have no ability to change this so my additions are running as environment. This could be the problem. Has anyone had success in get the X550 ports to support 2.5 gig?

Thanks,
Steve

[meyergru]

Those adapters cannot reliably auto-negotiate the intermediate speeds, and they cannot do this on other OSes, either.

This has been the case since 23.1 - potentially because of the underlying FreeBSD version, see https://github.com/opnsense/core/issues/6526

Thus, you can only do as much as advertise the capability on OpnSense and force the 2.5 Gpbs speed from the other side.

See also:

https://forum.opnsense.org/index.php?topic=33154.0

(and other topics in this forum)

[BrandyWine]

1st thing to try is what meyergru mentions, hard set both sides and see what you get.
After that, go see if there's some new firmware available from the Intel 34.0 bundle download.

I forget what driver is used for x550, but maybe try "dmesg |grep x550" and lets see where its at on nvm version.

[pfry]

Those adapters cannot reliably auto-negotiate the intermediate speeds, and they cannot do this on other OSes, either. [...]

Interestingly, Intel indicates "NBASE-T support in Linux Only", and while multi-gig/NBASE-T rates are listed in some places ("Data Rate Per Port 10GbE/5GbE/2.5GbE/1GbE/100Mb"), they are not in others (e.g. the adapter product brief).

[BrandyWine]

Interestingly, Intel indicates "NBASE-T support in Linux Only"

It's more weird than interesting. Probably has to do with linux driver, which gets more attention than any other.

Really need to go back and understand 802.3bz, because this spec has in it ability for "learning" best speed based on cable quality.
Is freeBSD driver as robust? Not sure.

Could it be switch firmware? Are there any updates available for that item?

Is your cable a quality twisted pair than is certified for 802.3bz? If not then get a certified cable is and see if that's better.

Hard to know for sure where/what is causing the issue. Try new cable and auto, try hard set, do you get your desired 2.5Gb?

Did you read this Intel doc https://cdrdv2.intel.com/v1/dl/getContent/335253
freeBSD 14.x is a supported OS

[pfry]

[...]
Really need to go back and understand 802.3bz, because this spec has in it ability for "learning" best speed based on cable quality.[...]

I'm not aware of any cable testing or training as part of any Ethernet link establishment - the link is simply negotiated to the highest common advertised speed (and not always successfully). Cable quality (assuming a cable that allows minimal communication between the endpoints) should not affect autonegotiation. Any testing and link speed reduction would be manual (as in the rest of your post). Heck, the only relevant testing I know of (without specialized equipment) is "looking at counters" (on both ends). (I suppose TDR can be useful if you don't know how long your cable is.)

I don't know of any good resources that cover later 802 standards (for less than IEEE prices).

[meyergru]

Folks, note, that I wrote: "It cannot handle auto-negotiation of NBASE-T", not: "it cannot handle them it all".

AFAIK, it can handle those speeds in any OS. With autonegotiation, it normally will only use 100, 1 or 10000 Mbps and that pertains also to Linux. That is even in the Intel docs:

# ethtool <ethX>

By default, devices based on the Intel(R) Ethernet Controller x550 do not
advertise 2.5 Gbps or 5 Gbps. To have your device advertise these speeds, use
the following:

# ethtool -s <ethX> advertise N

Where N is a combination of the following.
100baseTFull   0x008
1000baseTFull  0x020
2500baseTFull  0x800000000000
5000baseTFull  0x1000000000000
10000baseTFull 0x1000

For example, to turn on all modes:
# ethtool -s <ethX> advertise 0x1800000001028

For more details please refer to the ethtool man page.

NOTE: On Linux systems with INTERFACES(5), this can be specified as a pre-up
command in /etc/network/interfaces so that the interface is always brought up
with NBASE-T support. For example:

# iface <ethX> inet dhcp
    pre-up ethtool -s <ethX> advertise 0x1800000001028 || true

Caution: Only experienced network administrators should force speed and duplex
or change autonegotiation advertising manually. The settings at the switch must
always match the adapter settings. Adapter performance may suffer or your
adapter may not operate if you configure the adapter differently from your
switch.

In Linux, you can ask the driver to advertise those speeds, though. So, it is either that or disable autonegotitation on either side.

[meyergru]

BTW: The FreeBSD Intel ix driver has a section in it that reads:

if (hw->mac.type == ixgbe_mac_X550 &&
            hw->phy.autoneg_advertised == 0) {
            /*
            * 2.5G and 5G autonegotiation speeds on X550
            * are disabled by default due to reported
            * interoperability issues with some switches.
            *
            * The second condition checks if any operations
            * involving setting autonegotiation speeds have
            * been performed prior to this ixgbe_config_link()
            * call.
            *
            * If hw->phy.autoneg_advertised does not
            * equal 0, this means that the user might have
            * set autonegotiation speeds via the sysctl
            * before bringing the interface up. In this
            * case, we should not disable 2.5G and 5G
            * since that speeds might be selected by the
            * user.
            *
            * Otherwise (i.e. if hw->phy.autoneg_advertised
            * is set to 0), it is the first time we set
            * autonegotiation preferences and the default
            * set of speeds should exclude 2.5G and 5G.
            */
            autoneg &= ~(IXGBE_LINK_SPEED_2_5GB_FULL |
                IXGBE_LINK_SPEED_5GB_FULL);
        }

The FreeBSD ix docs are quite unhelpful as to what is to be specified to get the correct advertisements, so:

Code Select Expand

#define IXGBE_LINK_SPEED_UNKNOWN 0
#define IXGBE_LINK_SPEED_10_FULL 0x0002
#define IXGBE_LINK_SPEED_100_FULL 0x0008
#define IXGBE_LINK_SPEED_1GB_FULL 0x0020
#define IXGBE_LINK_SPEED_2_5GB_FULL 0x0400
#define IXGBE_LINK_SPEED_5GB_FULL 0x0800
#define IXGBE_LINK_SPEED_10GB_FULL 0x0080
#define IXGBE_LINK_SPEED_82598_AUTONEG (IXGBE_LINK_SPEED_1GB_FULL | \
IXGBE_LINK_SPEED_10GB_FULL)
#define IXGBE_LINK_SPEED_82599_AUTONEG (IXGBE_LINK_SPEED_100_FULL | \
IXGBE_LINK_SPEED_1GB_FULL | \
IXGBE_LINK_SPEED_10GB_FULL)

Thus, I assume that "hw.ix.advertise_speed = 0x0CAA" or "hw.ix.advertise_speed = 3242" might do the trick, but I haven't tried.

[BrandyWine]

Thus, I assume that "hw.ix.advertise_speed = 0x0CAA" or "hw.ix.advertise_speed = 3242" might do the trick, but I haven't tried.

That sysctl to define what the default advertise speed is for all, should work, but I think you add up all the values 1st.  (https://man.freebsd.org/cgi/man.cgi?ix)

There's another per-nic setting to look at, dev.ix.<X>.advertise_speed
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#intel-ix-4-cards

It looks like you add up the hex values for each speed you want, that total goes into the tunable.

For example, to advertise speeds of 1G, 2.5G, 5G, and 10G, add their corresponding values: 0x2+0x10+0x20+0x4 (hex) or 2+16+32+4 (decimal) for a total of 0x36 (hex) or 54 (decimal). Values in hexadecimal must start with 0x.

[meyergru]

That could be, although I read somewhere that Netgate uses their own driver for ix, so IDK if YMMV with OpnSense.

Also note that the hex values for pfSense differ from what can be read from the FreeBSD ix driver source code. I added up all the values from the constants contained in there to get at 0x0CAA hex / 3242 decimal.

[BrandyWine]

That could be, although I read somewhere that Netgate uses their own driver for ix, so IDK if YMMV with OpnSense.

Also note that the hex values for pfSense differ from what can be read from the FreeBSD ix driver source code. I added up all the values from the constants contained in there to get at 0x0CAA hex / 3242 decimal.

Good info.

And look at that chart on netgate page, kinda odd, the text mentions numbers but those numbers are not in the chart.

If possible, use the per-nic sysctl on a unused nic, then cable it up and see what it does.

[spetrillo]

So here is the problem that I see...

First off if you read the text at the top of /boot/loader.conf it says the file is autogenerated and to use /boot/loader.conf.local. I did that and entered two parameter lines: sysctl dev.ix.0.advertise_speed=23 and sysctl dev.ix.1.advertise_speed=23. This tells both ports that it can support 2.5 gig. It does not seem this gets loaded after a reboot bc when I run sysctl dev.ix.0.advertise_speed and sysctl dev.ix.1.advertise_speed they both output 7, which is the default to support 100M, 1G, and 10G.

I just tried hex instead of decimal and that made no change. I think the first thing is to understand if using loader.conf.local works bc right now it does not seem to be working. Second I am going to remove the changes and try to force it on the switch side and see what happens. More to come.

[Patrick M. Hausen]

In loader.conf or loader.conf.local you just write the variables and their values without the "sysctl" in front.

Better use the UI: System > Settings > Tunables.

[pfry]

[...]
Thus, I assume that "hw.ix.advertise_speed = 0x0CAA" or "hw.ix.advertise_speed = 3242" might do the trick, but I haven't tried.

Nice. Nitpick: I'd probably reset bit 1 (10BASE-T) on ix, as some of Intel's docs suggest that it's not supported. Of course, you could always try it. I don't know if anyone would need/want to set bit 2 (100BASE-TX half-duplex), but it's there.

As far as the "make the special case general" in the driver: Grrr.

[meyergru]

@spetrillo: What makes you believe that 23 is the correct value? Do you have any reliable source for that assumption?

It is neither for the Netgate version according to their docs nor is it according to the source code of the ix driver in FreeBSD, which should be the one you use in OpnSense. The value 0x0CAA (or 3242 decimal) will enable autonegotiation for all speeds, 23 is most probably incorrect, IMHO.