E610-XT2

[BrandyWine]

For now I am going to open a Git request and see if there is any appetite to port E610 support into the 14.3 software.

Well, if you start a dev env, probably easiest to install a v15 VM, then git clone the 14.3 src tree.
From there things are much more native for compiling, you just need to compile against the 14.3 kernel source.

As for port, well, the E610 code is 1) in 15-stable, and 2) not even sure if the E610 code was completed yet. So perhaps porting from -stable is on-par with just compiling what's there now.

[BrandyWine]

I have a e610 driver ko, but I cannot load it, kernel thinks it's the same driver as one already in the kernel. Mine compiles out as "ix2".
It's not easy trying to just compile out a ko just for the 610, because they wrap sets of device ID's into the ixbge code, and if you don't remove all the other device ID's the kernel will read the driver and try and bind ("attach") the driver code to the device ID's it finds on pci. We only want the 610 parts so we can load it in as it's own ko just for the 610. I managed to rename the driver as ibgxe_if which will attach to pci bus, but there's some other ID that needs to change, probably in an entrypoint section.

The e610 code is weird, looks like they share code from x550 because the e610 code pulls in a x550 h file.

I see some e610 code floating around on git, but need to hunt down actual Intel src code, might make it easier to build a sole 610 ko that can load up without issue, the 14.3 ixbe does not include 610 code, so a sperate 610 ko would be ideal as a temp fix for running that card on 14.3.


The latest e610 driver is in the Intel 30.4.2 download bundle (in igbe ix), Intel's version of it is 3.4.31.
Make it easy to compile down an ix.ko, but that's still an issue because as I mentioned, ix has other devices in it, and, it's already in the 14.3 kernel being used by opnsense.


I think to avoid this in the future it might be better to compile kernel w/o all the driver code, and then just include all the common drivers as klm's via loader.conf, this way we could load out/in whatever we wanted to with ease.

I am testing something now on my opnsense. edit: which shows me the if_ix.ko is just a file not being used.
See, OPNsense uses the GENERIC setup for kernel build, and GENERIC leans monolithic.

[pfry]

I have a e610 driver ko, but I cannot load it, kernel thinks it's the same driver as one already in the kernel.[...]

I thought FreeBSD was fully modular, that is I'd expect you could kldunload the existing driver and kldload yours. Of course, I haven't tried it.

[Patrick M. Hausen]

It isn't. Most drivers are compiled statically and cannot be un/loaded.

[BrandyWine]

It isn't. Most drivers are compiled statically and cannot be un/loaded.

In Linux I can write to a unbind of a device and the kernel will process that and unbind the driver.
In freeBSD can't do that.

I did some more reading last night to see another path, and on all the nix forums it's suggested to recompile kernel without ix, so that's what I have done. I will now compile down the Intel 610 driver code and load that in as klm.

[BrandyWine]

Success on kernel build while excluding the pci/ix completely.
Next is to compile the Intel 610 src. Trying now.
Success on kldload and kldunload of the ix ko from 14.3 (opnsense 25.7).
Working on Intel compile.
Edit. Intel driver compiled a-ok, i can kldload and kldunload that driver.

So, who wants to test kernel and this Intel ix? Nobody should trust my files, but they are test worthy to see if the kernel will bind driver to 610 nic w/o err. Intel ix man4 attached.

I made a new dev in tree (so not to muck around with ix)
/usr/src/14.3-RELEASE/sys/dev/ix610/
untar the Intel src to ix610 (or whatever you name it)
modify Makefile to understand the src tree

comment added, next two lines added

Code Select Expand

#.PATH:  ${.CURDIR}
.PATH: ${SRCTOP}/sys/dev/ix610
.PATH: ${SRCTOP}/sys/${MACHINE_CPUARCH}/${MACHINE_CPUARCH}
Setup env

Code Select Expand

SRCTOP=/usr/src/14.3-RELEASE
export SRCTOP
Move existing ix, build

Code Select Expand

mv /boot/kernel/if_ix.ko /boot/kernel/if_ix.ko.orig
cd $SRCTOP/sys/dev/ix610
make
make install
The new ix is now in /boot/kernel, but you cannot load it until you swap out kernel and reboot system

Intel readme

Code Select Expand

ix FreeBSD* Driver for Intel(R) Ethernet 10 Gigabit PCI Express Server Adapters
*******************************************************************************

July 17, 2025


Contents
^^^^^^^^

* ix FreeBSD* Driver for Intel(R) Ethernet 10 Gigabit PCI Express
  Server Adapters

  * Overview

  * Related Documentation

  * Identifying Your Adapter

  * The VF Driver

  * Building and Installation

  * Configuration and Tuning

  * Known Issues/Troubleshooting

  * Support

  * Trademarks


Overview
========

This file describes the FreeBSD* driver for Intel(R) Ethernet. This
driver has been developed for use with all community-supported
versions of FreeBSD.

For questions related to hardware requirements, refer to the
documentation supplied with your Intel Ethernet Adapter. All hardware
requirements listed apply to use with FreeBSD.

Note:

  Devices based on the Intel(R) Ethernet Controller X552 and Intel(R)
  Ethernet Controller X553 do not support the following features:

  * Low Latency Interrupts (LLI)

The associated Virtual Function (VF) driver for this driver is ixv.


Related Documentation
=====================

See the "Intel(R) Ethernet Adapters and Devices User Guide" for
additional information on features. It is available on the Intel
website at either of the following:

* https://cdrdv2.intel.com/v1/dl/getContent/705831

* https://www.intel.com/content/www/us/en/download/19373/adapter-user-
  guide-for-intel-ethernet-adapters.html


Identifying Your Adapter
========================

The driver is compatible with devices based on the following:

* Intel(R) Ethernet Controller 82598

* Intel(R) Ethernet Controller 82599

* Intel(R) Ethernet Controller X520

* Intel(R) Ethernet Controller X540

* Intel(R) Ethernet Controller x550

* Intel(R) Ethernet Controller X552

* Intel(R) Ethernet Controller X553

* Intel(R) Ethernet Controller E610

For information on how to identify your adapter, and for the latest
Intel network drivers, refer to the Intel Support website at
http://www.intel.com/support.


SFP+ Devices with Pluggable Optics
----------------------------------


82599-Based Adapters
~~~~~~~~~~~~~~~~~~~~

Note:

  * If your 82599-based Intel(R) Network Adapter came with Intel
    optics or is an Intel(R) Ethernet Server Adapter X520-2, then it
    only supports Intel optics and/or the direct attach cables listed
    below.

  * When 82599-based SFP+ devices are connected back to back, they
    should be set to the same Speed setting via ethtool. Results may
    vary if you mix speed settings.

+-----------------------------------+-----------------------------------+-----------------------------------+
| Supplier                          | Type                              | Part Numbers                      |
|===================================|===================================|===================================|
| **SR Modules:**                  |                                  |                                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel                            | DUAL RATE 1G/10G SFP+ SR (bailed) | FTLX8571D3BCV-IT                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel                            | DUAL RATE 1G/10G SFP+ SR (bailed) | AFBR-703SDZ-IN2                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel                            | DUAL RATE 1G/10G SFP+ SR (bailed) | AFBR-703SDDZ-IN1                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| **LR Modules:**                  |                                  |                                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel                            | DUAL RATE 1G/10G SFP+ LR (bailed) | FTLX1471D3BCV-IT                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel                            | DUAL RATE 1G/10G SFP+ LR (bailed) | AFCT-701SDZ-IN2                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel                            | DUAL RATE 1G/10G SFP+ LR (bailed) | AFCT-701SDDZ-IN1                  |
+-----------------------------------+-----------------------------------+-----------------------------------+

The following is a list of third-party SFP+ modules that have received
some testing. Not all modules are applicable to all devices.

+-----------------------------------+-----------------------------------+-----------------------------------+
| Supplier                          | Type                              | Part Numbers                      |
|===================================|===================================|===================================|
| Finisar                          | SFP+ SR bailed, 10g single rate  | FTLX8571D3BCL                    |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Avago                            | SFP+ SR bailed, 10g single rate  | AFBR-700SDZ                      |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Finisar                          | SFP+ LR bailed, 10g single rate  | FTLX1471D3BCL                    |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Finisar                          | DUAL RATE 1G/10G SFP+ SR (No      | FTLX8571D3QCV-IT                  |
|                                  | Bail)                            |                                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Avago                            | DUAL RATE 1G/10G SFP+ SR (No      | AFBR-703SDZ-IN1                  |
|                                  | Bail)                            |                                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Finisar                          | DUAL RATE 1G/10G SFP+ LR (No      | FTLX1471D3QCV-IT                  |
|                                  | Bail)                            |                                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Avago                            | DUAL RATE 1G/10G SFP+ LR (No      | AFCT-701SDZ-IN1                  |
|                                  | Bail)                            |                                  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Finisar                          | 1000BASE-T SFP                    | FCLF8522P2BTL                    |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Avago                            | 1000BASE-T                        | ABCU-5710RZ                      |
+-----------------------------------+-----------------------------------+-----------------------------------+
| HP                                | 1000BASE-SX SFP                  | 453153-001                        |
+-----------------------------------+-----------------------------------+-----------------------------------+

82599-based adapters support all passive and active limiting direct
attach cables that comply with SFF-8431 v4.1 and SFF-8472 v10.4
specifications.


Turning the laser off or on for SFP+
""""""""""""""""""""""""""""""""""""

Use the following to turn off the laser for 82599-based SFP+ fiber
adapters:

  ifconfig ixX down

Use the following to turn on the laser:

  ifconfig ixX up


82599-Based QSFP+ Adapters
--------------------------

Note:

  * If your 82599-based Intel(R) Network Adapter came with Intel
    optics, it only supports Intel optics.

  * 82599-based QSFP+ adapters only support 4x10 Gbps connections.
    1x40 Gbps connections are not supported. QSFP+ link partners must
    be configured for 4x10 Gbps.

  * 82599-based QSFP+ adapters do not support automatic link speed
    detection. The link speed must be configured to either 10 Gbps or
    1 Gbps to match the link partners speed capabilities. Incorrect
    speed configurations will result in failure to link.

  * Intel(R) Ethernet Converged Network Adapter X520-Q1 only supports
    the optics and direct attach cables listed below.

+-----------------------------------+-----------------------------------+-----------------------------------+
| Supplier                          | Type                              | Part Numbers                      |
|===================================|===================================|===================================|
| Intel                            | DUAL RATE 1G/10G QSFP+ SRL        | E10GQSFPSR                        |
|                                  | (bailed)                          |                                  |
+-----------------------------------+-----------------------------------+-----------------------------------+

82599-based QSFP+ adapters support all passive and active limiting
QSFP+ direct attach cables that comply with SFF-8436 v4.1
specifications.


82598-Based Adapters
--------------------

Note:

  * Intel(R) Ethernet Network Adapters that support removable optical
    modules only support their original module type (for example, the
    Intel(R) 10 Gigabit SR Dual Port Express Module only supports SR
    optical modules). If you plug in a different type of module, the
    driver will not load.

  * Hot Swapping/hot plugging optical modules is not supported.

  * Only single speed, 10 gigabit modules are supported.

  * LAN on Motherboard (LOMs) may support DA, SR, or LR modules. Other
    module types are not supported. Please see your system
    documentation for details.

The following is a list of SFP+ modules and direct attach cables that
have received some testing. Not all modules are applicable to all
devices.

+-----------------------------------+-----------------------------------+-----------------------------------+
| Supplier                          | Type                              | Part Numbers                      |
|===================================|===================================|===================================|
| Finisar                          | SFP+ SR bailed, 10g single rate  | FTLX8571D3BCL                    |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Avago                            | SFP+ SR bailed, 10g single rate  | AFBR-700SDZ                      |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Finisar                          | SFP+ LR bailed, 10g single rate  | FTLX1471D3BCL                    |
+-----------------------------------+-----------------------------------+-----------------------------------+

82598-based adapters support all passive direct attach cables that
comply with SFF-8431 v4.1 and SFF-8472 v10.4 specifications. Active
direct attach cables are not supported.

Third-party optic modules and cables referred to above are listed only
for the purpose of highlighting third party specifications and
potential compatibility, and are not recommendations or endorsements
or sponsorship of any third party's product by Intel. Intel is not
endorsing or promoting products made by any third party and the third
party reference is provided only to share information regarding
certain optic modules and cables with the above specifications. There
may be other manufacturers or suppliers, producing or supplying optic
modules and cables with similar or matching descriptions. Customers
must use their own discretion and diligence to purchase optic modules
and cables from any third party of their choice. Customers are solely
responsible for assessing the suitability of the product and/or
devices and for the selection of the vendor for purchasing any
product. THE OPTIC MODULES AND CABLES REFERRED TO ABOVE ARE NOT
WARRANTED OR SUPPORTED BY INTEL. INTEL ASSUMES NO LIABILITY
WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY,
RELATING TO SALE AND/OR USE OF SUCH THIRD PARTY PRODUCTS OR SELECTION
OF VENDOR BY CUSTOMERS.


The VF Driver
=============

In the FreeBSD guest, the ixv driver would be loaded and will function
using the VF device assigned to it.

The VF driver provides most of the same functionality as the core
driver, but is actually a subordinate to the host. Access to many
controls is accomplished by a request to the host via what is called
the "Admin queue." These are startup and initialization events,
however; once in operation, the device is self-contained and should
achieve near native performance.

Some notable limitations of the VF environment:

* For security reasons, the driver is never permitted to be
  promiscuous, therefore a tcpdump will not behave the same with the
  interface.

* Media info is not available from the PF, so it will always appear as
  auto.


Building and Installation
=========================

Note:

  This driver package is to be used only as a standalone archive and
  the user should not attempt to incorporate it into the kernel source
  tree.

In the instructions below, "x.x.x" is the driver version as indicated
in the name of the driver tar file.

1. Move the base driver tar file to the directory of your choice. For
  example, use "/home/username/ix" or "/usr/local/src/ix".

2. Untar/unzip the archive:

      tar xzf ix-x.x.x.tar.gz

  This will create the "ix-x.x.x" directory.

3. To install the man page:

      cd ix-x.x.x
      gzip -c ix.4 > /usr/share/man/man4/ix.4.gz

4. To load the driver onto a running system:

      cd ix-x.x.x/src
      make
      kldload ./if_ix.ko

5. To assign an IP address to the interface, enter the following,
  where "X" is the interface number for the device:

      ifconfig ixX <IP_address>

6. Verify that the interface works. Enter the following, where
  "<IP_address>" is the IP address for another machine on the same
  subnet as the interface that is being tested:

      ping <IP_address>

7. If you want the driver to load automatically when the system is
  booted:

      cd ix-x.x.x/src
      make
      make install

  Then edit "/boot/loader.conf", and add the following line:

      if_ix_load="YES"

8. If you want the device to connect to a network without manual
  intervention after a boot, edit "/etc/rc.conf" and create the
  appropriate "ifconfig_<driver>X" entry:

      ifconfig_ixX="<ifconfig_settings>"

  Example usage:

      ifconfig_ix0="inet 192.168.10.1 netmask 255.255.255.0"

  Note:

    For assistance, see the ifconfig man page.


Configuration and Tuning
========================


Important System Configuration Changes
--------------------------------------

* Change the file "/etc/sysctl.conf", and add the line:

    hw.intr_storm_threshold: 0

  The default is 1000.

* Best throughput results are seen with a large MTU; use 9710 if
  possible. The default number of descriptors per ring is 1024.
  Increasing this may improve performance, depending on your use case.

* If you have a choice, run on a 64-bit OS rather than a 32-bit OS.


Speed and Duplex Configuration
------------------------------

In addressing speed and duplex configuration issues, you need to
distinguish between copper-based adapters and fiber-based adapters.

In the default mode, an Intel(R) Ethernet Network Adapter using copper
connections will attempt to auto-negotiate with its link partner to
determine the best setting. If the adapter cannot establish link with
the link partner using auto-negotiation, you may need to manually
configure the adapter and link partner to identical settings to
establish link and pass packets. This should only be needed when
attempting to link with an older switch that does not support auto-
negotiation or one that has been forced to a specific speed or duplex
mode. Your link partner must match the setting you choose. 1 Gbps
speeds and higher cannot be forced. Use the autonegotiation
advertising setting to manually set devices for 1 Gbps and higher.

Caution:

  Only experienced network administrators should force speed and
  duplex or change autonegotiation advertising manually. The settings
  at the switch must always match the adapter settings. Adapter
  performance may suffer or your adapter may not operate if you
  configure the adapter differently from your switch.

An Intel(R) Ethernet Network Adapter using fiber-based connections,
however, will not attempt to auto-negotiate with its link partner
since those adapters operate only in full duplex and only at their
native speed.

By default, the adapter auto-negotiates the speed and duplex of the
connection. If there is a specific need, the ifconfig utility can be
used to configure the speed and duplex settings on the adapter.

Note:

  For the Intel(R) Ethernet Connection X552 10 GbE SFP+, you must
  specify the desired speed.

Example usage:

  ifconfig ixX <IP_address> media 100baseTX mediaopt full-duplex

Note:

  Only use "mediaopt" to set the driver to full-duplex. If "mediaopt"
  is not specified and you are not running at gigabit speed, the
  driver defaults to half-duplex.

If the interface is currently forced to 100 full duplex, you must use
this command to change to half duplex:

  ifconfig ixX <IP_address> media 100baseTX -mediaopt full-duplex

This driver supports the following media type options:

autoselect:
  Enables auto-negotiation for speed and duplex.

10baseT/UTP:
  Sets speed to 10 Mbps. Use the "ifconfig mediaopt" option to select
  full-duplex mode.

100baseTX:
  Sets speed to 100 Mbps. Use the "ifconfig mediaopt" option to
  select full-duplex mode.

1000baseTX:
  Sets speed to 1000 Mbps. In this case, the driver supports only
  full-duplex mode.

1000baseSX:
  Sets speed to 1000 Mbps. In this case, the driver supports only
  full-duplex mode.

For more information on the ifconfig utility, refer to the ifconfig
man page.


Jumbo Frames
------------

Jumbo Frames support is enabled by changing the Maximum Transmission
Unit (MTU) to a value larger than the default value of 1500.

Use the ifconfig command to increase the MTU size. For example, enter
the following where "X" is the interface number:

  ifconfig ixX mtu 9000

To confirm an interface's MTU value, use the ifconfig command.

To confirm the MTU used between two specific devices, use:

  route get <destination_IP_address>

Note:

  * The maximum MTU setting for jumbo frames is 9710. This corresponds
    to the maximum jumbo frame size of 9728 bytes.

  * This driver will attempt to use multiple page sized buffers to
    receive each jumbo packet. This should help to avoid buffer
    starvation issues when allocating receive packets.

  * Packet loss may have a greater impact on throughput when you use
    jumbo frames. If you observe a drop in performance after enabling
    jumbo frames, enabling flow control may mitigate the issue.

  * For 82599-based network connections, if you are enabling jumbo
    frames in a virtual function (VF), jumbo frames must first be
    enabled in the physical function (PF). The VF MTU setting cannot
    be larger than the PF MTU.


VLANS
-----

To create a new VLAN interface:

  ifconfig <vlan_name> create

To associate the VLAN interface with a physical interface and assign a
VLAN ID, IP address, and netmask:

  ifconfig <vlan_name> <ip_address> netmask <subnet_mask> vlan <vlan_id>
  vlandev <physical_interface>

Example:

  ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ix0

In this example, all packets will be marked on egress with 802.1Q VLAN
tags, specifying a VLAN ID of 10.

To remove a VLAN interface:

  ifconfig <vlan_name> destroy


Checksum Offload
----------------

Checksum offloading supports both TCP and UDP packets and is supported
for both transmit and receive.

Checksum offloading can be enabled or disabled using ifconfig. Both
transmit and receive offloading will be either enabled or disabled
together. You cannot enable/disable one without the other.

To enable checksum offloading:

  ifconfig ixX rxcsum

To disable checksum offloading:

  ifconfig ixX -rxcsum

To confirm the current setting:

  ifconfig ixX

Look for the presence or absence of the following line:

  options=3 <RXCSUM,TXCSUM>

See the ifconfig man page for further information.


TSO
---

TSO (TCP Segmentation Offload) supports both IPv4 and IPv6. TSO can be
disabled and enabled using the ifconfig utility or sysctl.

Note:

  TSO requires Tx checksum, if Tx checksum is disabled, TSO will also
  be disabled.

To enable/disable TSO in the stack:

  sysctl net.inet.tcp.tso=0 (or 1 to enable it)

Doing this disables/enables TSO in the stack and affects all installed
adapters.

To disable BOTH TSO IPv4 and IPv6, where "X" is the number of the
interface in use:

  ifconfig ixX -tso

To enable BOTH TSO IPv4 and IPv6:

  ifconfig ixX tso

You can also enable/disable IPv4 TSO or IPv6 TSO individually. Simply
replace "tso" or "-tso" in the above command with "tso4" or "tso6".
For example, to disable TSO IPv4:

  ifconfig ixX -tso4

To disable TSO IPv6:

  ifconfig ixX -tso6


LRO
---

LRO (Large Receive Offload) may provide Rx performance improvement.
However, it is incompatible with packet-forwarding workloads. You
should carefully evaluate the environment and enable LRO when
possible.

To enable:

  ifconfig ixX lro

It can be disabled by using:

  ifconfig ixX -lro


Link-Level Flow Control (LFC)
-----------------------------

Ethernet Flow Control (IEEE 802.3x) can be configured with sysctl to
enable receiving and transmitting pause frames for ix. When transmit
is enabled, pause frames are generated when the receive packet buffer
crosses a predefined threshold. When receive is enabled, the transmit
unit will halt for the time delay specified when a pause frame is
received.

Note:

  You must have a flow control capable link partner.

Flow Control is enabled by default.

The ix driver also supports a "hw.ix.flow_control" load-time parameter
to set the initial flow control configuration on all supported
interfaces in a system. You can set the tunable using the kenv command
and then reload the driver, or by adding a line to "/boot/loader.conf"
and rebooting the system.

Use sysctl to change the flow control settings for a single interface
without reloading the driver.

The available values for flow control are:

* 0 = Disable flow control

* 1 = Enable Rx pause

* 2 = Enable Tx pause

* 3 = Enable Rx and Tx pause

Examples:

* To set the tunable in the config file, add the following line to
  "/boot/loader.conf" and then reboot the system:

    hw.ix.flow_control=3

* To set the tunable using kenv, use the following command and then
  reload the driver:

    kenv hw.ix.flow_control=3

* To remove the tunable and return the system to its default settings:

    kenv -u hw.ix.flow_control

* To enable a flow control setting with sysctl:

    sysctl dev.ix.<interface #>.fc=3

* To disable flow control using sysctl:

    sysctl dev.ix.<interface #>.fc=0

Note:

  * The ix driver requires flow control on both the port and link
    partner. If flow control is disabled on one of the sides, the port
    may appear to hang on heavy traffic.

  * For 82598 backplane cards entering 1 gigabit mode, flow control
    default behavior is changed to off. Flow control in 1 gigabit mode
    on these devices can lead to transmit hangs.

  * The VF driver does not have access to flow control. It must be
    managed from the host side.


DMAC
----

Valid Range: 0, 41-10000

This parameter enables or disables the DMA Coalescing (DMAC) feature.
Values are in microseconds and set the internal DMA Coalescing
internal timer. DMAC is available on Intel(R) X550 (and later) based
adapters.

DMA (Direct Memory Access) allows the network device to move packet
data directly to the system's memory, reducing CPU utilization.
However, the frequency and random intervals at which packets arrive do
not allow the system to enter a lower power state. DMA Coalescing
allows the adapter to collect packets before it initiates a DMA event.
This may increase network latency but also increases the chances that
the system will enter a lower power state. Turning on DMA Coalescing
may save energy. DMA Coalescing must be enabled across all active
ports in order to save platform power.

Interrupt Throttle Rate (ITR) should be set to dynamic. When "ITR=0",
DMA Coalescing is automatically disabled.


Firmware Logs
-------------

The ix driver allows you to generate firmware logs on Intel(R) E610
Ethernet Controller devices for supported categories of events, to
help debug issues with Customer Support. Firmware logs are enabled by
default. Refer to the Intel(R) Ethernet Adapters and Devices User
Guide for an overview of this feature and additional tips.

At a high level, you must do the following to capture a firmware log:

1. Set the configuration for the firmware log.

2. Perform the necessary steps to generate the issue you are trying to
  debug.

3. Capture the firmware log.

4. Stop capturing the firmware log.

5. Reset your firmware log settings as needed.

6. Work with Customer Support to debug your issue.

Note:

  Firmware logs are generated in a binary format and must be decoded
  by Customer Support. Information collected is related only to
  firmware and hardware for debug purposes.

Once the driver is loaded, it will create the fw_log sysctl node under
the debug section of the driver's sysctl list. The driver groups these
events into categories, called "modules." Supported modules include:

general:
  General (Bit 0)

ctrl:
  Control (Bit 1)

link:
  Link Management (Bit 2)

link_topo:
  Link Topology Detection (Bit 3)

dnl:
  Link Control Technology (Bit 4)

i2c:
  I2C (Bit 5)

sdp:
  SDP (Bit 6)

mdio:
  MDIO (Bit 7)

adminq:
  Admin Queue (Bit 8)

hdma:
  Host DMA (Bit 9)

lldp:
  LLDP (Bit 10)

dcbx:
  DCBx (Bit 11)

dcb:
  DCB (Bit 12)

xlr:
  XLR (function-level resets; Bit 13)

nvm:
  NVM (Bit 14)

auth:
  Authentication (Bit 15)

vpd:
  Vital Product Data (Bit 16)

iosf:
  Intel On-Chip System Fabric (Bit 17)

parser:
  Parser (Bit 18)

sw:
  Switch (Bit 19)

scheduler:
  Scheduler (Bit 20)

txq:
  TX Queue Management (Bit 21)

acl:
  ACL (Access Control List; Bit 22)

post:
  Post (Bit 23)

watchdog:
  Watchdog (Bit 24)

task_dispatch:
  Task Dispatcher (Bit 25)

mng:
  Manageability (Bit 26)

synce:
  SyncE (Bit 27)

health:
  Health (Bit 28)

tsdrv:
  Time Sync (Bit 29)

pfreg:
  PF Registration (Bit 30)

mdlver:
  Module Version (Bit 31)

You can change the verbosity level of the firmware logs. You can set
only one log level per module, and each level includes the verbosity
levels lower than it. For instance, setting the level to "normal" will
also log warning and error messages. Available verbosity levels are:

* 0 = none

* 1 = error

* 2 = warning

* 3 = normal

* 4 = verbose

To set the desired verbosity level for a module, use the following
sysctl command and then register it:

  sysctl dev.ix.<interface #>.debug.fw_log.severity.<module>=<level>

For example:

  sysctl dev.ix.0.debug.fw_log.severity.link=1
  sysctl dev.ix.0.debug.fw_log.severity.link_topo=2
  sysctl dev.ix.0.debug.fw_log.register=1

To log firmware messages before the driver initializes, use the kenv
command to set the tunable. The "on_load" setting tells the device to
register the variable as soon as possible during driver load. For
example:

  kenv dev.ix.0.debug.fw_log.severity.link=1
  kenv dev.ix.0.debug.fw_log.severity.link_topo=2
  kenv dev.ix.0.debug.fw_log.on_load=1

To view the firmware logs and redirect them to a file, use the
following command:

  dmesg > log_output

Note:

  Logging a large number of modules or too high of a verbosity level
  will add extraneous messages to dmesg and could hinder debug
  efforts.


Debug Dump
----------

Intel(R) E610 Ethernet Controller devices support debug dump, which
allows you to obtain runtime register values from the firmware for
"clusters" of events and then write the results to a single dump file,
for debugging complicated issues in the field.

This debug dump contains a snapshot of the device and its existing
hardware configuration, such as switch tables, transmit scheduler
tables, and other information. Debug dump captures the current state
of the specified cluster(s) and is a stateless snapshot of the whole
device.

Note:

  * Like with firmware logs, the contents of the debug dump are not
    human-readable. You must work with Customer Support to decode the
    file.

  * Debug dump is per device, not per PF.

  * Debug dump writes all information to a single file.

To generate a debug dump file in FreeBSD, do the following:

1. Specify the cluster(s) to include in the dump file, using a bitmask
  and the following command:

      sysctl dev.ix.<interface #>.debug.dump.clusters=<bitmask>

  * To print the complete cluster bitmask and parameter list to the
    screen, pass the "-d" argument. For example:

        sysctl -d dev.ix.0.debug.dump.clusters

  * Possible bitmask values for "clusters" are:

        * 0x1 - Link

        * 0x2 - Full CSR Space, excluding RCW registers

    For example, to dump the Link cluster, use the following:

        sysctl dev.ix.0.debug.dump.clusters=0x1

    To dump full CSR Space, excluding RCW registers, use the
    following:

        sysctl dev.ix.0.debug.dump.clusters=0x2

    To dump all clusters, use the following:

        sysctl dev.ice.0.debug.dump.clusters=0

    Note:

      Using "0" will skip Manageability Transactions data.

  * If you don't specify a cluster, the driver will dump all clusters
    to a single file.

2. Issue the debug dump command, using the following:

      sysctl -b dev.ix.<interface #>.debug.dump.dump=1 > dump.bin

Note:

  * The driver will not receive the command if you do not write "1" to
    the sysctl.

  * Replace "dump.bin" above with the file name you want to use.

  * If you did not specify any clusters, the dump file will include
    information about all clusters.

To clear the clusters bitmap before a subsequent debug dump and then
do the dump:

  sysctl dev.ix.0.debug.dump.clusters=0
  sysctl dev.ix.0.debug.dump.dump=1


Known Issues/Troubleshooting
============================


UDP Stress Test Dropped Packet Issue
------------------------------------

Under small packet UDP stress with the ix driver, the system may drop
UDP packets due to socket buffers being full. Setting the driver Intel
Ethernet Flow Control variables to the minimum may resolve the issue.


Error message "ixX: could not setup receive structures"
-------------------------------------------------------

Attempting to configure larger MTUs with a large numbers of processors
may generate the error message "ixX: could not setup receive
structures."

When using the ix driver with RSS autoconfigured based on the number
of cores (the default setting) and that number is larger than 4,
increase the memory resources allocated for the mbuf pool as follows:

  Add the following to the "sysctl.conf" file for the system:

      kern.ipc.nmbclusters=262144
      kern.ipc.nmbjumbop=262144


Lower than expected performance
-------------------------------

Some PCIe x8 slots are actually configured as x4 slots. These slots
have insufficient bandwidth for full line rate with dual port and quad
port devices. In addition, if you put a PCIe v4.0 or v3.0-capable
adapter into a PCIe v2.x slot, you cannot get full bandwidth. The
driver detects this situation and writes one of the following messages
in the system log:

  PCI-Express bandwidth available for this card is not sufficient for optimal
  performance. For optimal performance a x8 PCI-Express slot is required.

or:

  PCI-Express bandwidth available for this device may be insufficient for
  optimal performance. Please move the device to a different PCI-e link with
  more lanes and/or higher transfer rate.

If this error occurs, moving your adapter to a true PCIe v3.0 x8 slot
will resolve the issue.


"no PRT entry for x.x.INTx" error on pcib4 in dmesg
---------------------------------------------------

On systems that use legacy interrupts, you may see the following in
dmesg:

  pci4: on pcib4
  pcib4: no PRT entry for <x>.0.INTA
  pcib4: no PRT entry for <x>.0.INTB

For example:

  pci4: on pcib4
  pcib4: no PRT entry for 3.0.INTA
  pcib4: no PRT entry for 3.0.INTB

To work around the issue, add the following lines to
"/boot/loader.conf", where "<x>" is the number found in dmesg message
(either 3 or 4):

  hw.pci0.<x>.0.INTA.irq=16
  hw.pci0.<x>.0.INTB.irq=17


Support
=======

For general information, go to the Intel support website at
http://www.intel.com/support/.

If an issue is identified with the released source code on a supported
kernel with a supported adapter, email the specific information
related to the issue to freebsd@intel.com.

Copyright(c) 1999 - 2024 Intel Corporation.


Trademarks
==========

Intel is a trademark or registered trademark of Intel Corporation or
its subsidiaries in the United States and/or other countries.

Other names and brands may be claimed as the property of others.


[sensuary]

I have been struggling with this for some days now.
I have a e610 xt4 cards and have been able to compile the official driver from Intel, move the .ko file into /boot/modules (like is explained at thread https://forum.opnsense.org/index.php?topic=21517.msg101148#msg101148) and gotten the driver to sort of work.
I do see the ports of the card in ifconfig but when I attach a cable to it then ifconfig always shows status as "no carrier".
I named my driver something like if_ix_updated.ko and that is the driver being shown in the output of kldstat. Also at close to the top of dmesg I see a message about not being able to load if_ix.ko since if_ix_updated.ko is already loaded, so seems that my driver takes precedence over the inbuilt driver.
But I am wondering if the Intel driver is causing me problems because the kernel expects the builtin if_ix driver but gets an external one from Intel instead? Does that mean I *have* to recompile the kernel and exclude the if_ix driver from it?

[BrandyWine]

@aensuary
It won't work the way you are doing it.
You 1st need to completely remove pci/ix from kernel before loading in a new ix. The name of the ko file is just user space name stuff, like what kld tools will pipe to stdout. What's inside the ix code is what matters.

As for that em driver link you included, something seems off with that thread. From my research, freeBSD has no mechanism (not via loader or other) to remove a static driver from kernel to allow you to load in same as klm ko.

As example, "Modified /boot/loader.conf.local: #if_em_load="YES" & if_igb_load="YES" "
This would actually do nothing, loader would attempt to load in the ko's but the process would fail because both pci/em and pci/igb are already in the kernel module as static. Running kldstat shows klm ko's only, kldstat -v -n kernel will show you the static modules in kernel.

However, it is possible to compile the same driver to have different ID stuff for the kernel to load in, as example it could be pci/em2 ko named "my_if_em.ko", but doing this will cause conflict in kernel because the kernel will try and bind device ID's to multiple drivers.

Onto the next tidbit. ix is a suite built from ixgbe. 610 apparently shares some base stuff from 550, so it makes sense the 610 code reuses some existing stuff from 550. It is 100% possible though to edit code so that when build comes you end up with a new ko for just the one nic model, and that's what I tries to do, successfully building a ko just for 610 (hence it would not conflict with existing static ix because that ix does not contain the 610 device ID's used by kernel to bind), so a unique driver. I could not kldload the 610 driver I built from freeBSD 15 src compiled against 14.3, kernel balking on the load saying the driver was already installed, even after I did a bunch of name changes in the code, I perhaps missed entrypoint code stuff because from within ixgbe the entrypoint name stuff will be the same for ix.

The root of this problem come from kernel/mod build using GENERIC cfg. It will build lots of stuff as static in kernel module. Good for "it just works when I boot my machine", but bad for when you need to add in something new, like support for 610nic but they (intel and the distros) are reusing the suite (ixgbe) to add in code for 610, which means you needs a whole new ix driver, which for freeBSD GENERIC is not easy to do because of all the static stuff that's in kernel module. With GENERIC you have to recompile the tree (with the new ixbge stuff), which btw might not always work because new driver code could be dependent upon new kernel code, hence why you'll see sometimes "must have kernel 2.3.x.xx or newer for this to work", and building a new tree is basically an OS upgrade (14.3 to 15 as example).

The goal here was to get support for 610 in v14.3 OS. It's possible to pull out just the e610 code from ixgbe and build a driver just for that, as I did. The other way is to build the full ixgbe (ix) from v15 src tree, or the ixbge from Intel supply. Building ix from 15 src tree seems acceptable because Intel docs do not indicate kernel version dependency. But ok, now is have a new if_ix.ko, what do I do with it? Cannot just kldload that into OPNsense os 14.3. The only viable solution (problem free way) is to recompile kernel without pci/ix, this way you can now kldload the ix ko.

A recompile (using your own kernel config, read the build docs) also allows you to remove the bloat, things that are not needed by your platform, and-or undo static compiling allowing you to use klm ko's.

The recompile method does not constitute back port, because the latest Intel driver code does not have dependency on freeBSD 15. It may however need a true back port for other OS's.

[BrandyWine]

I have a e610 driver ko, but I cannot load it, kernel thinks it's the same driver as one already in the kernel.[...]

I thought FreeBSD was fully modular, that is I'd expect you could kldunload the existing driver and kldload yours. Of course, I haven't tried it.

You are correct, but being modular does not mean everything is dynamic, it only means you have ability to load/unload klm's. Monolithic completely removes that ability. In our modern day stuff, to make it "just work", lots of things (mostly the common stuff) is static in kernel. Obviously we now see why that's a pita for some OS builds, like freeBSD GENERIC. This can be changed though, don't build using the provided GENERIC config, as example.