Just purchased the E610 card and was w9ndering if anyone knew where i can look to see if its supported in FreeBSD 14?
E610 support was added to FreeBSD in August and has not yet made it into an official release. Assuming it will be backported for 14.4 you can expect it in OPNsense 26.1 or 26.7.
At the moment the answer is "no".
Reference: https://cgit.freebsd.org/src/commit/?id=dea5f973d0c8d29a79b433283d0a2de8f4615957
wah I was about to post "Ragu", but I was in a v15 src folder locally.
E610 are pricey too.
Hmmm, I wonder why the "introduce new device ID's" don't show these other two two that are in the driver code? They must be new because they are not in 14.3 -release.
#define IXGBE_DEV_ID_E610_VF 0x57AD
#define IXGBE_SUBDEV_ID_E610_VF_HV 0x0001
Introduce new PCI device IDs:
• 57AE: Intel(R) E610 (Backplane)
• 57AF: Intel(R) E610 (SFP)
• 57B0: Intel(R) E610 (10 GbE)
• 57B1: Intel(R) E610 (2.5 GbE)
• 57B2: Intel(R) E610 (SGMII)
If its not explicitly supported does that mean I have to compile a driver for it or am I SOL until next year?
I really was hoping this could replace my X550 bc the X550 is really toasty and I cannot put the cover to my mini pc on.
Quote from: spetrillo on September 25, 2025, 11:33:30 PMIf its not explicitly supported does that mean I have to compile a driver for it or am I SOL until next year?
I really was hoping this could replace my X550 bc the X550 is really toasty and I cannot put the cover to my mini pc on.
Well, if the src code in 15 -stable is not complete yet (works but can be buggy, that's what "stable" means), then even compiling that down to a ko will be questionable. You might try to compile it against the 14.3 src tree and see if anything complains. If there are complaints then you'll need to map them out to see how yo can bring some of the v15 code over to 14.3.
That last commit in v15 for ixbge was about 35 days ago, so fairly new with a discrepancy as I noted.
It's not easy to follow where a v15 dependency is because these src code files all have lots of includes, which come from various parts of the src tree, and those files too could have been modified.
It's worth a try just to see if you can take v15 ixbge src and compile it against v14.3 tree. If nothing complains then you made it past the 50% line, the other 50% is seeing if the -stable code (now a ko driver) is complete enough to run your E610 without issue. I'll venture a guess and say your XT2 is device ID 0x57B0
Edit:
As example, lets say you compile on ubuntu, download the 14.3 release src tree, swap out the v14 ixgbe folder for the v15 one, get into that ixbge and compile there. You probably have to adjust some compile options so that gcc knows you are compiling against that specific tree.
I grab files for my ubuntu and see if I can compile the v15 stuff, see what err's it spits out.
Quote from: Patrick M. Hausen on September 25, 2025, 07:15:10 PM[...]At the moment the answer is "no".[...]
Argh! I didn't look at the version. Dang, they're late - they should have it committed prior to launch. And that's a hell of a patch for a jumped-up X550, which itself is apparently a jumped-up 82599.
Quote from: BrandyWine on September 26, 2025, 12:43:12 AMAs example, lets say you compile on ubuntu, [...]
I am not quite sure that is possible. Have you built FreeBSD on Linux?
I would go about it like this:
- install FreeBSD 14.3 in a VM or on some spare machine
- checkout the releng/14.3 source tree
- cherry pick the changes for the ixgbe directory - link in my first post above
- make buildworld && make buildkernel
- copy if_ix.ko to the OPNsense system, /boot/kernel directory, keep fingers crossed, reboot
Quote from: Patrick M. Hausen on September 26, 2025, 10:17:48 AMI am not quite sure that is possible
100% possible (done all the time). You have to setup a x-compile environment which targets freebsd14, which is easy to do. Compiling a new gcc and having it install to a x-compile location is the longest part of the setup, and many times throws err for missing or not-found items, so some tweaking to get done. I just did this on my WSL Ubuntu. So it's basically setting up for freebsd14 target and then gcc against the 14.3 kernel.
It's absoultely much easier to compile on a like-for-like setup, I didn't want to install 14.3 for this task.
I not picking out anything, 15 already has lots of include changes, i will just compile ixbge against the 14.3 kernel
Quote from: Patrick M. Hausen on September 26, 2025, 10:17:48 AMcopy if_ix.ko to the OPNsense system, /boot/kernel directory, keep fingers crossed, reboot
This part is question.
Where currently does the opnsense system load in the ix.ko?
Currently the ix is in kernel, not loaded in as a KLM, kldstat does not show ix.ko as klm, I see it in kernel kldstat -v , not as klm. So how to unbind ix that's in kernel? After that we can kldload in ix.ko. Some OS's allows doing KLM over a resident kernel module, but from what I know of that, it's bad practice to allow it.
Quote from: BrandyWine on September 26, 2025, 04:02:12 PMSo it's basically setting up for freebsd14 target and then gcc against the 14.3 kernel.
FreeBSD uses Clang for kernel compilation. I'd rather create a VM and run the build process natively.
Quote from: BrandyWine on September 26, 2025, 04:02:12 PMCurrently the ix is in kernel, not loaded in as a KLM
Then you need to copy over the kernel instead of the module. Or best build OPNsense from source after cherry picking.
So the real answer for me is I am SOL until next year. That's ok.
The E610 is a very low wattage adapter, so when its supported it will be ready to replace my X550.
Quote from: Patrick M. Hausen on September 26, 2025, 10:17:48 AM[...]cherry pick the changes for the ixgbe directory[...]
~16 files, but that's just from that commit. Franky, I don't know how to use these RCS interfaces. I'd grab the whole directory from the latest tree, plus add the two lines to /sys/conf/files (I wouldn't just grab it without diffing it, as who knows what else has been altered).
Quote from: spetrillo on September 26, 2025, 04:29:27 PMSo the real answer for me is I am SOL until next year. That's ok.
The E610 is a very low wattage adapter, so when its supported it will be ready to replace my X550.
I get that. It looks like the only solution (for low-power multi-port 10GBASE-T). Everyone else besides Realtek is chasing the high end, and I haven't seen the 8127 yet (and it's not in the re driver).
Maybe @franco can be convinced to cherry pick that commit? @spetrillo try a feature request on github. The link to the commit can be found in my first post.
Clang and gcc are viable compilers.
I git clone 15stable and 14.3release source trees.
From there I setup my x-compile env.
Now, I can
1) compile the ixgbe inside 15 tree using 14.3 src tree as the reference for kernel, so essentially all the 15 code compiled for 14.3 kernel.
2) The 2nd thing that can be done is, copy ixgbe from 15 over to 14.3 sys/dev/ as ixgbe15, then compile the same way.
The 1st will pick up all the includes from 15 tree, the 2nd will pick up all the includes from 14.3 tree. Knowing that some other changes are made in 15 tree I suspect compiling in 15 tree is the best hope for success. The big unknown is, how complete is 15 stable, specifically regarding E610 stuff.
Quote from: Patrick M. Hausen on September 26, 2025, 04:27:00 PMThen you need to copy over the kernel instead of the module
This is the confusing part, I am not sure why the OPNsense install has all those /boot kernel ko's there if most of them are compiled into the kernel module. I don't think building new kernel is good because current build options are unknown, so rather not mess with that.
I was wondering if a boot process was picking up those ko's and loading them in, but from all the docs I have read there's no hiding a klm from kldstat.
So, more fun.
Quote from: spetrillo on September 26, 2025, 04:29:27 PMSo the real answer for me is I am SOL until next year. That's ok.
The E610 is a very low wattage adapter, so when its supported it will be ready to replace my X550.
Your patience needs to match "community" of OPNsense product. ;)
Unless there are others who are willing to test a new ko, you are the testor.
Trusting compiled stuff from anonymous is not the best, and not acceptable for critical stuff.
Once the process is somewhat well-documented you can then do it yourself, this way if there are issues, it's on you.
Or if you prefer, shelve the 610 for now and wait.
Or, be community oriented, try test fix try test fix, report your findings, etc.
Quote from: BrandyWine on September 26, 2025, 06:57:26 PMQuote from: spetrillo on September 26, 2025, 04:29:27 PMSo the real answer for me is I am SOL until next year. That's ok.
The E610 is a very low wattage adapter, so when its supported it will be ready to replace my X550.
Your patience needs to match "community" of OPNsense product. ;)
Unless there are others who are will to test a new ko, you are the testor.
Trusting compiled stuff from anonymous is not the best, and not acceptable for critical stuff.
Once the process is somewhat well-documented you can then do it yourself, this was if there are issues, it's on you.
Or if you prefer, shelve the 610 for now and wait.
Or, be community oriented, try test fix try test fix, report your findings, etc.
I am patient when I need to patient.
I am happy to be the testor. I have a spare machine that I can put the E610 into and setup a dev environment. I have never compiled anything on this platform, so if I do go down this path I will need some help in setting up the dev environment properly.
For now I am going to open a Git request and see if there is any appetite to port E610 support into the 14.3 software.
Quote from: spetrillo on September 26, 2025, 07:10:55 PMFor now I am going to open a Git request and see if there is any appetite to port E610 support into the 14.3 software.
Well, if you start a dev env, probably easiest to install a v15 VM, then git clone the 14.3 src tree.
From there things are much more native for compiling, you just need to compile against the 14.3 kernel source.
As for port, well, the E610 code is 1) in 15-stable, and 2) not even sure if the E610 code was completed yet. So perhaps porting from -stable is on-par with just compiling what's there now.
I have a e610 driver ko, but I cannot load it, kernel thinks it's the same driver as one already in the kernel. Mine compiles out as "ix2".
It's not easy trying to just compile out a ko just for the 610, because they wrap sets of device ID's into the ixbge code, and if you don't remove all the other device ID's the kernel will read the driver and try and bind ("attach") the driver code to the device ID's it finds on pci. We only want the 610 parts so we can load it in as it's own ko just for the 610. I managed to rename the driver as ibgxe_if which will attach to pci bus, but there's some other ID that needs to change, probably in an entrypoint section.
The e610 code is weird, looks like they share code from x550 because the e610 code pulls in a x550 h file.
I see some e610 code floating around on git, but need to hunt down actual Intel src code, might make it easier to build a sole 610 ko that can load up without issue, the 14.3 ixbe does not include 610 code, so a sperate 610 ko would be ideal as a temp fix for running that card on 14.3.
The latest e610 driver is in the Intel 30.4.2 download bundle (in igbe ix), Intel's version of it is 3.4.31.
Make it easy to compile down an ix.ko, but that's still an issue because as I mentioned, ix has other devices in it, and, it's already in the 14.3 kernel being used by opnsense.
I think to avoid this in the future it might be better to compile kernel w/o all the driver code, and then just include all the common drivers as klm's via loader.conf, this way we could load out/in whatever we wanted to with ease.
I am testing something now on my opnsense. edit: which shows me the if_ix.ko is just a file not being used.
See, OPNsense uses the GENERIC setup for kernel build, and GENERIC leans monolithic.
Quote from: BrandyWine on September 30, 2025, 03:48:15 AMI have a e610 driver ko, but I cannot load it, kernel thinks it's the same driver as one already in the kernel.[...]
I thought FreeBSD was fully modular, that is I'd expect you could kldunload the existing driver and kldload yours. Of course, I haven't tried it.
It isn't. Most drivers are compiled statically and cannot be un/loaded.
Quote from: Patrick M. Hausen on September 30, 2025, 10:26:38 AMIt isn't. Most drivers are compiled statically and cannot be un/loaded.
In Linux I can write to a unbind of a device and the kernel will process that and unbind the driver.
In freeBSD can't do that.
I did some more reading last night to see another path, and on all the nix forums it's suggested to recompile kernel without ix, so that's what I have done. I will now compile down the Intel 610 driver code and load that in as klm.
Success on kernel build while excluding the pci/ix completely.
Next is to compile the Intel 610 src. Trying now.
Success on kldload and kldunload of the ix ko from 14.3 (opnsense 25.7).
Working on Intel compile.
Edit. Intel driver compiled a-ok, i can kldload and kldunload that driver.
So, who wants to test kernel and this Intel ix? Nobody should trust my files, but they are test worthy to see if the kernel will bind driver to 610 nic w/o err. Intel ix man4 attached.
I made a new dev in tree (so not to muck around with ix)
/usr/src/14.3-RELEASE/sys/dev/ix610/
untar the Intel src to ix610 (or whatever you name it)
modify Makefile to understand the src tree
comment added, next two lines added
#.PATH: ${.CURDIR}
.PATH: ${SRCTOP}/sys/dev/ix610
.PATH: ${SRCTOP}/sys/${MACHINE_CPUARCH}/${MACHINE_CPUARCH}
Setup env
SRCTOP=/usr/src/14.3-RELEASE
export SRCTOP
Move existing ix, build
mv /boot/kernel/if_ix.ko /boot/kernel/if_ix.ko.orig
cd $SRCTOP/sys/dev/ix610
make
make install
The new ix is now in /boot/kernel, but you cannot load it until you swap out kernel and reboot system
Intel readme
ix FreeBSD* Driver for Intel(R) Ethernet 10 Gigabit PCI Express Server Adapters
*******************************************************************************
July 17, 2025
Contents
^^^^^^^^
* ix FreeBSD* Driver for Intel(R) Ethernet 10 Gigabit PCI Express
Server Adapters
* Overview
* Related Documentation
* Identifying Your Adapter
* The VF Driver
* Building and Installation
* Configuration and Tuning
* Known Issues/Troubleshooting
* Support
* Trademarks
Overview
========
This file describes the FreeBSD* driver for Intel(R) Ethernet. This
driver has been developed for use with all community-supported
versions of FreeBSD.
For questions related to hardware requirements, refer to the
documentation supplied with your Intel Ethernet Adapter. All hardware
requirements listed apply to use with FreeBSD.
Note:
Devices based on the Intel(R) Ethernet Controller X552 and Intel(R)
Ethernet Controller X553 do not support the following features:
* Low Latency Interrupts (LLI)
The associated Virtual Function (VF) driver for this driver is ixv.
Related Documentation
=====================
See the "Intel(R) Ethernet Adapters and Devices User Guide" for
additional information on features. It is available on the Intel
website at either of the following:
* https://cdrdv2.intel.com/v1/dl/getContent/705831
* https://www.intel.com/content/www/us/en/download/19373/adapter-user-
guide-for-intel-ethernet-adapters.html
Identifying Your Adapter
========================
The driver is compatible with devices based on the following:
* Intel(R) Ethernet Controller 82598
* Intel(R) Ethernet Controller 82599
* Intel(R) Ethernet Controller X520
* Intel(R) Ethernet Controller X540
* Intel(R) Ethernet Controller x550
* Intel(R) Ethernet Controller X552
* Intel(R) Ethernet Controller X553
* Intel(R) Ethernet Controller E610
For information on how to identify your adapter, and for the latest
Intel network drivers, refer to the Intel Support website at
http://www.intel.com/support.
SFP+ Devices with Pluggable Optics
----------------------------------
82599-Based Adapters
~~~~~~~~~~~~~~~~~~~~
Note:
* If your 82599-based Intel(R) Network Adapter came with Intel
optics or is an Intel(R) Ethernet Server Adapter X520-2, then it
only supports Intel optics and/or the direct attach cables listed
below.
* When 82599-based SFP+ devices are connected back to back, they
should be set to the same Speed setting via ethtool. Results may
vary if you mix speed settings.
+-----------------------------------+-----------------------------------+-----------------------------------+
| Supplier | Type | Part Numbers |
|===================================|===================================|===================================|
| **SR Modules:** | | |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel | DUAL RATE 1G/10G SFP+ SR (bailed) | FTLX8571D3BCV-IT |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel | DUAL RATE 1G/10G SFP+ SR (bailed) | AFBR-703SDZ-IN2 |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel | DUAL RATE 1G/10G SFP+ SR (bailed) | AFBR-703SDDZ-IN1 |
+-----------------------------------+-----------------------------------+-----------------------------------+
| **LR Modules:** | | |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel | DUAL RATE 1G/10G SFP+ LR (bailed) | FTLX1471D3BCV-IT |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel | DUAL RATE 1G/10G SFP+ LR (bailed) | AFCT-701SDZ-IN2 |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Intel | DUAL RATE 1G/10G SFP+ LR (bailed) | AFCT-701SDDZ-IN1 |
+-----------------------------------+-----------------------------------+-----------------------------------+
The following is a list of third-party SFP+ modules that have received
some testing. Not all modules are applicable to all devices.
+-----------------------------------+-----------------------------------+-----------------------------------+
| Supplier | Type | Part Numbers |
|===================================|===================================|===================================|
| Finisar | SFP+ SR bailed, 10g single rate | FTLX8571D3BCL |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Avago | SFP+ SR bailed, 10g single rate | AFBR-700SDZ |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Finisar | SFP+ LR bailed, 10g single rate | FTLX1471D3BCL |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Finisar | DUAL RATE 1G/10G SFP+ SR (No | FTLX8571D3QCV-IT |
| | Bail) | |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Avago | DUAL RATE 1G/10G SFP+ SR (No | AFBR-703SDZ-IN1 |
| | Bail) | |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Finisar | DUAL RATE 1G/10G SFP+ LR (No | FTLX1471D3QCV-IT |
| | Bail) | |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Avago | DUAL RATE 1G/10G SFP+ LR (No | AFCT-701SDZ-IN1 |
| | Bail) | |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Finisar | 1000BASE-T SFP | FCLF8522P2BTL |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Avago | 1000BASE-T | ABCU-5710RZ |
+-----------------------------------+-----------------------------------+-----------------------------------+
| HP | 1000BASE-SX SFP | 453153-001 |
+-----------------------------------+-----------------------------------+-----------------------------------+
82599-based adapters support all passive and active limiting direct
attach cables that comply with SFF-8431 v4.1 and SFF-8472 v10.4
specifications.
Turning the laser off or on for SFP+
""""""""""""""""""""""""""""""""""""
Use the following to turn off the laser for 82599-based SFP+ fiber
adapters:
ifconfig ixX down
Use the following to turn on the laser:
ifconfig ixX up
82599-Based QSFP+ Adapters
--------------------------
Note:
* If your 82599-based Intel(R) Network Adapter came with Intel
optics, it only supports Intel optics.
* 82599-based QSFP+ adapters only support 4x10 Gbps connections.
1x40 Gbps connections are not supported. QSFP+ link partners must
be configured for 4x10 Gbps.
* 82599-based QSFP+ adapters do not support automatic link speed
detection. The link speed must be configured to either 10 Gbps or
1 Gbps to match the link partners speed capabilities. Incorrect
speed configurations will result in failure to link.
* Intel(R) Ethernet Converged Network Adapter X520-Q1 only supports
the optics and direct attach cables listed below.
+-----------------------------------+-----------------------------------+-----------------------------------+
| Supplier | Type | Part Numbers |
|===================================|===================================|===================================|
| Intel | DUAL RATE 1G/10G QSFP+ SRL | E10GQSFPSR |
| | (bailed) | |
+-----------------------------------+-----------------------------------+-----------------------------------+
82599-based QSFP+ adapters support all passive and active limiting
QSFP+ direct attach cables that comply with SFF-8436 v4.1
specifications.
82598-Based Adapters
--------------------
Note:
* Intel(R) Ethernet Network Adapters that support removable optical
modules only support their original module type (for example, the
Intel(R) 10 Gigabit SR Dual Port Express Module only supports SR
optical modules). If you plug in a different type of module, the
driver will not load.
* Hot Swapping/hot plugging optical modules is not supported.
* Only single speed, 10 gigabit modules are supported.
* LAN on Motherboard (LOMs) may support DA, SR, or LR modules. Other
module types are not supported. Please see your system
documentation for details.
The following is a list of SFP+ modules and direct attach cables that
have received some testing. Not all modules are applicable to all
devices.
+-----------------------------------+-----------------------------------+-----------------------------------+
| Supplier | Type | Part Numbers |
|===================================|===================================|===================================|
| Finisar | SFP+ SR bailed, 10g single rate | FTLX8571D3BCL |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Avago | SFP+ SR bailed, 10g single rate | AFBR-700SDZ |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Finisar | SFP+ LR bailed, 10g single rate | FTLX1471D3BCL |
+-----------------------------------+-----------------------------------+-----------------------------------+
82598-based adapters support all passive direct attach cables that
comply with SFF-8431 v4.1 and SFF-8472 v10.4 specifications. Active
direct attach cables are not supported.
Third-party optic modules and cables referred to above are listed only
for the purpose of highlighting third party specifications and
potential compatibility, and are not recommendations or endorsements
or sponsorship of any third party's product by Intel. Intel is not
endorsing or promoting products made by any third party and the third
party reference is provided only to share information regarding
certain optic modules and cables with the above specifications. There
may be other manufacturers or suppliers, producing or supplying optic
modules and cables with similar or matching descriptions. Customers
must use their own discretion and diligence to purchase optic modules
and cables from any third party of their choice. Customers are solely
responsible for assessing the suitability of the product and/or
devices and for the selection of the vendor for purchasing any
product. THE OPTIC MODULES AND CABLES REFERRED TO ABOVE ARE NOT
WARRANTED OR SUPPORTED BY INTEL. INTEL ASSUMES NO LIABILITY
WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY,
RELATING TO SALE AND/OR USE OF SUCH THIRD PARTY PRODUCTS OR SELECTION
OF VENDOR BY CUSTOMERS.
The VF Driver
=============
In the FreeBSD guest, the ixv driver would be loaded and will function
using the VF device assigned to it.
The VF driver provides most of the same functionality as the core
driver, but is actually a subordinate to the host. Access to many
controls is accomplished by a request to the host via what is called
the "Admin queue." These are startup and initialization events,
however; once in operation, the device is self-contained and should
achieve near native performance.
Some notable limitations of the VF environment:
* For security reasons, the driver is never permitted to be
promiscuous, therefore a tcpdump will not behave the same with the
interface.
* Media info is not available from the PF, so it will always appear as
auto.
Building and Installation
=========================
Note:
This driver package is to be used only as a standalone archive and
the user should not attempt to incorporate it into the kernel source
tree.
In the instructions below, "x.x.x" is the driver version as indicated
in the name of the driver tar file.
1. Move the base driver tar file to the directory of your choice. For
example, use "/home/username/ix" or "/usr/local/src/ix".
2. Untar/unzip the archive:
tar xzf ix-x.x.x.tar.gz
This will create the "ix-x.x.x" directory.
3. To install the man page:
cd ix-x.x.x
gzip -c ix.4 > /usr/share/man/man4/ix.4.gz
4. To load the driver onto a running system:
cd ix-x.x.x/src
make
kldload ./if_ix.ko
5. To assign an IP address to the interface, enter the following,
where "X" is the interface number for the device:
ifconfig ixX <IP_address>
6. Verify that the interface works. Enter the following, where
"<IP_address>" is the IP address for another machine on the same
subnet as the interface that is being tested:
ping <IP_address>
7. If you want the driver to load automatically when the system is
booted:
cd ix-x.x.x/src
make
make install
Then edit "/boot/loader.conf", and add the following line:
if_ix_load="YES"
8. If you want the device to connect to a network without manual
intervention after a boot, edit "/etc/rc.conf" and create the
appropriate "ifconfig_<driver>X" entry:
ifconfig_ixX="<ifconfig_settings>"
Example usage:
ifconfig_ix0="inet 192.168.10.1 netmask 255.255.255.0"
Note:
For assistance, see the ifconfig man page.
Configuration and Tuning
========================
Important System Configuration Changes
--------------------------------------
* Change the file "/etc/sysctl.conf", and add the line:
hw.intr_storm_threshold: 0
The default is 1000.
* Best throughput results are seen with a large MTU; use 9710 if
possible. The default number of descriptors per ring is 1024.
Increasing this may improve performance, depending on your use case.
* If you have a choice, run on a 64-bit OS rather than a 32-bit OS.
Speed and Duplex Configuration
------------------------------
In addressing speed and duplex configuration issues, you need to
distinguish between copper-based adapters and fiber-based adapters.
In the default mode, an Intel(R) Ethernet Network Adapter using copper
connections will attempt to auto-negotiate with its link partner to
determine the best setting. If the adapter cannot establish link with
the link partner using auto-negotiation, you may need to manually
configure the adapter and link partner to identical settings to
establish link and pass packets. This should only be needed when
attempting to link with an older switch that does not support auto-
negotiation or one that has been forced to a specific speed or duplex
mode. Your link partner must match the setting you choose. 1 Gbps
speeds and higher cannot be forced. Use the autonegotiation
advertising setting to manually set devices for 1 Gbps and higher.
Caution:
Only experienced network administrators should force speed and
duplex or change autonegotiation advertising manually. The settings
at the switch must always match the adapter settings. Adapter
performance may suffer or your adapter may not operate if you
configure the adapter differently from your switch.
An Intel(R) Ethernet Network Adapter using fiber-based connections,
however, will not attempt to auto-negotiate with its link partner
since those adapters operate only in full duplex and only at their
native speed.
By default, the adapter auto-negotiates the speed and duplex of the
connection. If there is a specific need, the ifconfig utility can be
used to configure the speed and duplex settings on the adapter.
Note:
For the Intel(R) Ethernet Connection X552 10 GbE SFP+, you must
specify the desired speed.
Example usage:
ifconfig ixX <IP_address> media 100baseTX mediaopt full-duplex
Note:
Only use "mediaopt" to set the driver to full-duplex. If "mediaopt"
is not specified and you are not running at gigabit speed, the
driver defaults to half-duplex.
If the interface is currently forced to 100 full duplex, you must use
this command to change to half duplex:
ifconfig ixX <IP_address> media 100baseTX -mediaopt full-duplex
This driver supports the following media type options:
autoselect:
Enables auto-negotiation for speed and duplex.
10baseT/UTP:
Sets speed to 10 Mbps. Use the "ifconfig mediaopt" option to select
full-duplex mode.
100baseTX:
Sets speed to 100 Mbps. Use the "ifconfig mediaopt" option to
select full-duplex mode.
1000baseTX:
Sets speed to 1000 Mbps. In this case, the driver supports only
full-duplex mode.
1000baseSX:
Sets speed to 1000 Mbps. In this case, the driver supports only
full-duplex mode.
For more information on the ifconfig utility, refer to the ifconfig
man page.
Jumbo Frames
------------
Jumbo Frames support is enabled by changing the Maximum Transmission
Unit (MTU) to a value larger than the default value of 1500.
Use the ifconfig command to increase the MTU size. For example, enter
the following where "X" is the interface number:
ifconfig ixX mtu 9000
To confirm an interface's MTU value, use the ifconfig command.
To confirm the MTU used between two specific devices, use:
route get <destination_IP_address>
Note:
* The maximum MTU setting for jumbo frames is 9710. This corresponds
to the maximum jumbo frame size of 9728 bytes.
* This driver will attempt to use multiple page sized buffers to
receive each jumbo packet. This should help to avoid buffer
starvation issues when allocating receive packets.
* Packet loss may have a greater impact on throughput when you use
jumbo frames. If you observe a drop in performance after enabling
jumbo frames, enabling flow control may mitigate the issue.
* For 82599-based network connections, if you are enabling jumbo
frames in a virtual function (VF), jumbo frames must first be
enabled in the physical function (PF). The VF MTU setting cannot
be larger than the PF MTU.
VLANS
-----
To create a new VLAN interface:
ifconfig <vlan_name> create
To associate the VLAN interface with a physical interface and assign a
VLAN ID, IP address, and netmask:
ifconfig <vlan_name> <ip_address> netmask <subnet_mask> vlan <vlan_id>
vlandev <physical_interface>
Example:
ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ix0
In this example, all packets will be marked on egress with 802.1Q VLAN
tags, specifying a VLAN ID of 10.
To remove a VLAN interface:
ifconfig <vlan_name> destroy
Checksum Offload
----------------
Checksum offloading supports both TCP and UDP packets and is supported
for both transmit and receive.
Checksum offloading can be enabled or disabled using ifconfig. Both
transmit and receive offloading will be either enabled or disabled
together. You cannot enable/disable one without the other.
To enable checksum offloading:
ifconfig ixX rxcsum
To disable checksum offloading:
ifconfig ixX -rxcsum
To confirm the current setting:
ifconfig ixX
Look for the presence or absence of the following line:
options=3 <RXCSUM,TXCSUM>
See the ifconfig man page for further information.
TSO
---
TSO (TCP Segmentation Offload) supports both IPv4 and IPv6. TSO can be
disabled and enabled using the ifconfig utility or sysctl.
Note:
TSO requires Tx checksum, if Tx checksum is disabled, TSO will also
be disabled.
To enable/disable TSO in the stack:
sysctl net.inet.tcp.tso=0 (or 1 to enable it)
Doing this disables/enables TSO in the stack and affects all installed
adapters.
To disable BOTH TSO IPv4 and IPv6, where "X" is the number of the
interface in use:
ifconfig ixX -tso
To enable BOTH TSO IPv4 and IPv6:
ifconfig ixX tso
You can also enable/disable IPv4 TSO or IPv6 TSO individually. Simply
replace "tso" or "-tso" in the above command with "tso4" or "tso6".
For example, to disable TSO IPv4:
ifconfig ixX -tso4
To disable TSO IPv6:
ifconfig ixX -tso6
LRO
---
LRO (Large Receive Offload) may provide Rx performance improvement.
However, it is incompatible with packet-forwarding workloads. You
should carefully evaluate the environment and enable LRO when
possible.
To enable:
ifconfig ixX lro
It can be disabled by using:
ifconfig ixX -lro
Link-Level Flow Control (LFC)
-----------------------------
Ethernet Flow Control (IEEE 802.3x) can be configured with sysctl to
enable receiving and transmitting pause frames for ix. When transmit
is enabled, pause frames are generated when the receive packet buffer
crosses a predefined threshold. When receive is enabled, the transmit
unit will halt for the time delay specified when a pause frame is
received.
Note:
You must have a flow control capable link partner.
Flow Control is enabled by default.
The ix driver also supports a "hw.ix.flow_control" load-time parameter
to set the initial flow control configuration on all supported
interfaces in a system. You can set the tunable using the kenv command
and then reload the driver, or by adding a line to "/boot/loader.conf"
and rebooting the system.
Use sysctl to change the flow control settings for a single interface
without reloading the driver.
The available values for flow control are:
* 0 = Disable flow control
* 1 = Enable Rx pause
* 2 = Enable Tx pause
* 3 = Enable Rx and Tx pause
Examples:
* To set the tunable in the config file, add the following line to
"/boot/loader.conf" and then reboot the system:
hw.ix.flow_control=3
* To set the tunable using kenv, use the following command and then
reload the driver:
kenv hw.ix.flow_control=3
* To remove the tunable and return the system to its default settings:
kenv -u hw.ix.flow_control
* To enable a flow control setting with sysctl:
sysctl dev.ix.<interface #>.fc=3
* To disable flow control using sysctl:
sysctl dev.ix.<interface #>.fc=0
Note:
* The ix driver requires flow control on both the port and link
partner. If flow control is disabled on one of the sides, the port
may appear to hang on heavy traffic.
* For 82598 backplane cards entering 1 gigabit mode, flow control
default behavior is changed to off. Flow control in 1 gigabit mode
on these devices can lead to transmit hangs.
* The VF driver does not have access to flow control. It must be
managed from the host side.
DMAC
----
Valid Range: 0, 41-10000
This parameter enables or disables the DMA Coalescing (DMAC) feature.
Values are in microseconds and set the internal DMA Coalescing
internal timer. DMAC is available on Intel(R) X550 (and later) based
adapters.
DMA (Direct Memory Access) allows the network device to move packet
data directly to the system's memory, reducing CPU utilization.
However, the frequency and random intervals at which packets arrive do
not allow the system to enter a lower power state. DMA Coalescing
allows the adapter to collect packets before it initiates a DMA event.
This may increase network latency but also increases the chances that
the system will enter a lower power state. Turning on DMA Coalescing
may save energy. DMA Coalescing must be enabled across all active
ports in order to save platform power.
Interrupt Throttle Rate (ITR) should be set to dynamic. When "ITR=0",
DMA Coalescing is automatically disabled.
Firmware Logs
-------------
The ix driver allows you to generate firmware logs on Intel(R) E610
Ethernet Controller devices for supported categories of events, to
help debug issues with Customer Support. Firmware logs are enabled by
default. Refer to the Intel(R) Ethernet Adapters and Devices User
Guide for an overview of this feature and additional tips.
At a high level, you must do the following to capture a firmware log:
1. Set the configuration for the firmware log.
2. Perform the necessary steps to generate the issue you are trying to
debug.
3. Capture the firmware log.
4. Stop capturing the firmware log.
5. Reset your firmware log settings as needed.
6. Work with Customer Support to debug your issue.
Note:
Firmware logs are generated in a binary format and must be decoded
by Customer Support. Information collected is related only to
firmware and hardware for debug purposes.
Once the driver is loaded, it will create the fw_log sysctl node under
the debug section of the driver's sysctl list. The driver groups these
events into categories, called "modules." Supported modules include:
general:
General (Bit 0)
ctrl:
Control (Bit 1)
link:
Link Management (Bit 2)
link_topo:
Link Topology Detection (Bit 3)
dnl:
Link Control Technology (Bit 4)
i2c:
I2C (Bit 5)
sdp:
SDP (Bit 6)
mdio:
MDIO (Bit 7)
adminq:
Admin Queue (Bit 8)
hdma:
Host DMA (Bit 9)
lldp:
LLDP (Bit 10)
dcbx:
DCBx (Bit 11)
dcb:
DCB (Bit 12)
xlr:
XLR (function-level resets; Bit 13)
nvm:
NVM (Bit 14)
auth:
Authentication (Bit 15)
vpd:
Vital Product Data (Bit 16)
iosf:
Intel On-Chip System Fabric (Bit 17)
parser:
Parser (Bit 18)
sw:
Switch (Bit 19)
scheduler:
Scheduler (Bit 20)
txq:
TX Queue Management (Bit 21)
acl:
ACL (Access Control List; Bit 22)
post:
Post (Bit 23)
watchdog:
Watchdog (Bit 24)
task_dispatch:
Task Dispatcher (Bit 25)
mng:
Manageability (Bit 26)
synce:
SyncE (Bit 27)
health:
Health (Bit 28)
tsdrv:
Time Sync (Bit 29)
pfreg:
PF Registration (Bit 30)
mdlver:
Module Version (Bit 31)
You can change the verbosity level of the firmware logs. You can set
only one log level per module, and each level includes the verbosity
levels lower than it. For instance, setting the level to "normal" will
also log warning and error messages. Available verbosity levels are:
* 0 = none
* 1 = error
* 2 = warning
* 3 = normal
* 4 = verbose
To set the desired verbosity level for a module, use the following
sysctl command and then register it:
sysctl dev.ix.<interface #>.debug.fw_log.severity.<module>=<level>
For example:
sysctl dev.ix.0.debug.fw_log.severity.link=1
sysctl dev.ix.0.debug.fw_log.severity.link_topo=2
sysctl dev.ix.0.debug.fw_log.register=1
To log firmware messages before the driver initializes, use the kenv
command to set the tunable. The "on_load" setting tells the device to
register the variable as soon as possible during driver load. For
example:
kenv dev.ix.0.debug.fw_log.severity.link=1
kenv dev.ix.0.debug.fw_log.severity.link_topo=2
kenv dev.ix.0.debug.fw_log.on_load=1
To view the firmware logs and redirect them to a file, use the
following command:
dmesg > log_output
Note:
Logging a large number of modules or too high of a verbosity level
will add extraneous messages to dmesg and could hinder debug
efforts.
Debug Dump
----------
Intel(R) E610 Ethernet Controller devices support debug dump, which
allows you to obtain runtime register values from the firmware for
"clusters" of events and then write the results to a single dump file,
for debugging complicated issues in the field.
This debug dump contains a snapshot of the device and its existing
hardware configuration, such as switch tables, transmit scheduler
tables, and other information. Debug dump captures the current state
of the specified cluster(s) and is a stateless snapshot of the whole
device.
Note:
* Like with firmware logs, the contents of the debug dump are not
human-readable. You must work with Customer Support to decode the
file.
* Debug dump is per device, not per PF.
* Debug dump writes all information to a single file.
To generate a debug dump file in FreeBSD, do the following:
1. Specify the cluster(s) to include in the dump file, using a bitmask
and the following command:
sysctl dev.ix.<interface #>.debug.dump.clusters=<bitmask>
* To print the complete cluster bitmask and parameter list to the
screen, pass the "-d" argument. For example:
sysctl -d dev.ix.0.debug.dump.clusters
* Possible bitmask values for "clusters" are:
* 0x1 - Link
* 0x2 - Full CSR Space, excluding RCW registers
For example, to dump the Link cluster, use the following:
sysctl dev.ix.0.debug.dump.clusters=0x1
To dump full CSR Space, excluding RCW registers, use the
following:
sysctl dev.ix.0.debug.dump.clusters=0x2
To dump all clusters, use the following:
sysctl dev.ice.0.debug.dump.clusters=0
Note:
Using "0" will skip Manageability Transactions data.
* If you don't specify a cluster, the driver will dump all clusters
to a single file.
2. Issue the debug dump command, using the following:
sysctl -b dev.ix.<interface #>.debug.dump.dump=1 > dump.bin
Note:
* The driver will not receive the command if you do not write "1" to
the sysctl.
* Replace "dump.bin" above with the file name you want to use.
* If you did not specify any clusters, the dump file will include
information about all clusters.
To clear the clusters bitmap before a subsequent debug dump and then
do the dump:
sysctl dev.ix.0.debug.dump.clusters=0
sysctl dev.ix.0.debug.dump.dump=1
Known Issues/Troubleshooting
============================
UDP Stress Test Dropped Packet Issue
------------------------------------
Under small packet UDP stress with the ix driver, the system may drop
UDP packets due to socket buffers being full. Setting the driver Intel
Ethernet Flow Control variables to the minimum may resolve the issue.
Error message "ixX: could not setup receive structures"
-------------------------------------------------------
Attempting to configure larger MTUs with a large numbers of processors
may generate the error message "ixX: could not setup receive
structures."
When using the ix driver with RSS autoconfigured based on the number
of cores (the default setting) and that number is larger than 4,
increase the memory resources allocated for the mbuf pool as follows:
Add the following to the "sysctl.conf" file for the system:
kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=262144
Lower than expected performance
-------------------------------
Some PCIe x8 slots are actually configured as x4 slots. These slots
have insufficient bandwidth for full line rate with dual port and quad
port devices. In addition, if you put a PCIe v4.0 or v3.0-capable
adapter into a PCIe v2.x slot, you cannot get full bandwidth. The
driver detects this situation and writes one of the following messages
in the system log:
PCI-Express bandwidth available for this card is not sufficient for optimal
performance. For optimal performance a x8 PCI-Express slot is required.
or:
PCI-Express bandwidth available for this device may be insufficient for
optimal performance. Please move the device to a different PCI-e link with
more lanes and/or higher transfer rate.
If this error occurs, moving your adapter to a true PCIe v3.0 x8 slot
will resolve the issue.
"no PRT entry for x.x.INTx" error on pcib4 in dmesg
---------------------------------------------------
On systems that use legacy interrupts, you may see the following in
dmesg:
pci4: on pcib4
pcib4: no PRT entry for <x>.0.INTA
pcib4: no PRT entry for <x>.0.INTB
For example:
pci4: on pcib4
pcib4: no PRT entry for 3.0.INTA
pcib4: no PRT entry for 3.0.INTB
To work around the issue, add the following lines to
"/boot/loader.conf", where "<x>" is the number found in dmesg message
(either 3 or 4):
hw.pci0.<x>.0.INTA.irq=16
hw.pci0.<x>.0.INTB.irq=17
Support
=======
For general information, go to the Intel support website at
http://www.intel.com/support/.
If an issue is identified with the released source code on a supported
kernel with a supported adapter, email the specific information
related to the issue to freebsd@intel.com.
Copyright(c) 1999 - 2024 Intel Corporation.
Trademarks
==========
Intel is a trademark or registered trademark of Intel Corporation or
its subsidiaries in the United States and/or other countries.
Other names and brands may be claimed as the property of others.
I have been struggling with this for some days now.
I have a e610 xt4 cards and have been able to compile the official driver from Intel, move the .ko file into /boot/modules (like is explained at thread https://forum.opnsense.org/index.php?topic=21517.msg101148#msg101148) and gotten the driver to sort of work.
I do see the ports of the card in ifconfig but when I attach a cable to it then ifconfig always shows status as "no carrier".
I named my driver something like if_ix_updated.ko and that is the driver being shown in the output of kldstat. Also at close to the top of dmesg I see a message about not being able to load if_ix.ko since if_ix_updated.ko is already loaded, so seems that my driver takes precedence over the inbuilt driver.
But I am wondering if the Intel driver is causing me problems because the kernel expects the builtin if_ix driver but gets an external one from Intel instead? Does that mean I *have* to recompile the kernel and exclude the if_ix driver from it?