upgrade to 25.7.2 from 25.7.1 killed my wan interface (intel i210)

[lebowski]

I'm sorry, i'm a bit confused here. I'm not very familiar with *nix systems. I tried to follow up on @BrandyWine 's post, where he mentioned:

Also check if the proper driver is listed from kernel (the if_ items, etc), because there was a kernel update.
ls /boot/kernel | grep -v kernel

My ifconfig -a shows me "igc" for the coppers, so in kernel list i see "if_igc.ko", etc.

So i did that, logged in with putty in my opnsense box, and ran ls /boot/kernel | grep -v kernel . After this, i got a long list of modules, where i found if_igb.ko and if_igc.ko . I'm actually not sure what i'm looking at .

After reading his remark for the thirtied time ;-) , i know understand the ifconfig -a remark, and yes, now i understand that the interface name was meant. In my case it is igb0 and igb1.

The result of kldstat is:

# kldstat
Id Refs Address                Size Name
 1   56 0xffffffff80200000  216dad8 kernel
 2    3 0xffffffff8236e000    908a0 pf.ko
 3    1 0xffffffff823ff000     4be0 if_enc.ko
 4    1 0xffffffff82404000     3c10 pflog.ko
 5    1 0xffffffff82408000   5e9300 zfs.ko
 6    1 0xffffffff829f2000    1e280 opensolaris.ko
 7    1 0xffffffff82a11000    11a78 pfsync.ko
 8    1 0xffffffff82a23000    16650 if_lagg.ko
 9    2 0xffffffff82a3a000     3558 if_infiniband.ko
10    1 0xffffffff82a3e000     aa30 if_gre.ko
11    1 0xffffffff82a4a000     fb90 carp.ko
12    1 0xffffffff82a5a000     ed60 if_bridge.ko
13    2 0xffffffff82a69000     8990 bridgestp.ko
14    1 0xffffffff83710000     4250 ichsmb.ko
15    1 0xffffffff83715000     2178 smbus.ko
16    1 0xffffffff83718000     3360 uhid.ko
17    1 0xffffffff8371c000     3360 wmt.ko
18    1 0xffffffff83720000     4364 ums.ko
19    1 0xffffffff83725000     4850 nullfs.ko
20    1 0xffffffff8372a000     20f0 coretemp.ko

I don't know how to find out which flavour my i210 controller is without pulling the entire firewall out of the 19" rack. Is there an easier way to find out?
According to this website, it should be the i210-AT nic.

[Patrick M. Hausen]

Try

Code Select Expand

grep igb /var/run/dmesg.boot
and

Code Select Expand

pciconf -lv
That should give you enough information to identify the exact model of your interfaces.

[lebowski]

[1] igb0: <Intel(R) I210 (Copper)> port 0xd000-0xd01f mem 0x88900000-0x8897ffff,                          0x88980000-0x88983fff at device 0.0 on pci2
[1] igb0: EEPROM V3.16-0 eTrack 0x800004d9
[1] igb0: Using 1024 TX descriptors and 1024 RX descriptors
[1] igb0: Using 4 RX queues 4 TX queues
[1] igb0: Using MSI-X interrupts with 5 vectors
[1] igb0: Ethernet address: **:**:**:**:**:**
[1] igb0: netmap queues/slots: TX 4/1024, RX 4/1024
[1] igb1: <Intel(R) I210 (Copper)> port 0xc000-0xc01f mem 0x88700000-0x8877ffff,                          0x88780000-0x88783fff at device 0.0 on pci5
[1] igb1: EEPROM V3.16-0 eTrack 0x800004d9
[1] igb1: Using 1024 TX descriptors and 1024 RX descriptors
[1] igb1: Using 4 RX queues 4 TX queues
[1] igb1: Using MSI-X interrupts with 5 vectors
[1] igb1: Ethernet address: **:**:**:**:**:**
[1] igb1: netmap queues/slots: TX 4/1024, RX 4/1024
[17] igb1: link state changed to UP
[29] igb1: link state changed to DOWN
[33] igb1: link state changed to UP
[548] igb1: link state changed to DOWN
[552] igb1: link state changed to UP
[1] igb0: <Intel(R) I210 (Copper)> port 0xd000-0xd01f mem 0x88900000-0x8897ffff,                          0x88980000-0x88983fff at device 0.0 on pci2
[1] igb0: EEPROM V3.16-0 eTrack 0x800004d9
[1] igb0: Using 1024 TX descriptors and 1024 RX descriptors
[1] igb0: Using 4 RX queues 4 TX queues
[1] igb0: Using MSI-X interrupts with 5 vectors
[1] igb0: Ethernet address: **:**:**:**:**:**
[1] igb0: netmap queues/slots: TX 4/1024, RX 4/1024
[1] igb1: <Intel(R) I210 (Copper)> port 0xc000-0xc01f mem 0x88700000-0x8877ffff,                          0x88780000-0x88783fff at device 0.0 on pci5
[1] igb1: EEPROM V3.16-0 eTrack 0x800004d9
[1] igb1: Using 1024 TX descriptors and 1024 RX descriptors
[1] igb1: Using 4 RX queues 4 TX queues
[1] igb1: Using MSI-X interrupts with 5 vectors
[1] igb1: Ethernet address: **:**:**:**:**:**
[1] igb1: netmap queues/slots: TX 4/1024, RX 4/1024
[17] igb1: link state changed to UP
[29] igb1: link state changed to DOWN
[32] igb1: link state changed to UP
[239] igb1: link state changed to DOWN
[243] igb1: link state changed to UP
[372] igb0: link state changed to UP
[386] igb0: link state changed to DOWN
[390] igb0: link state changed to UP
[485] igb0: link state changed to DOWN
[508] igb0: link state changed to UP
[515] igb0: link state changed to DOWN
[530] igb0: link state changed to UP
[533] igb0: link state changed to DOWN
[535] igb0: link state changed to UP
[674] igb0: link state changed to DOWN
[675] igb1: link state changed to DOWN
[680] igb1: link state changed to UP
[1] igb0: <Intel(R) I210 (Copper)> port 0xd000-0xd01f mem 0x88900000-0x8897ffff,                          0x88980000-0x88983fff at device 0.0 on pci2
[1] igb0: EEPROM V3.16-0 eTrack 0x800004d9
[1] igb0: Using 1024 TX descriptors and 1024 RX descriptors
[1] igb0: Using 4 RX queues 4 TX queues
[1] igb0: Using MSI-X interrupts with 5 vectors
[1] igb0: Ethernet address: **:**:**:**:**:**
[1] igb0: netmap queues/slots: TX 4/1024, RX 4/1024
[1] igb1: <Intel(R) I210 (Copper)> port 0xc000-0xc01f mem 0x88700000-0x8877ffff,                          0x88780000-0x88783fff at device 0.0 on pci5
[1] igb1: EEPROM V3.16-0 eTrack 0x800004d9
[1] igb1: Using 1024 TX descriptors and 1024 RX descriptors
[1] igb1: Using 4 RX queues 4 TX queues
[1] igb1: Using MSI-X interrupts with 5 vectors
[1] igb1: Ethernet address: **:**:**:**:**:**
[1] igb1: netmap queues/slots: TX 4/1024, RX 4/1024
[17] igb1: link state changed to UP
[29] igb1: link state changed to DOWN
[32] igb0: link state changed to UP
[33] igb1: link state changed to UP
[1] igb0: <Intel(R) I210 (Copper)> port 0xd000-0xd01f mem 0x88900000-0x8897ffff,                          0x88980000-0x88983fff at device 0.0 on pci2
[1] igb0: EEPROM V3.16-0 eTrack 0x800004d9
[1] igb0: Using 1024 TX descriptors and 1024 RX descriptors
[1] igb0: Using 4 RX queues 4 TX queues
[1] igb0: Using MSI-X interrupts with 5 vectors
[1] igb0: Ethernet address: **:**:**:**:**:**
[1] igb0: netmap queues/slots: TX 4/1024, RX 4/1024
[1] igb1: <Intel(R) I210 (Copper)> port 0xc000-0xc01f mem 0x88700000-0x8877ffff,                          0x88780000-0x88783fff at device 0.0 on pci5
[1] igb1: EEPROM V3.16-0 eTrack 0x800004d9
[1] igb1: Using 1024 TX descriptors and 1024 RX descriptors
[1] igb1: Using 4 RX queues 4 TX queues
[1] igb1: Using MSI-X interrupts with 5 vectors
[1] igb1: Ethernet address: **:**:**:**:**:**
[1] igb1: netmap queues/slots: TX 4/1024, RX 4/1024

hostb0@pci0:0:0:0:      class=0x060000 rev=0x0e hdr=0x00 vendor=0x8086 device=0x0f00 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series SoC Transaction Register'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:2:0:     class=0x030000 rev=0x0e hdr=0x00 vendor=0x8086 device=0x0f31 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series Graphics & Display'
    class      = display
    subclass   = VGA
ahci0@pci0:0:19:0:      class=0x010601 rev=0x0e hdr=0x00 vendor=0x8086 device=0x0f23 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series SATA AHCI Controller'
    class      = mass storage
    subclass   = SATA
none0@pci0:0:26:0:      class=0x108000 rev=0x0e hdr=0x00 vendor=0x8086 device=0x0f18 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series Trusted Execution Engine'
    class      = encrypt/decrypt
pcib1@pci0:0:28:0:      class=0x060400 rev=0x0e hdr=0x01 vendor=0x8086 device=0x0f48 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 1'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:28:2:      class=0x060400 rev=0x0e hdr=0x01 vendor=0x8086 device=0x0f4c subvendor=0x15d9 subdevice=0x0816
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 3'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:28:3:      class=0x060400 rev=0x0e hdr=0x01 vendor=0x8086 device=0x0f4e subvendor=0x15d9 subdevice=0x0816
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 4'
    class      = bridge
    subclass   = PCI-PCI
ehci0@pci0:0:29:0:      class=0x0c0320 rev=0x0e hdr=0x00 vendor=0x8086 device=0x0f34 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series USB EHCI'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:      class=0x060100 rev=0x0e hdr=0x00 vendor=0x8086 device=0x0f1c subvendor=0x15d9 subdevice=0x0816
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series Power Control Unit'
    class      = bridge
    subclass   = PCI-ISA
ichsmb0@pci0:0:31:3:    class=0x0c0500 rev=0x0e hdr=0x00 vendor=0x8086 device=0x0f12 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800/CE2700 Series SMBus Controller'
    class      = serial bus
    subclass   = SMBus
igb0@pci0:2:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
pcib4@pci0:3:0:0:       class=0x060400 rev=0xab hdr=0x01 vendor=0x10b5 device=0x8605 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8605 PCI Express 4-port Gen2 Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:4:1:0:       class=0x060400 rev=0xab hdr=0x01 vendor=0x10b5 device=0x8605 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8605 PCI Express 4-port Gen2 Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib6@pci0:4:2:0:       class=0x060400 rev=0xab hdr=0x01 vendor=0x10b5 device=0x8605 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8605 PCI Express 4-port Gen2 Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib7@pci0:4:3:0:       class=0x060400 rev=0xab hdr=0x01 vendor=0x10b5 device=0x8605 subvendor=0x15d9 subdevice=0x0816
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8605 PCI Express 4-port Gen2 Switch'
    class      = bridge
    subclass   = PCI-PCI
igb1@pci0:5:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
ahci1@pci0:7:0:0:       class=0x010601 rev=0x11 hdr=0x00 vendor=0x1b4b device=0x9230 subvendor=0x1b4b subdevice=0x9230
    vendor     = 'Marvell Technology Group Ltd.'
    device     = '88SE9230 PCIe 2.0 x2 4-port SATA 6 Gb/s RAID Controller'
    class      = mass storage
    subclass   = SATA

[BrandyWine]

According to this website, it should be the i210-AT nic.

igb1@pci0:5:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533

ls /boot/kernel | grep -v kernel . After this, i got a long list of modules, where i found if_igb.ko and if_igc.ko . I'm actually not sure what i'm looking at .

Kudos for you doing some digging.
The lastest driver for i210 is v5.19.4 from July 2025. I not sure when your OS (my version of freeBSD 14.3-RELEASE-p2) was compiled, can find that on freeBSD ste, but then need to know what i210 driver code version was compiled into the kernel.

That long list you asked about shows the driver tree that is compiled into the kernel. kldstat shows you the drivers that were loaded in manually ("dynamically").

Sometimes less can be easier on the eyes
pciconf -l

1 step further is we find the device ID and check that with intel docs (https://www.intel.com/content/dam/www/public/us/en/documents/faqs/ethernet-controller-i210-i211-faq.pdf?asset=9597&-907791053.1537776712&)

Is indeed the AT fully programmed flavor.

Issues with i210, makes me wonder if the firmware version in EEPROM is causing issue? Yours looks like v3.16, other sites seem to show v3.30 "fixing security issues".
Flashing the nic firmware is a bit tricky, perhaps too hard for most.

Just for reference:
https://www.intel.com/content/www/us/en/products/details/ethernet/gigabit-network-adapters/i210-server-adapters/resource.html
https://www.intel.com/content/www/us/en/support/articles/000005790/software/manageability-products.html
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=5r8tk
https://github.com/dgrafe/i210-tools/tree/master/i210-flash

New stuff as of 7/2025
https://www.intel.com/content/www/us/en/products/sku/64400/intel-ethernet-controller-i210at/downloads.html
https://www.intel.com/content/www/us/en/download/14098/intel-network-adapter-driver-for-82575-6-82580-i350-and-i210-211-based-gigabit-network-connections-for-linux.html

The Intel I210 Ethernet controller has updatable firmware that can enhance its features and fix issues. You can update the firmware using a utility provided by Intel, which typically involves downloading the firmware package and running it on your system.

Overview of Intel I210 EEPROM Firmware

The Intel I210 Ethernet Controller requires firmware updates to enhance functionality and fix issues. The firmware is stored in the EEPROM (Electrically Erasable Programmable Read-Only Memory) and can be updated using specific utilities.
Updating the Firmware
Supported Operating Systems

    Windows
    Linux

Update Process

    Download the Firmware Update Utility:
        For Windows, download the I210_NVMUpdatePackage_v2_00_Windows.exe.
        For Linux, use the appropriate command-line tools.

    Installation Steps:
        Windows:
            Run the downloaded executable as an administrator.
            Follow the prompts to complete the update.
        Linux:
            Boot from a live USB (e.g., Ubuntu).
            Install necessary packages:
                sudo apt install gcc-12 linux-headers-$(uname -r) make ethtool
            Use the bootutil command to flash the firmware.

Important Considerations

    Ensure that the firmware version is compatible with your specific I210 model.
    Back up existing configurations before proceeding with the update.
    Be cautious, as improper updates can lead to bricking the device.

Features of the Firmware Update

    Fixes security vulnerabilities.
    Improves network stability and performance.
    Unlocks new features for enhanced functionality.

By following the correct procedures, users can successfully update the Intel I210 EEPROM firmware to maintain optimal performance and security.

[frakkin64]

makes me wonder if the firmware version in EEPROM is causing issue? Yours looks like v3.16, other sites seem to show v3.30

This probably makes the most sense, considering OP is adamant the cable is good and it was just a upgrade & reboot scenario.  I have seen this sort of issue on an HP backplane swap-out w/ HP-UX, but it was the other way around (backplane delivered newer firmware, OS was running older driver).  Oddly it rendered as NIC-related latency issues.  The infrastructure team had to bring in HP techs to diagnose.

[lebowski]

Well, this was a LOT of work. Upgrading the firmware on linux was too hard, i didn't understand that. So i created a windows to go live usb stick (which took a LONG time), booted from it (which also took a LONG time), and started the windows firmware upgrade package.

And then.... nothing 😔. The tool showed my two i210-at nics, with the message that there werent any updates available. According to the tool, my nics are on fw 3.22 now.

What i did find was other folks asking for firmware for their intel nics on the intel boards, which then got replied by the intel folks by stating that for in order to get access to the firmware for a  intel nic, you have to sign a NDA approval (!) to even get possibly access to the firmware. This is not a matter of just downloading the fw.

An other thing is that sometimes the hardware vendor for the appliance can provide nic firmwares, but in my case, there is no such update package on the supermicro website for my appliance, so that too is a dead end.

But today it seems is the day of miracles. Because i did not touch any network cable, i only tried updating the nic firmwares, but i did reboot the opnsense appliance a couple of times and now miraculously my wan port negotiated at 1000baset and my network internet throughput is passing the 100mbit mark again. My subscription is for 125/25 and my actual speed is now 128.2 / 25.78 . Nice. Let's hope and pray it stays this way.



-edit: my tinypic screenshot isn't loading, don't know how or why...

[frakkin64]

What i did find was other folks asking for firmware for their intel nics on the intel boards, which then got replied by the intel folks by stating that for in order to get access to the firmware for a  intel nic, you have to sign a NDA approval (!) to even get possibly access to the firmware. This is not a matter of just downloading the fw.

Yeah you would have to go back to SuperMicro.  At least your not dealing with a Chinese device manufacturer who will never provide a firmware update ever.  I'm kind of tired of the Chinese dumping for everything, literal garbage that just goes into my local landfill.

[BrandyWine]

[1] igb1: EEPROM V3.16-0 eTrack 0x800004d9

It's was on 3.22 now? Hmmm, that does not seem to align with your output post #17. Your EEPROM looks like 3.16, or it was.
Also, I do believe v3.30 or higher is available from somewhere.

Maybe the tool did do upgrade to 3.22. What does "dmesg | grep igb" show you now?

Also to note, integrator's like SM, Asrock, ASUS, HP Dell, etc etc etc, don't care to build firmware upgrades for flashable devices unless they really need to. Some like Dell and others will provide flash images when security issues demand it. Sometimes the integrator builds the firmware to suit their needs or restrictions, other times a device manufacturer has full-featured firmware. As example, some years ago I had a DVD-rw drive in my PC, but the model I had could not rw Blueray. Well, turns out it was only firmware change that was needed, I exported my existing firmware and applied a mod, now it can rw blueray (had to do it this way because the firmware is very specific to some things in the drive). The maker of dvd drive sells the blueray-enabled device for more money, same drive, different words on the box, different firmware.

The fun world of firmware and drivers.

[BrandyWine]

Was it the Dell tool you used? Their download for a "v3.30" looks like i210 tool + firmware.
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=5r8tk

[lebowski]

[1] igb1: EEPROM V3.16-0 eTrack 0x800004d9

It's was on 3.22 now? Hmmm, that does not seem to align with your output post #17. Your EEPROM looks like 3.16, or it was.
Also, I do believe v3.30 or higher is available from somewhere.

Maybe the tool did do upgrade to 3.22. What does "dmesg | grep igb" show you now?

Also to note, integrator's like SM, Asrock, ASUS, HP Dell, etc etc etc, don't care to build firmware upgrades for flashable devices unless they really need to. Some like Dell and others will provide flash images when security issues demand it. Sometimes the integrator builds the firmware to suit their needs or restrictions, other times a device manufacturer has full-featured firmware. As example, some years ago I had a DVD-rw drive in my PC, but the model I had could not rw Blueray. Well, turns out it was only firmware change that was needed, I exported my existing firmware and applied a mod, now it can rw blueray (had to do it this way because the firmware is very specific to some things in the drive). The maker of dvd drive sells the blueray-enabled device for more money, same drive, different words on the box, different firmware.

The fun world of firmware and drivers.

Yes it was on 3.22 . The screenshot which i made (and failed to upload) shows 3.22 being the decimal version, and 3.16 the hex version. This is a bit confusing. The tool showed for both nics "Status: update not available". And then "Tool execution completed with the following status: Device not found." (although before this sentence it shows the both i210 nics).

[lebowski]

Was it the Dell tool you used? Their download for a "v3.30" looks like i210 tool + firmware.
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=5r8tk

Yes i used the Dell tool, booted windows to go and ran it on windows, but it gave me "update not available".