[solved] Intel i226 Firmware (see post #39)

[BrandyWine]

Did you see the device actually reboot when you did the nvram update? It may have soft rebooted (re-booting just the OS)?
IIRC, The eeupdate util does not work on freeBSD.

Your device is the 1st reported one of such issue.

From all the util commands you posted, the devices do appear to be there, and not sure why one or more may have moved locations on PCI bus. Maybe perhaps you have other issue with that hardware. A bad solder joint can create the gremlin, etc.

I will suggest that the update procedure include removing power for 1min after a device shutdown. Then power it back up. But noted, you should not need to do this as long as the device actually is rebooted.

[pfry]

[...]
I will suggest that the update procedure include removing power for 1min after a device shutdown. Then power it back up. But noted, you should not need to do this as long as the device actually is rebooted.

Good idea. A reset (pulling PWR_OK down) should do it, but few mini boxes have a reset button. Heck, most don't even have a power switch.

[caplam]

Did you see the device actually reboot when you did the nvram update? It may have soft rebooted (re-booting just the OS)?
IIRC, The eeupdate util does not work on freeBSD.

Your device is the 1st reported one of such issue.

From all the util commands you posted, the devices do appear to be there, and not sure why one or more may have moved locations on PCI bus. Maybe perhaps you have other issue with that hardware. A bad solder joint can create the gremlin, etc.

I will suggest that the update procedure include removing power for 1min after a device shutdown. Then power it back up. But noted, you should not need to do this as long as the device actually is rebooted.

Now that you mention it, all my reboots (and there have been at least 5) prior to unplugging the power cord to move the device were done with the webgui and as i was not in the same room i didn't see it rebooting. Does a reboot initiated from the gui is a hard reboot ?

From what i posted you clearly see that device with @mac ending in 32:00 is igc5 but when i had the problem it became igc4. And at this time device with @mac ending in 31:ff was not detected (it's normally igc4).
All came back in order after power reset.
I mentionned eeupdate as it was part of the firmware package i found. I first intended to use it in EFI shell which i didn't managed to do. I am still struggling to make a bootable efi usb media with my mac.

[BrandyWine]

Does a reboot initiated from the gui is a hard reboot ?

It's an OS "reboot". It should restart from BIOS, but I guess that depends on the BIOS version and settings in BIOS.

[Panja]

Damn damn damn I have a serious hearth attack with this update.
I updated to v2.32 on 3 of the 4 NIC's on my device. 2 unused ones, which went well.
After that the 3rd one: WAN interface. (the 4th is my SSH/management interface so I left that one behind on purpose).
WAN did not come up afterwards. Rebooted 6 more times. Still no luck....
Finally I came to read the last posts in this thread about power cycling (pull the power plug) and now I'm back online!
Thanks! :-D

[pOpY2k25]

Hi guys.
FIRST THX FOR THIS NICE INFORMATIVE THREAD!

Running an I226-V in on an unraid server 7.2.0 (kernel 6.12.24) and have ASPM Issues.
When ASPM L0 is enabled for the card it just can handle ~300mBit/s after some idle time.
I have to disable ASPM for the card so it reaches 2,5 Gbit/s all the time.

Sadly the "dmsg | grep igc" doesnt show me the EEPROM firmware version.

@BrandyWine and others which have updated the firmware:

• Do you had this issue i am describing?
• Has the update solved the issue and is ASPM finally workling?

I know this is OPNSense forum, but would be nice if we can figure this out.

Here is an reference link to powertop on unraid: https://github.com/fenrus75/powertop/issues/143

thx

[BrandyWine]

You should keep ASPM disabled.

You listed command as "dmsg", should that be "dmesg |grep EEPROM"

unraid? Hmmmmm, slackware.
I guess 1st step will be to understand that customized distro. Does igc show up at all?
run lspci -vv and lsmod -v , find your i226v and driver module for it. It's possible the unraid folks compiled the Intel igc source to a different name if igc does not show up.

After that, I would use the flash procedures in the thread and flash them to v2.32. Highly probable they need updating. Use the correct flash utility for linux. I suspect after updating the 226's your issues should go away, but still disable ASPM because 2.5Gb needs more power to operate and if ASPM enters a lower power state it can cause Tx/Rx issues if the nic does not re-negotiate link down to 1Gb. The driver should handle this scenario, but it's hard to diagnose because if the the device does go into lower power state but the actual link info stays at 2.5G, which it can't do because of the power state it's in.

Btw, you can use the flash utility to just probe the hardware for info.

Since this is an OPNsense forum I will suggest you can get much more info for unraid (slackware) from linuxquestions org

[Panja]

I don't have any option in the bios for ASPM.
How can I disable it otherwise?

[meyergru]

https://forum.opnsense.org/index.php?topic=42985.0, #23

[Panja]

https://forum.opnsense.org/index.php?topic=42985.0, #23

Thanks for the reply.
Though I do not see anything related to ASPM in #23 or in the thread linked.

[meyergru]

Do not look in the thread, but in the first (tutorial) posting, point #23. It explains how to disable ASPM. You would have found several postings on how do to that if you heeded #16 from the same posting.

[Panja]

Thanks, understood!

[pOpY2k25]

You should keep ASPM disabled.

You listed command as "dmsg", should that be "dmesg |grep EEPROM"

unraid? Hmmmmm, slackware.
I guess 1st step will be to understand that customized distro. Does igc show up at all?
run lspci -vv and lsmod -v , find your i226v and driver module for it. It's possible the unraid folks compiled the Intel igc source to a different name if igc does not show up.

After that, I would use the flash procedures in the thread and flash them to v2.32. Highly probable they need updating. Use the correct flash utility for linux. I suspect after updating the 226's your issues should go away, but still disable ASPM because 2.5Gb needs more power to operate and if ASPM enters a lower power state it can cause Tx/Rx issues if the nic does not re-negotiate link down to 1Gb. The driver should handle this scenario, but it's hard to diagnose because if the the device does go into lower power state but the actual link info stays at 2.5G, which it can't do because of the power state it's in.

Btw, you can use the flash utility to just probe the hardware for info.

Since this is an OPNsense forum I will suggest you can get much more info for unraid (slackware) from linuxquestions org

Sorry i meant "dmesg |grep EEPROM".
Yes, the log has some igc stuff but the etrack version was missing.

I finally upgraded my rather old 2.13 to 2.32.
The key in unraid was to use the linux I225 nvmupdate64e binary from the "Intel® Ethernet Adapter Complete Driver Pack" from here: https://www.intel.com/content/www/us/en/download/15084/intel-ethernet-adapter-complete-driver-pack.html

The other process was the same:

Inventory before (XXXXXXXXXXXX is your MAC):

Code Select Expand

./nvmupdate64e -i  -l

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.41.3.1
Copyright(C) 2013 - 2024 Intel Corporation.

Config file will not be read.
Inventory
[00:001:00:00]: Intel(R) Ethernet Controller I226-V
        Alternate MAC address is not set.
        Flash inventory started.
        Shadow RAM inventory started.
        Shadow RAM inventory finished.
        Flash inventory finished.
[00:001:00:00]: Intel(R) Ethernet Controller I226-V
        Vendor                 : 8086
        Device                 : 125C
        Subvendor              : 8086
        Subdevice              : 0000
        Revision               : 4
        LAN MAC                : XXXXXXXXXXXX
        Alt MAC                : 000000000000
        SAN MAC                : 000000000000
        ETrackId               : 80000286
        SerialNumber           : XXXXXXXXXXXXXXXX
        NVM Version            : 2.19(2.13)
        PBA                    : G23456-000
        VPD status             : Not set
        VPD size               : 0
        NVM update             : No config file entry
          checksum             : Valid


modified/checked nvm.cfg (pay attention to the REPLACES! Which must be the old ETrackId id):
But laso check all other entries like VENDOR, ....

Code Select Expand

CURRENT FAMILY: 1.0.0
CONFIG VERSION: 1.20.0

; NIC device
BEGIN DEVICE
DEVICENAME: Intel(R) Ethernet Controller I226-V
VENDOR: 8086
DEVICE: 125C
SUBVENDOR: 8086
SUBDEVICE: 0000
NVM IMAGE: FXVL_125C_V_1MB_2.32.bin
EEPID: 80000425
RESET TYPE: REBOOT
REPLACES: 80000286
END DEVICE

Update, directly over SSH (XXXXXXXXXXXX is your MAC):

Code Select Expand

./nvmupdate64e -b -l nvm.log -m XXXXXXXXXXXX-f -u -c nvm.cfg

Inventory after (XXXXXXXXXXXX is your MAC):

Code Select Expand

./nvmupdate64e -i  -l

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.41.3.1
Copyright(C) 2013 - 2024 Intel Corporation.

Config file will not be read.
Inventory
[00:001:00:00]: Intel(R) Ethernet Controller I226-V
        Alternate MAC address is not set.
        Flash inventory started.
        Shadow RAM inventory started.
        Shadow RAM inventory finished.
        Flash inventory finished.
[00:001:00:00]: Intel(R) Ethernet Controller I226-V
        Vendor                 : 8086
        Device                 : 125C
        Subvendor              : 8086
        Subdevice              : 0000
        Revision               : 4
        LAN MAC                : XXXXXXXXXXXX
        Alt MAC                : 000000000000
        SAN MAC                : 000000000000
        ETrackId               : 80000425
        SerialNumber           : XXXXXXXXXXXXXXXX
        NVM Version            : 2.50(2.32)
        PBA                    : G23456-000
        VPD status             : Not set
        VPD size               : 0
        NVM update             : No config file entry
          checksum             : Valid

After that check nvm.log and reboot your device.
The log shoukld say succeed please reboot.

Sadly, my ASPM issue is not solved with the new version.

If i activate ASPM for the card:

Code Select Expand

echo 1 > /sys/bus/pci/devices/0000:01:00.0/link/l1_aspm

and run an iperf 3 after some idle time the bandwidth drops to ~600 MB/s...

Code Select Expand

iperf3 -c 192.168.0.5 -t 200
Connecting to host 192.168.0.5, port 5201
[  5] local 192.168.0.1 port 62245 connected to 192.168.0.5 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  67.9 MBytes   563 Mbits/sec
[  5]   1.01-2.00   sec  68.0 MBytes   576 Mbits/sec
[  5]   2.00-3.00   sec  67.2 MBytes   564 Mbits/sec
[  5]   3.00-4.00   sec  68.9 MBytes   577 Mbits/sec
[  5]   4.00-5.00   sec  69.1 MBytes   580 Mbits/sec
[  5]   5.00-6.01   sec  72.0 MBytes   600 Mbits/sec
[  5]   6.01-7.01   sec  68.0 MBytes   572 Mbits/sec
[  5]   7.01-8.01   sec  32.2 MBytes   269 Mbits/sec
[  5]   8.01-9.00   sec  41.8 MBytes   354 Mbits/sec
[  5]   9.00-10.01  sec  33.4 MBytes   278 Mbits/sec
[  5]  10.01-11.00  sec  49.4 MBytes   417 Mbits/sec
[  5]  11.00-12.01  sec  24.0 MBytes   200 Mbits/sec
[  5]  12.01-13.01  sec  33.1 MBytes   276 Mbits/sec
[  5]  13.01-14.01  sec  45.0 MBytes   378 Mbits/sec
[  5]  14.01-15.00  sec  31.5 MBytes   268 Mbits/sec
[  5]  15.00-16.01  sec  48.4 MBytes   400 Mbits/sec
[  5]  16.01-17.01  sec  28.4 MBytes   240 Mbits/sec
[  5]  17.01-18.00  sec  32.2 MBytes   271 Mbits/sec

So i will leave this in my boot script to disable ASPM for the card!

Code Select Expand

echo 0 > /sys/bus/pci/devices/0000:01:00.0/link/l1_aspm

iperf with ASPM disabled...

Code Select Expand

[  5]  37.01-38.01  sec   283 MBytes  2.37 Gbits/sec
[  5]  38.01-39.01  sec   285 MBytes  2.37 Gbits/sec
[  5]  39.01-40.01  sec   280 MBytes  2.37 Gbits/sec
[  5]  40.01-41.01  sec   283 MBytes  2.37 Gbits/sec
[  5]  41.01-42.01  sec   283 MBytes  2.37 Gbits/sec
[  5]  42.01-42.10  sec  26.9 MBytes  2.38 Gbits/sec

Anyway, thx for this really informative thread.
Now my card is up to date and i have to hope for an kernel/driver fix for my issue.

[BrandyWine]

There's a boot kernel cmdline argument to disable ASPM. Almost every *nix kernel has it. 

And yep, I think it was mentioned in this thread, there are various flashing utils depending on type of OS.

I don't think REPLACES is a match tag like you think. IIRC it tells the flash util that the flashing will proceed if the supplied ETID is matched. You can target the nic by MAC, or without any targeting at all, so the util provides some safeguards when you want to target just a subset of all 226's, etc. As example, if some 226's were already at 2.32 but others were older, you could target just the older ETID's. Kinda same thing using MAC as an attribute for target. The utils need updating, they are currently crude and convoluted, and the docs are not clear. When the nic device has area for OROM, then things get even more complicated.

Glad to hear you got your slackware 226's updated.

[pOpY2k25]

There's a boot kernel cmdline argument to disable ASPM. Almost every *nix kernel has it. 

And yep, I think it was mentioned in this thread, there are various flashing utils depending on type of OS.

I don't think REPLACES is a match tag like you think. IIRC it tells the flash util that the flashing will proceed if the supplied ETID is matched. You can target the nic by MAC, or without any targeting at all, so the util provides some safeguards when you want to target just a subset of all 226's, etc. As example, if some 226's were already at 2.32 but others were older, you could target just the older ETID's. Kinda same thing using MAC as an attribute for target. The utils need updating, they are currently crude and convoluted, and the docs are not clear. When the nic device has area for OROM, then things get even more complicated.

Glad to hear you got your slackware 226's updated.

Thx for your detailed answer.
I don't want to disable ASPM all together.
With my approach I can have ASPM enabled for all other devices and just disable it for the i226-B, with this I am reaching C3 on my system an save a little bit of juice.

P.S.: when enabling ASPM for the i226-V card I am reaching C6 and saving another 2-3 Watts. Sadly with just 600 MB/s