Messages

You had responded to another user with "I am having similar issue on unraid, post #110", but their issue was with a 2MB nvm, yours is 226v 1MB nvm.
We also have a good amount of people who said the issue went away after upgrading the nvm, no other ASPM tweaks done.

Sorry, i have not followed the thread so deeply and oversight this.

I cant find anywhere where Intel states they have underlying silicon issue in i226 chips. Maybe its supply chain issue? The git page (notes) does suggest they found something with that L state being too slow, but still not very clear to me if that's a systemic issue related to silicon, nvm, or supply chain. Some/most of these lingering reports of issues do seem to be related to items bought from questionable sources (no name brands from Aliexpress, ebay, etc).

I am also baffled as to why that igc change (the git page) hard codes out the one L level. We can disable some/all ASPM states for specific devices.
eg; echo disabled | sudo tee /sys/bus/pci/devices/0000:00:1f.0/link/l1.2_aspm

I also cant find any source for that.
With "but it seems to ben an hardware (silicon) issue to me" i meant i assume that's an non fixable hardware issue.
Why? Because when it would be fixable they have done it by: setting an correct register value or fix the firmware (nvm).
But they have disabled the L1.2 substrate completely in the driver for all newer linux kernels.

Again: That are my personal thought's and assumptions.

[BUT Finally!!!!!!!!!!!!!!!!!!!!!!!!!!!!]

BUT Finally!!!!!!!!!!!!!!!!!!!!!!!!!!!!

The i226-v driver bug i am reffering to in #110 posts (performance loss when ASPM is on) is fixed in unraid 7.3.0-rc1 (kernel 6.18.23).
Here is the commit which fixed the issue: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0325143b59c6c6d79987afc57d2456e7a20d13b7
Basically disabling aspm L1.2, preventing the performance issue with minimal higher power consumption.


So with this commit, i can recommend the Intel i226-v for low power opnsense/unraid builds WITH full ASPM support and no performance loss.
The c-states rabbit hole has become smaller for me

It's interesting, and maybe someone can shed some light.
That igc fix (jul 2025) seems to identify an issue related to i226 nvm from 2024 (https://lore.kernel.org/intel-wired-lan/15248b4f-3271-42dd-8e35-02bfc92b25e1@intel.com/), but it does not explicitly name the nvm version in the 2025 git fix.

If I follow the git igc update, their "fix" does not leave you with a "fully functional ASPM", it basically cripples a feature of ASPM in order to keep things from being problematic, also known as a workaround, it's not a fix.

And then the mystery, does this latest nvm we have use fixed code that would not cause problems if the igc code removes the workaround?

It almost appears like the kernel side is moving faster than the nvm, but technically it's the nvm that needed fixing, yet Intel submits igc "fixes" instead of releasing new fixed nvm. Probably because updating a distro kernel is easier than trying to have end-users flash hardware? Not sure.

All that said, you mention you installed the latest nvm and still had issue with ASPM fully active. Using what is mention in the git igc fix, perhaps there's still an issue with i226 itself or that 2MB nvm. What exact version of the i226 do you have? I suspect we match the git issue as described to that (your) specific i226 hardware version with that specific 2MB nvm. The git page (seems like Intel words) seems to indicate maybe a silicon issue, something they cant fix in nvm?

With the igc workaround as seen in the git commit, that PCIe Link state is disable for ALL 226's. So on one side some got a workaround, but for others using different i226, perhaps they get dinged because their system can no longer use that PCIe power state.

You are right, disabling L1.2 is a workaround rather than a fix.
But according to the developer it's a marginal power consumption between L1.1 and L1.2.
So for me before the kernel "workaround", i had to disable ASPM just for the card all together,
as a side effect my CPU stuck at C3 and did not go deeper.
If i enabled ASPM for the NIC, it was stuck @250 Mbit/s after an ASPM "sleep" but my CPU was reaching C6,
which results in an much lower overal power consumption for the whole system.

My hardware is an typical aliexpress i226-V 1MB card (like mentioned in the mailing list).
Back than i have upgraded my rather old nvm from 2.13 to 2.32 (see post #117).

I have not followed or studied the issue which lead to disabling the L1.2 substrate in the driver,
but it seems to ben an hardware (silicon) issue to me, which is not fixable with nvm,
or it just is not fixed in 2.32 as for now.
But for this to be answered, i think we have to reach out to the intel kernel developers.

Just i someone finds this thread with the same issues i had.
Here, on post #210 is the solution: https://forum.opnsense.org/index.php?topic=48695.210

[BUT Finally!!!!!!!!!!!!!!!!!!!!!!!!!!!!]

I have been using a "Fenvi" branded AliExpress 2 port i226v card since December without any issues.  Updated the card using the 2mb file.  Can sustain 2Gbps throughput.  I have officially 1.5Gbps cable internet that is over provisioned.

You can have a look in the BIOS.  Most BIOS have a setting to turn off ASPM or pci power management. 

Same model as mine.
I got to flash it correctly with the 2MB file too, but the goal was to fix the ASPM compatibilities to avoid having to disable it. (having it disabled is a 10w increase in power consumption)
The way I got it stable is forcing ASPM in bios, forcing it in the kernel cmdline and forcing the "performance" power profile, that disables ASPM for all devices. If it's disabled in BIOS, the card crashed on boot or crashes as soon as a cable is connected.
The OPNsense VM was behaving perfectly when the card was passed through because ASPM was automatically disabled for the device.

If after doing the NVM update and the iface still experiences something odd that seems related to ASPM, then I suspect an issue not related to NVM.

For a Aliexpress 226 pcie card, I might suspect there's a pcie issue between card and host mobo and/or OS. I would start looking there (log messages, etc).

Yes there's a pcie switch on the card (the card is a pcie x1). ASPM issues are probably related to the switch chip, at this point.

Hi,

i am had similar performance issues on unraid (see my post #110) in this thread.
Also hoping to fix it with an update of the nvm firmware, sadly not.

BUT Finally!!!!!!!!!!!!!!!!!!!!!!!!!!!!

The i226-v driver bug i am reffering to in #110 posts (performance loss when ASPM is on) is fixed in unraid 7.3.0-rc1 (kernel 6.18.23).
Here is the commit which fixed the issue: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0325143b59c6c6d79987afc57d2456e7a20d13b7
Basically disabling aspm L1.2, preventing the performance issue with minimal higher power consumption.


So with this commit, i can recommend the Intel i226-v for low power opnsense/unraid builds WITH full ASPM support and no performance loss.
The c-states rabbit hole has become smaller for me 😁

Hope this helps somebody to escape the c-states rabbit hole.

I finally upgraded my rather old 2.13 to 2.32.

Do you still have the 2.13 firmware backup? If so, can you share the hash (such as MD5) of the 2.13 files?

I have a card with 2.13 firmware but I was never able to upgrade to 2.32. I began to think there might be something special for my card's firmware. It will be great to see if mine is the same as yours. Thanks in advance!

here is my backup of 2.13: https://drive.google.com/file/d/1XVVNRI5ieIVpNAA7GTV5qA7a9RX8lDTL/view?usp=sharing

@BrandyWine
Sorry i meant Mb = Mbit.

I have found similar reports on the net regarding slow speeds when ASPM is enabled:
https://www.reddit.com/r/intelnuc/comments/1951rgk/very_slow_speeds_with_i226v_on_intel_nuc13anbi5/
https://lists.osuosl.org/pipermail/intel-wired-lan/Week-of-Mon-20241202/045288.html

When i am reading this correctly, it seems a driver issue which seems to fixed already in windows driver but not linux?

@BrandyWine With "link" i meant the PCIe link speed. Sorry but i dont get it. ASPM is designed to lower or cut off the PCIe link when the device is idle. So when it's not idle the whole PCIe link bandwidth should be available, hence also the 2,5GBE ethernet speed.

A few months back i had an realtek RTL8125B where ASPM was working as it should, up to the point where unraid (driver or linux kernel) removed ASPM support.
You had to use an own driver for it to get i working again, but ASPM was working fine with always 2,5 GBE speeds.

There is clearly an issue in the mainboard BIOS, card firmware, driver or linux kernel when it's stuck @ ~600 MB/s with ASPM on.

I will leave it off for now for the card.

Sure 2,5 GB/a will draw more power than GB speeds, but i think this could be handled by firmware and/or driver.

Like so:
If ASPM mode enabled and pci-e link is in an low power mode while bandwidth hits an certain threshold for an specified amount of time, card should requesting system to put link out of low power mode so 2,5 GB/s will be reached. If bandwidth is below an certain threshold, low power mode will be activated again.

This way we can have our systems/CPU's can reach higher C states and safe more juice out of our walls.

There's a boot kernel cmdline argument to disable ASPM. Almost every *nix kernel has it. 

And yep, I think it was mentioned in this thread, there are various flashing utils depending on type of OS.

I don't think REPLACES is a match tag like you think. IIRC it tells the flash util that the flashing will proceed if the supplied ETID is matched. You can target the nic by MAC, or without any targeting at all, so the util provides some safeguards when you want to target just a subset of all 226's, etc. As example, if some 226's were already at 2.32 but others were older, you could target just the older ETID's. Kinda same thing using MAC as an attribute for target. The utils need updating, they are currently crude and convoluted, and the docs are not clear. When the nic device has area for OROM, then things get even more complicated.

Glad to hear you got your slackware 226's updated.

Thx for your detailed answer.
I don't want to disable ASPM all together.
With my approach I can have ASPM enabled for all other devices and just disable it for the i226-B, with this I am reaching C3 on my system an save a little bit of juice.

P.S.: when enabling ASPM for the i226-V card I am reaching C6 and saving another 2-3 Watts. Sadly with just 600 MB/s

[dmesg]

You should keep ASPM disabled.

You listed command as "dmsg", should that be "dmesg |grep EEPROM"

unraid? Hmmmmm, slackware.
I guess 1st step will be to understand that customized distro. Does igc show up at all?
run lspci -vv and lsmod -v , find your i226v and driver module for it. It's possible the unraid folks compiled the Intel igc source to a different name if igc does not show up.

After that, I would use the flash procedures in the thread and flash them to v2.32. Highly probable they need updating. Use the correct flash utility for linux. I suspect after updating the 226's your issues should go away, but still disable ASPM because 2.5Gb needs more power to operate and if ASPM enters a lower power state it can cause Tx/Rx issues if the nic does not re-negotiate link down to 1Gb. The driver should handle this scenario, but it's hard to diagnose because if the the device does go into lower power state but the actual link info stays at 2.5G, which it can't do because of the power state it's in.

Btw, you can use the flash utility to just probe the hardware for info.

Since this is an OPNsense forum I will suggest you can get much more info for unraid (slackware) from linuxquestions org

Sorry i meant "dmesg |grep EEPROM".
Yes, the log has some igc stuff but the etrack version was missing.

I finally upgraded my rather old 2.13 to 2.32.
The key in unraid was to use the linux I225 nvmupdate64e binary from the "Intel® Ethernet Adapter Complete Driver Pack" from here: https://www.intel.com/content/www/us/en/download/15084/intel-ethernet-adapter-complete-driver-pack.html

The other process was the same:

Inventory before (XXXXXXXXXXXX is your MAC):

Code Select Expand

./nvmupdate64e -i  -l

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.41.3.1
Copyright(C) 2013 - 2024 Intel Corporation.

Config file will not be read.
Inventory
[00:001:00:00]: Intel(R) Ethernet Controller I226-V
        Alternate MAC address is not set.
        Flash inventory started.
        Shadow RAM inventory started.
        Shadow RAM inventory finished.
        Flash inventory finished.
[00:001:00:00]: Intel(R) Ethernet Controller I226-V
        Vendor                 : 8086
        Device                 : 125C
        Subvendor              : 8086
        Subdevice              : 0000
        Revision               : 4
        LAN MAC                : XXXXXXXXXXXX
        Alt MAC                : 000000000000
        SAN MAC                : 000000000000
        ETrackId               : 80000286
        SerialNumber           : XXXXXXXXXXXXXXXX
        NVM Version            : 2.19(2.13)
        PBA                    : G23456-000
        VPD status             : Not set
        VPD size               : 0
        NVM update             : No config file entry
          checksum             : Valid


modified/checked nvm.cfg (pay attention to the REPLACES! Which must be the old ETrackId id):
But laso check all other entries like VENDOR, ....

Code Select Expand

CURRENT FAMILY: 1.0.0
CONFIG VERSION: 1.20.0

; NIC device
BEGIN DEVICE
DEVICENAME: Intel(R) Ethernet Controller I226-V
VENDOR: 8086
DEVICE: 125C
SUBVENDOR: 8086
SUBDEVICE: 0000
NVM IMAGE: FXVL_125C_V_1MB_2.32.bin
EEPID: 80000425
RESET TYPE: REBOOT
REPLACES: 80000286
END DEVICE

Update, directly over SSH (XXXXXXXXXXXX is your MAC):

Code Select Expand

./nvmupdate64e -b -l nvm.log -m XXXXXXXXXXXX-f -u -c nvm.cfg

Inventory after (XXXXXXXXXXXX is your MAC):

Code Select Expand

./nvmupdate64e -i  -l

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.41.3.1
Copyright(C) 2013 - 2024 Intel Corporation.

Config file will not be read.
Inventory
[00:001:00:00]: Intel(R) Ethernet Controller I226-V
        Alternate MAC address is not set.
        Flash inventory started.
        Shadow RAM inventory started.
        Shadow RAM inventory finished.
        Flash inventory finished.
[00:001:00:00]: Intel(R) Ethernet Controller I226-V
        Vendor                 : 8086
        Device                 : 125C
        Subvendor              : 8086
        Subdevice              : 0000
        Revision               : 4
        LAN MAC                : XXXXXXXXXXXX
        Alt MAC                : 000000000000
        SAN MAC                : 000000000000
        ETrackId               : 80000425
        SerialNumber           : XXXXXXXXXXXXXXXX
        NVM Version            : 2.50(2.32)
        PBA                    : G23456-000
        VPD status             : Not set
        VPD size               : 0
        NVM update             : No config file entry
          checksum             : Valid

After that check nvm.log and reboot your device.
The log shoukld say succeed please reboot.

Sadly, my ASPM issue is not solved with the new version.

If i activate ASPM for the card:

Code Select Expand

echo 1 > /sys/bus/pci/devices/0000:01:00.0/link/l1_aspm

and run an iperf 3 after some idle time the bandwidth drops to ~600 MB/s...

Code Select Expand

iperf3 -c 192.168.0.5 -t 200
Connecting to host 192.168.0.5, port 5201
[  5] local 192.168.0.1 port 62245 connected to 192.168.0.5 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  67.9 MBytes   563 Mbits/sec
[  5]   1.01-2.00   sec  68.0 MBytes   576 Mbits/sec
[  5]   2.00-3.00   sec  67.2 MBytes   564 Mbits/sec
[  5]   3.00-4.00   sec  68.9 MBytes   577 Mbits/sec
[  5]   4.00-5.00   sec  69.1 MBytes   580 Mbits/sec
[  5]   5.00-6.01   sec  72.0 MBytes   600 Mbits/sec
[  5]   6.01-7.01   sec  68.0 MBytes   572 Mbits/sec
[  5]   7.01-8.01   sec  32.2 MBytes   269 Mbits/sec
[  5]   8.01-9.00   sec  41.8 MBytes   354 Mbits/sec
[  5]   9.00-10.01  sec  33.4 MBytes   278 Mbits/sec
[  5]  10.01-11.00  sec  49.4 MBytes   417 Mbits/sec
[  5]  11.00-12.01  sec  24.0 MBytes   200 Mbits/sec
[  5]  12.01-13.01  sec  33.1 MBytes   276 Mbits/sec
[  5]  13.01-14.01  sec  45.0 MBytes   378 Mbits/sec
[  5]  14.01-15.00  sec  31.5 MBytes   268 Mbits/sec
[  5]  15.00-16.01  sec  48.4 MBytes   400 Mbits/sec
[  5]  16.01-17.01  sec  28.4 MBytes   240 Mbits/sec
[  5]  17.01-18.00  sec  32.2 MBytes   271 Mbits/sec

So i will leave this in my boot script to disable ASPM for the card!

Code Select Expand

echo 0 > /sys/bus/pci/devices/0000:01:00.0/link/l1_aspm

iperf with ASPM disabled...

Code Select Expand

[  5]  37.01-38.01  sec   283 MBytes  2.37 Gbits/sec
[  5]  38.01-39.01  sec   285 MBytes  2.37 Gbits/sec
[  5]  39.01-40.01  sec   280 MBytes  2.37 Gbits/sec
[  5]  40.01-41.01  sec   283 MBytes  2.37 Gbits/sec
[  5]  41.01-42.01  sec   283 MBytes  2.37 Gbits/sec
[  5]  42.01-42.10  sec  26.9 MBytes  2.38 Gbits/sec

Anyway, thx for this really informative thread.
Now my card is up to date and i have to hope for an kernel/driver fix for my issue.

Hi guys.
FIRST THX FOR THIS NICE INFORMATIVE THREAD!

Running an I226-V in on an unraid server 7.2.0 (kernel 6.12.24) and have ASPM Issues.
When ASPM L0 is enabled for the card it just can handle ~300mBit/s after some idle time.
I have to disable ASPM for the card so it reaches 2,5 Gbit/s all the time.

Sadly the "dmsg | grep igc" doesnt show me the EEPROM firmware version.

@BrandyWine and others which have updated the firmware:

• Do you had this issue i am describing?
• Has the update solved the issue and is ASPM finally workling?

I know this is OPNSense forum, but would be nice if we can figure this out.

Here is an reference link to powertop on unraid: https://github.com/fenrus75/powertop/issues/143

thx

Hey guys.

First thanks a lot for your efforts von the NVM update and the ASPM issues with the I226.
I and others experiencing same issues in unraid, regarding ASPM, see here: https://github.com/fenrus75/powertop/issues/143

When we activating "PEG ASPM" in BIOS the NIC throughput of the I226-V drops significantly.
So we are currently disabling "PEG ASPM" in BIOS to get arround this issue.

Sure, the drawback is that the cpu does'nt reach higher c-states than C2/C3, or in other words, it draws more power from the wall.

When i got you right, you are trying to disable ASPM on the card all together.
Our goal (with powertop on unraid) is to enable it and the card should/could handle it, to save more energy.

Do you have any infos if that's a silicon issue and maybe is or can be fixed with an NVM update?
Maybe thats possible with disable a portion (L0, L1, ...) off the ASPM features on the card to preserve "some" ASPM feature and save energy.

Would be nice to hear your thought's.