Where is TCP processed - CPU? Networkinterface? Elsewhere?

[OPNenthu]

Understood, although there might be a reason why Protectli found that ASPM must be disabled globally rather than disabling it on a per-device basis with PCI sysctls.  Usually you don't use the nuclear option unless there's a reason, but who knows.

[Seimus]

To be honest usually you want to disable. e.g force disabled ASPM off globally on OS level cause the per-device per-line disabling may not work always as it should... I usually disable ASPM in BIOS on everything or if not available or I have suspicions its not enough I force disable it globally in Linux.

https://wiki.archlinux.org/title/Power_management#Active_State_Power_Management

Regards,
S.

[chemlud]

OK, so best bet is:

Code Select Expand

pcie_aspm=off
added to kernel boot line and reboot.

Will try... :-)

[chemlud]

I have here after reboot:

Code Select Expand

sudo dmesg | grep ASPM
[    0.018934] [      T0] PCIe ASPM is disabled
[    0.121764] [      T1] acpi PNP0A08:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI]


Code Select Expand

iperf -p 45678 -c FAST -t 30 -y C -P 1
20260121133626,SLOW,FAST,45678,1,0.0-30.0,3478388800,926208295

Other direction:

20260121133925,FAST,SLOW,45678,1,0.0-30.1,1693319232,450764744
So nothing really changed.

[OPNenthu]

Does openSUSE perhaps also require this step?

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

This is needed for RHEL-based distros, but not Debian.  I'm unsure about openSUSE.  According to ChatGPT (attached) it is needed.

There's an interesting note there that pcie_aspm=off is only telling Linux to not change it from whatever the firmware has set, so if coreboot has it enabled (but hides the option from the user) then Linux will also use it.*  Maybe the pcie_aspm.policy=performance option is better.

Apologies @chemlud.  I don't want to send you down a rabbit hole that might not be fruitful, but I'd rather share this extra info for you to decide.

* of course, ChatGPT is a serial liar, so... there's that!

[chemlud]

dmesg reports ASPM disabled. I edit the kernel boot line in YaST, that does the grub mkinit magic stuff automatically ;-)

[chemlud]

With pcie_aspm.policy=performance in the kernel boot line I see:

Code Select Expand

sudo dmesg | grep ASPM
[sudo] password for root:
[    0.121549] [      T1] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]

and the troughput is:

20260121142126,SLOW,FAST,45678,1,0.0-30.0,3487694912,929146430

20260121142412,FAST,SLOW,45678,1,0.0-30.0,1691353152,450561206

...so no change here.

[chemlud]

With EEE disabled in kernel I get:

Code Select Expand

sudo dmesg | grep EEE
[    0.000000] [      T0] Command line: BOOT_IMAGE=/boot/vmlinuz-6.18.5-1-default root=UUIDxxxxxxx splash=silent net.ifnames=0 kvm.enable_virt_at_load=0 ipv6.disable=1 quiet igb.EEE=0 mitigations=auto
[    0.018795] [      T0] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.18.5-1-default root=UUIDxxxxxxxx splash=silent net.ifnames=0 kvm.enable_virt_at_load=0 ipv6.disable=1 quiet igb.EEE=0 mitigations=auto

and still

20260121143745,FAST,SLOW,45678,1,0.0-30.1,1692139584,450325523

so no progress here.

[Seimus]

For the fun itself, can you spin the iperf and just test from the same device?

Basically in one CLI window start it as server and in another as client.

Regards,
S.

[chemlud]

Then I see on SLOW:

20260121150015,127.0.0.1,48856,127.0.0.1,45678,1,0.0-30.0,117651669056,31373002456

Tells me what? ;-)

[chemlud]

Ladies and Gentlemen,

apparently we are running out of options here. Don't want to mess with the driver, as long as the machine is remote, just in case I loose network connectivity.

EEE, ASPM, offloading apparently not the culprit, what might be an option?

[Seimus]

Tells me what? ;-)

Tells you if there is something on the device itself beyond the NIC that could case the behaviour.


Yea the next step would be to mess with the driver. Best do it indeed locally.

Regards,
S.