Messages

Ladies and Gentlemen,

apparently we are running out of options here. Don't want to mess with the driver, as long as the machine is remote, just in case I loose network connectivity.

EEE, ASPM, offloading apparently not the culprit, what might be an option?

Then I see on SLOW:

20260121150015,127.0.0.1,48856,127.0.0.1,45678,1,0.0-30.0,117651669056,31373002456

Tells me what? ;-)

With EEE disabled in kernel I get:

Code Select Expand

sudo dmesg | grep EEE
[    0.000000] [      T0] Command line: BOOT_IMAGE=/boot/vmlinuz-6.18.5-1-default root=UUIDxxxxxxx splash=silent net.ifnames=0 kvm.enable_virt_at_load=0 ipv6.disable=1 quiet igb.EEE=0 mitigations=auto
[    0.018795] [      T0] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.18.5-1-default root=UUIDxxxxxxxx splash=silent net.ifnames=0 kvm.enable_virt_at_load=0 ipv6.disable=1 quiet igb.EEE=0 mitigations=auto

and still

20260121143745,FAST,SLOW,45678,1,0.0-30.1,1692139584,450325523

so no progress here.

With pcie_aspm.policy=performance in the kernel boot line I see:

Code Select Expand

sudo dmesg | grep ASPM
[sudo] password for root:
[    0.121549] [      T1] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]

and the troughput is:

20260121142126,SLOW,FAST,45678,1,0.0-30.0,3487694912,929146430

20260121142412,FAST,SLOW,45678,1,0.0-30.0,1691353152,450561206

...so no change here.

dmesg reports ASPM disabled. I edit the kernel boot line in YaST, that does the grub mkinit magic stuff automatically ;-)

I have here after reboot:

Code Select Expand

sudo dmesg | grep ASPM
[    0.018934] [      T0] PCIe ASPM is disabled
[    0.121764] [      T1] acpi PNP0A08:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI]


Code Select Expand

iperf -p 45678 -c FAST -t 30 -y C -P 1
20260121133626,SLOW,FAST,45678,1,0.0-30.0,3478388800,926208295

Other direction:

20260121133925,FAST,SLOW,45678,1,0.0-30.1,1693319232,450764744
So nothing really changed.

OK, so best bet is:

Code Select Expand

pcie_aspm=off
added to kernel boot line and reboot.

Will try... :-)

@OPNenthu Thanks for reading, yes, ASPM and offloading are apparently off the list at that point.

EEE (enabled, but apparently "inactive", see above) and the "wrong" driver (8169, which works perfectly on another Tumbleweed with old ATOM CPU with legacy BIOS and Realtek 8168 hardware, btw...) are on the list.

Not much left, apparently...

RE 1: Checked the status for offloading:

Code Select Expand

sudo ethtool --show-offload eth0
[sudo] password for root:
Features for eth0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off
        tx-tcp-accecn-segmentation: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
So GSO and TXO apparently off at the moment

@klevinsourd, that was my impression, explaining the thread title ;-)

Will try the suggestions and report back :-)

RE ASPM

I did

Code Select Expand

sudo lspci -vvv


01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 07)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: I/O ports at 2000 [size=256]
        Region 2: Memory at d1300000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at d1200000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 1
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10W TEE-IO-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00000800
        Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: No such device
                Not readable
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
        Kernel driver in use: r8169
        Kernel modules: r8169
so apparently

Code Select Expand

...
LnkCtl: ASPM Disabled
...
ASPM should not be in the way, or?

... BTW based on this when you only run the update for the slow, result is the same correct?

Yes, SLOW ist slow for years, all other Tumbleweeds are normal download speed.

RE ASPM

Apparently regulated in BIOS (not reachable with Coreboot) or via a kernel boot command.

Have to check when not remote, don't want to loose the machine by tinkering with kerne boot.

Thanks for reading.

RE 2: Mirrors are "hardcoded" and identical on FAST and SLOW. The download of weekly packages on FAST and SLOW is simultaneously in the attachment of OP, so how/why only one client should be rate limited?

Congestion algorithm? Hmmm... ;-)

RE: BBR

I had in Tumbleweed:

Code Select Expand

sudo sysctl net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_available_congestion_control = reno cubic

Then I followed this:

https://www.techrepublic.com/article/how-to-enable-tcp-bbr-to-improve-network-speed-on-linux/

and did

Code Select Expand

sudo nano /etc/sysctl.conf

-> and add the following two lines:

net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr


Then I have after

Code Select Expand

sudo sysctl -p

Code Select Expand

sudo sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr

To make things even more complicated:

- I downloaded a large file on SLOW with FF 147.0 and get an amazing 100MB/s...

What's going on here? Only the weekly updates of Tumbleweed slow?!?! But it's not the server, see OP.

1 What Distro you are using?
2 What realtek NIC does it use?
3 What is the realtek driver loaded for the NIC?
4 Did you try to upgrade the BIOS?
5 What are the temps during high volume downloads/uploads?
6 Can you post the NIC statistics (counters)?
7 Did you disabled ASPM?

1 opensuse Tumbleweed

2 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 07)
  Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller

3 Kernel driver in use: r8169
  Kernel modules: r8169

4 Yes, but it's Coreboot, the company Libretrend dose not provide any newer Coreboot version

5 Unremarkable temps, cooling is appropriate

6 like this? On the SLOW i have:
   

Code Select Expand

sudo ethtool -S eth0
    NIC statistics:
    tx_packets: 3175999
    rx_packets: 8314948
    tx_errors: 0
    rx_errors: 0
    rx_missed: 36
    align_errors: 0
    tx_single_collisions: 0
    tx_multi_collisions: 0
    unicast: 8313262
    broadcast: 1686
    multicast: 0
    tx_aborted: 0
    tx_underrun: 0
7 No. It's an onboard NIC, so relevant?
   

Code Select Expand

sudo ethtool --show-eee eth0 
    EEE settings for eth0:
    enabled - inactive
    0 (us)
        Supported EEE link modes:  100baseT/Full
                                  1000baseT/Full
        Advertised EEE link modes:  100baseT/Full
                                    1000baseT/Full
        Link partner advertised EEE link modes:  Not reported