Where is TCP processed - CPU? Networkinterface? Elsewhere?

[chemlud]

Hi, have a problem that bugs me for years, no solution found yet:

Two clients with Linux, same update status:

1. SLOW: Libretrend i7 with Coreboot, Realtek NIC

2. FAST: Old Dell Precision M6500 notebook with Intel NIC.

Problem: when downloading e.g. updates, the FAST ist 30-times faster than SLOW, see attached.

Same mirrors,RJ45 cables changed twice, both attached to the same switch. So no real explanation.

Yesterday I did some iperf between the two clients.

For UDP, it does not matter, which is server and whicch is client:

20260118145326,SLOW,FAST,45678,1,0.0-30.0,3935190,1048952,0.025,0,2677,0.000,0

20260118145446,FAST,SLOW,45678,1,0.0-30.0,3935190,1048950,-nan,0,-1,-0.000,0

But for TCP direction matters:

20260118144505,SLOW,FAST,45678,1,0.0-30.0,3533963328,941434163

20260118144740,FAST,SLOW,45678,1,0.0-30.1,1690173504,449722645

-------------------------

First thought: RJ45 in SLOW machine is Realtek. But I have Realtek in other machines with same linux, always maxing out the bandwith possible. So I don't think it'S simply Realtek.

Why only TCP makes a difference? Is there offloading and that doesn't work properly in the SLOW machine? Maybe due to Coreboot?

Any ideas how this difference in TCP-speed might be explainable?

:-)

[Seimus]

Your topic name is a bit intriguing.
When you think about a packet, the finial destination on a device is the CPU. If a packet is delivered its being pegged to the CPU to process it. Of cource the packet needs to be 1st processed on the NIC.

What Distro you are using?
What realtek NIC does it use?
What is the realtek driver loaded for the NIC?
Did you try to upgrade the BIOS?
What are the temps during high volume downloads/uploads?
Can you post the NIC statistics (counters)?
Did you disabled ASPM?

Regards,
S.

[chemlud]

1 What Distro you are using?
2 What realtek NIC does it use?
3 What is the realtek driver loaded for the NIC?
4 Did you try to upgrade the BIOS?
5 What are the temps during high volume downloads/uploads?
6 Can you post the NIC statistics (counters)?
7 Did you disabled ASPM?

1 opensuse Tumbleweed

2 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 07)
  Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller

3 Kernel driver in use: r8169
  Kernel modules: r8169

4 Yes, but it's Coreboot, the company Libretrend dose not provide any newer Coreboot version

5 Unremarkable temps, cooling is appropriate

6 like this? On the SLOW i have:
   

Code Select Expand

sudo ethtool -S eth0
    NIC statistics:
    tx_packets: 3175999
    rx_packets: 8314948
    tx_errors: 0
    rx_errors: 0
    rx_missed: 36
    align_errors: 0
    tx_single_collisions: 0
    tx_multi_collisions: 0
    unicast: 8313262
    broadcast: 1686
    multicast: 0
    tx_aborted: 0
    tx_underrun: 0
7 No. It's an onboard NIC, so relevant?
   

Code Select Expand

sudo ethtool --show-eee eth0 
    EEE settings for eth0:
    enabled - inactive
    0 (us)
        Supported EEE link modes:  100baseT/Full
                                  1000baseT/Full
        Advertised EEE link modes:  100baseT/Full
                                    1000baseT/Full
        Link partner advertised EEE link modes:  Not reported

[chemlud]

To make things even more complicated:

- I downloaded a large file on SLOW with FF 147.0 and get an amazing 100MB/s...

What's going on here? Only the weekly updates of Tumbleweed slow?!?! But it's not the server, see OP.

[Seimus]

I would definitely advice to disable ASPM, either via BIOS or in Linux.
ASPM enabled can do a lot of performance related problems and realtek is not excluded from this.

The NIC stats, look good, there is no errors or dirty packets seen.

In regards of your testing, you have here some interesting results;
1. Iperf > fast to slow = throughput limited
2. Linux package updates = throughput slow
3. Browser download = fast

For
1. Iperf > fast to slow
Can you try to restest this but set P2 at least to trigger multicore spread of iperf? And post the results
Try scenarios where the slow is the client as well server, and during scenario where its client try with and without the flag -R

2. Linux package updates
This one is curious, cause you can be rate limited, try to refresh your mirrors


3. Browser download
No clue about this, I would assume similar results as for Iperf, but maybe this can be due to the fact the browsers is using multiple cores to process the packets.

As well what kind of congestion algoritm are you using? Maybe you can try to switch it to BBR.


Regards,
S.

[chemlud]

Thanks for reading.

RE 2: Mirrors are "hardcoded" and identical on FAST and SLOW. The download of weekly packages on FAST and SLOW is simultaneously in the attachment of OP, so how/why only one client should be rate limited?

Congestion algorithm? Hmmm... ;-)

RE: BBR

I had in Tumbleweed:

Code Select Expand

sudo sysctl net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_available_congestion_control = reno cubic

Then I followed this:

https://www.techrepublic.com/article/how-to-enable-tcp-bbr-to-improve-network-speed-on-linux/

and did

Code Select Expand

sudo nano /etc/sysctl.conf

-> and add the following two lines:

net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr


Then I have after

Code Select Expand

sudo sysctl -p

Code Select Expand

sudo sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr

[Seimus]

RE 2: Mirrors are "hardcoded" and identical on FAST and SLOW. The download of weekly packages on FAST and SLOW is simultaneously in the attachment of OP, so how/why only one client should be rate limited?

True if this is the case. I assumed you have dynamic mirrors and downloads are in different intervals. If you would be rate limited it would be for the Public IP. But if both of them are NATed behind one, it would affect them both. BTW based on this when you only run the update for the slow, result is the same correct?

Congestion algorithm? Hmmm... ;-)

I could talk about this whole day long, but to simplify it, its the often omitted brother of the TCPs scaling window.
Sender > Responsible for the Scaling Window
Receiver > Responsible for the Congestion management

Congestion tells the sender to slow down, and how to recover from the congestion event (this not 100% correct but enough to see both of them are important).


Current enabled congestion algo, default should be at least CUBIC

Code Select Expand

sysctl net.ipv4.tcp_congestion_control
All available congestion algos

Code Select Expand

sysctl net.ipv4.tcp_available_congestion_control
For BRR you you usally need to enable some extra flags in system but nothing to hard...

Regards,
S.

[chemlud]

... BTW based on this when you only run the update for the slow, result is the same correct?

Yes, SLOW ist slow for years, all other Tumbleweeds are normal download speed.

RE ASPM

Apparently regulated in BIOS (not reachable with Coreboot) or via a kernel boot command.

Have to check when not remote, don't want to loose the machine by tinkering with kerne boot.

[chemlud]

RE ASPM

I did

Code Select Expand

sudo lspci -vvv


01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 07)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: I/O ports at 2000 [size=256]
        Region 2: Memory at d1300000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at d1200000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 1
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10W TEE-IO-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00000800
        Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: No such device
                Not readable
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
        Kernel driver in use: r8169
        Kernel modules: r8169
so apparently

Code Select Expand

...
LnkCtl: ASPM Disabled
...
ASPM should not be in the way, or?

[Seimus]

Yes, that means ASPM is disabled for the NIC.


Regards,
S.