[SOLVED] LAN internal data transfer is slower than internet data transfer

[Sheldon]

My LAN internal data transfer was not just slower than expected, but it was even slower than a download from the internet. I ran into this issue and couldn't figure out the cause, so i started writing things down to create a forum post. At the end i was able to fix it myself and decided to still post it here, because it might help someone else.

• Have i tried to turn it off and on again? Yes.
• There is no traffic shaping configured in OPNsense.
• The CPU load on the OPNsense box doesn't seem to be much affected by traffic.
• OPNsense box is APU board with CPU AMD GX-412TC SOC (4 cores).
• Desktop-PC 1 has a 1000 MBit NIC and is connected to LAN 1 of the OPNsense box.
• Desktop-PC 2 has a 1000 MBit NIC and is connected to LAN 2 of the OPNsense box.

Code: [Select]

Test: 100 MByte transfer from internet to PC 1
Protocol: HTTPS
Duration: 22 seconds
Transfer rate: 4.8 MByte/second
Note: This is a good transfer rate for WAN to LAN.

Test: 100 MByte transfer from internet to PC 2
Protocol: HTTPS
Duration: 21 seconds
Transfer rate: 4.8 MByte/second
Note: This is a good transfer rate for WAN to LAN.

Test: 500 MByte transfer from PC 2 to PC 1
Protocol: FTP
Duration: 2m 14s
Transfer rate: 3.7 MByte/second
Note: This is a bad transfer rate for LAN to LAN. Even with my slow harddrive i expected a much higher rate.

Test: 500 MByte transfer from PC 1 to PC 2
Protocol: HTTP
Duration: 7m 33s
Transfer rate: 1.4 MByte/second
Note: This is a very bad transfer rate for LAN to LAN. Even with my slow harddrive i expected a much higher rate.

Test: 500 MByte transfer from PC 1 to OPNsense box
Protocol: scp
Duration: 19 seconds
Transfer rate: 26 MByte/second
Note: This is a good transfer rate for LAN to LAN.

Test: 500 MByte transfer from OPNsense box to PC 1
Protocol: scp
Duration: 27 seconds
Transfer rate: 18 MByte/second
Note: This is a good transfer rate for LAN to LAN.

When i first noticed the issue, the OPNsense interface configuration "Speed and duplex" was set to "Default" for both LAN 1 and LAN 2. For testing i changed this to "100baseTX full-duplex", but the transfer rate didn't change. I reverted the changes back to "Default".

In the end i got this fixed by replacing the cable between PC 2 and LAN 2. I don't know if the old cable is broken or just not plugged in correctly, but i know that the new cable increased the transfer rate between LAN 1 and LAN 2 to at least 40 MByte/second :-)

[testo_cz]

Hi

Dodgy cable ? There is always some....
If not using network switch with packet statistics, one may have a look onto several things to see if the low network layers are fine:

Code: [Select]

ifconfig
netstat -n -d -i
then going higher protocols, for example:

Code: [Select]

netstat -s -p tcp

If not 'ifconfig | grep media' , then dropped/error packets might diagnose that there is a bad cabling.
Must be checked on both sides of  the cable.

T.

[Sheldon]

@testo_cz Thank you for your additional information!

I didn't even look for packet loss, because the transfer rates seemed to be stable. Or not. When i now think back, i remember there was at least one test with a "flapping" transfer rate.

Anyway, this is very helpful information, good to know for the next time debugging :-)