Odd LAN Performance Issue

[BCCHowdy]

Hello!

I'm pretty much new to opnsense and homebrew routers, but I can find my way around ok.  A little about my setup:

Router platform is a Qotom J1900 with 4x intel nics.  4 core celeron (atom?) @ 2.00 Ghz. 8 GB memory 32GB msata
5/1 Mbit wireless connection to the interwebs connected to WAN port (em0) on router box, DHCP.
LAN connected to em1, serves DHCP
From there connected to a couple 8 port switches, raspberry pi running ubiquity unifi controller, two ubiquity AP's around the house.

My wireless internet connection isn't that stellar but I have very limited options in my area.  I may try to switch over to a cell modem setup soon just to get more bandwidth.  I don't have much turned on in opnsense beyond unbound for local DNS.  I have a few DNS overrides configured for static IP's.  Nothing else really.  No firewall rules, no packet inspection, proxy, vpn.

I've noticed recently that the LAN interface will drop to 100 Mbit full duplex when it should be gigabit.  If I reboot the box then it'll reset to gigabit.  Check it the next morning, back to 100 mbit.  This tells me I have data issues on this port that's dropping the speed, but I can't figure out where.

I rebooted so it would reconnect at 1GB, installed iperf3 and did some tests.  I had the router be the server and had a PC be the client.  The connection would consistently max out at half speed, 500 Mbit.  I tried this a few times throughout the day and the results were consistently half of what they should be.  I tried a different cable, no result.

This has been bothering me as I'd expect this platform to push 1 Gb without hesitation.  I did notice when bandwidth testing CPU usage would go to approx 50%.  Today I logged in, saw the interface had rolled back to 100 Mbit again.  I forced it back to 1 Gb through the webui and decided to reverse roles.  I set the router box to be the client and my PC to be the host.  Here's the result:

Code: [Select]

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   108 MBytes   905 Mbits/sec    0    163 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]   8.00-9.00   sec   103 MBytes   859 Mbits/sec   42   1.41 KBytes       
[  5]   9.00-10.00  sec  41.8 MBytes   350 Mbits/sec  134   2.83 KBytes       
[  5]  10.00-11.00  sec   983 KBytes  8.08 Mbits/sec   11   17.1 KBytes       
[  5]  11.00-12.00  sec  47.5 MBytes   398 Mbits/sec   31   51.3 KBytes       
[  5]  12.00-13.00  sec  54.4 MBytes   456 Mbits/sec   53    110 KBytes       
[  5]  13.00-14.00  sec   103 MBytes   867 Mbits/sec    8    154 KBytes       
[  5]  14.00-15.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  15.00-16.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  16.00-17.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  17.00-18.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  18.00-19.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  19.00-20.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  20.00-21.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  21.00-22.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  22.00-23.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  23.00-24.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  24.00-25.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  25.00-26.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  26.00-27.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  27.00-28.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  28.00-29.00  sec   112 MBytes   940 Mbits/sec    0    208 KBytes       
[  5]  29.00-30.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  30.00-31.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  31.00-32.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  32.00-33.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  33.00-34.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  34.00-35.00  sec  90.4 MBytes   758 Mbits/sec    0    208 KBytes       
[  5]  35.00-36.00  sec  71.2 MBytes   597 Mbits/sec    0    208 KBytes       
[  5]  36.00-37.00  sec  65.7 MBytes   551 Mbits/sec    0    208 KBytes       
[  5]  37.00-38.00  sec   108 MBytes   910 Mbits/sec    0    208 KBytes       
[  5]  38.00-39.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  39.00-40.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  40.00-41.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  41.00-42.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  42.00-43.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  43.00-44.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  44.00-45.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  45.00-46.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  46.00-47.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  47.00-48.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  48.00-49.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  49.00-50.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  50.00-51.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  51.00-52.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  52.00-53.00  sec   112 MBytes   942 Mbits/sec    0    208 KBytes       
[  5]  53.00-54.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  54.00-55.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  55.00-56.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  56.00-57.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  57.00-58.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  58.00-59.00  sec   112 MBytes   941 Mbits/sec    0    208 KBytes       
[  5]  59.00-60.00  sec   112 MBytes   940 Mbits/sec    0    208 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  6.25 GBytes   895 Mbits/sec  279             sender
[  5]   0.00-60.00  sec  6.25 GBytes   895 Mbits/sec                  receiver

279 retries on sending packets out, but you'll notice I'm at near gigabit speed outgoing.  Incoming is still around half.  One run through iperf when the router was sending I didn't receive any data for 15-20 seconds!  I'm looking at the Interfaces:Overview page for LAN and it shows In/Out errors of 7/0 for the first time.  Usually this would be 0/0 but the interface would be rolled back to 100 Mbit.

Looks like I have a problem and it doesn't appear to be the cable.  I don't know where to look to try and find more info on what's going on or why the interface is dropping link speed.  I've looked at the logs and diagnostics pages but don't see anything of value.  Is there somewhere in the freebsd logs I can look?  I may just have to try and configure one of the other ports as LAN and see if I have a hardware port issue on the box.

I realize it's silly to be concerned about not running the LAN port at gigabit speed when I'm only receiving 5 Mbit, but perhaps this accounts for some of the internet slowness I've been experiencing.  I've been blaming my ISP when maybe it's been my hardware all along.

Thanks for any input.  Happy to provide any more detail.

-Howdy

[bartjsmit]

Make sure you can get to the console physically. In the web interface set: Interfaces, LAN, speed and duplex. Fix on gigabit/full. Add the interface statistics widget to your dashboard and check for errors.

If you lose connection, log into the console, select option 8 and run:

ifconfig <LAN interface> media 100baseTX mediaopt full-duplex

It is more likely to be a issue at layer 2 or below.

Bart...

[BertM]

Hi BCCHowdy,

It does indeed sound like a autonegotiation/duplex mismatch issue.
A nice explanation that can help you better understand this can found at
https://www.appliedtrust.com/resources/performance/untangling-ethernet-performance-problems

Kind regards,
Bert

[nbfedafdf]

What are your settings on these:
Hw checksum offload
Hw TSO
Hw LRO
If you leave them at the default (disable), do you still get the issue ?

Seems a bit like the e1000e Linux driver trouble with unexpected resets.
Actually, pretty much all of the intel gigabit adapters. I guess Intel don't bother making a driver that works properly at gigabit.

Anyway, some workarounds collected from different places:
1. Disable gso, gro and tso
2. Turning off Active-State Power Management, pcie_aspm=off
3. Disable C1E in BIOS/UEFI

(1.) Worked here so no need to try 2 & 3