OPNsense Forum
English Forums => Hardware and Performance => Topic started by: BCCHowdy on October 06, 2017, 11:21:39 pm
-
Hello!
I'm pretty much new to opnsense and homebrew routers, but I can find my way around ok. A little about my setup:
Router platform is a Qotom J1900 with 4x intel nics. 4 core celeron (atom?) @ 2.00 Ghz. 8 GB memory 32GB msata
5/1 Mbit wireless connection to the interwebs connected to WAN port (em0) on router box, DHCP.
LAN connected to em1, serves DHCP
From there connected to a couple 8 port switches, raspberry pi running ubiquity unifi controller, two ubiquity AP's around the house.
My wireless internet connection isn't that stellar but I have very limited options in my area. I may try to switch over to a cell modem setup soon just to get more bandwidth. I don't have much turned on in opnsense beyond unbound for local DNS. I have a few DNS overrides configured for static IP's. Nothing else really. No firewall rules, no packet inspection, proxy, vpn.
I've noticed recently that the LAN interface will drop to 100 Mbit full duplex when it should be gigabit. If I reboot the box then it'll reset to gigabit. Check it the next morning, back to 100 mbit. This tells me I have data issues on this port that's dropping the speed, but I can't figure out where.
I rebooted so it would reconnect at 1GB, installed iperf3 and did some tests. I had the router be the server and had a PC be the client. The connection would consistently max out at half speed, 500 Mbit. I tried this a few times throughout the day and the results were consistently half of what they should be. I tried a different cable, no result.
This has been bothering me as I'd expect this platform to push 1 Gb without hesitation. I did notice when bandwidth testing CPU usage would go to approx 50%. Today I logged in, saw the interface had rolled back to 100 Mbit again. I forced it back to 1 Gb through the webui and decided to reverse roles. I set the router box to be the client and my PC to be the host. Here's the result:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 108 MBytes 905 Mbits/sec 0 163 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 8.00-9.00 sec 103 MBytes 859 Mbits/sec 42 1.41 KBytes
[ 5] 9.00-10.00 sec 41.8 MBytes 350 Mbits/sec 134 2.83 KBytes
[ 5] 10.00-11.00 sec 983 KBytes 8.08 Mbits/sec 11 17.1 KBytes
[ 5] 11.00-12.00 sec 47.5 MBytes 398 Mbits/sec 31 51.3 KBytes
[ 5] 12.00-13.00 sec 54.4 MBytes 456 Mbits/sec 53 110 KBytes
[ 5] 13.00-14.00 sec 103 MBytes 867 Mbits/sec 8 154 KBytes
[ 5] 14.00-15.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 15.00-16.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 16.00-17.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 17.00-18.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 18.00-19.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 19.00-20.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 20.00-21.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 21.00-22.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 22.00-23.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 23.00-24.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 24.00-25.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 25.00-26.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 26.00-27.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 27.00-28.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 28.00-29.00 sec 112 MBytes 940 Mbits/sec 0 208 KBytes
[ 5] 29.00-30.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 30.00-31.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 31.00-32.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 32.00-33.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 33.00-34.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 34.00-35.00 sec 90.4 MBytes 758 Mbits/sec 0 208 KBytes
[ 5] 35.00-36.00 sec 71.2 MBytes 597 Mbits/sec 0 208 KBytes
[ 5] 36.00-37.00 sec 65.7 MBytes 551 Mbits/sec 0 208 KBytes
[ 5] 37.00-38.00 sec 108 MBytes 910 Mbits/sec 0 208 KBytes
[ 5] 38.00-39.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 39.00-40.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 40.00-41.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 41.00-42.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 42.00-43.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 43.00-44.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 44.00-45.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 45.00-46.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 46.00-47.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 47.00-48.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 48.00-49.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 49.00-50.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 50.00-51.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 51.00-52.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 52.00-53.00 sec 112 MBytes 942 Mbits/sec 0 208 KBytes
[ 5] 53.00-54.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 54.00-55.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 55.00-56.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 56.00-57.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 57.00-58.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 58.00-59.00 sec 112 MBytes 941 Mbits/sec 0 208 KBytes
[ 5] 59.00-60.00 sec 112 MBytes 940 Mbits/sec 0 208 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 6.25 GBytes 895 Mbits/sec 279 sender
[ 5] 0.00-60.00 sec 6.25 GBytes 895 Mbits/sec receiver
279 retries on sending packets out, but you'll notice I'm at near gigabit speed outgoing. Incoming is still around half. One run through iperf when the router was sending I didn't receive any data for 15-20 seconds! I'm looking at the Interfaces:Overview page for LAN and it shows In/Out errors of 7/0 for the first time. Usually this would be 0/0 but the interface would be rolled back to 100 Mbit.
Looks like I have a problem and it doesn't appear to be the cable. I don't know where to look to try and find more info on what's going on or why the interface is dropping link speed. I've looked at the logs and diagnostics pages but don't see anything of value. Is there somewhere in the freebsd logs I can look? I may just have to try and configure one of the other ports as LAN and see if I have a hardware port issue on the box.
I realize it's silly to be concerned about not running the LAN port at gigabit speed when I'm only receiving 5 Mbit, but perhaps this accounts for some of the internet slowness I've been experiencing. I've been blaming my ISP when maybe it's been my hardware all along.
Thanks for any input. Happy to provide any more detail.
-Howdy
-
Make sure you can get to the console physically. In the web interface set: Interfaces, LAN, speed and duplex. Fix on gigabit/full. Add the interface statistics widget to your dashboard and check for errors.
If you lose connection, log into the console, select option 8 and run:
ifconfig <LAN interface> media 100baseTX mediaopt full-duplex
It is more likely to be a issue at layer 2 or below.
Bart...
-
Hi BCCHowdy,
It does indeed sound like a autonegotiation/duplex mismatch issue.
A nice explanation that can help you better understand this can found at
https://www.appliedtrust.com/resources/performance/untangling-ethernet-performance-problems
Kind regards,
Bert
-
What are your settings on these:
Hw checksum offload
Hw TSO
Hw LRO
If you leave them at the default (disable), do you still get the issue ?
Seems a bit like the e1000e Linux driver trouble with unexpected resets.
Actually, pretty much all of the intel gigabit adapters. I guess Intel don't bother making a driver that works properly at gigabit.
Anyway, some workarounds collected from different places:
1. Disable gso, gro and tso
2. Turning off Active-State Power Management, pcie_aspm=off
3. Disable C1E in BIOS/UEFI
(1.) Worked here so no need to try 2 & 3