Slow Internet behind Spectrum/Charter/TWC 300 mpbs connection

[angel2040]

Starting new topic on this as I did not want to hijack "slow internet connection".  So

I have one of these:

Mini PC J1900 Quad core CPU, 4 LAN (Intel) with 8 gigs of ram and 500 gig ssd.
4 x Intel I211AT  10/100/1000 Controller

ibg0 is my WAN interface.

root@ FW:~ # pciconf -B -l | egrep igb0
igb0@pci0:1:0:0:        class=0x020000 card=0x00008086 chip=0x15398086 rev=0x03 hdr=0x00


root@ FW:~ # dmesg | egrep igb0
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe000-0xe01f mem 0xd0900000-0xd091ffff,0xd0920000-0xd0923fff irq 16 at device 0.0 on pci1
igb0: Using MSIX interrupts with 3 vectors
igb0: Ethernet address: 00:0e:c4:ce:da:87
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
igb0: permanently promiscuous mode enabled
igb0: link state changed to UP
igb0: link state changed to DOWN
igb0: link state changed to UP
igb0: link state changed to DOWN
igb0: link state changed to UP
ifa_maintain_loopback_route: deletion failed for interface igb0: 3
ifa_maintain_loopback_route: insertion failed for interface igb0: 17
igb0: link state changed to DOWN
igb0: link state changed to UP


The cpu doesn't get over 10% and only using 14% of the 8 gigs.

So I just switched from uverse (75mps) to spectrum because you know, who doesn't want 300mps internet for $20 more per month.   

If shutdown both spectrum ARIS modem (TM1602) and the 17.1.10 opnsense FW, turn it back on, the very first speed test is 300mps down, 20 mps up.  Any subsequent tests after that I get 100mps down, 20 mps up??  I've checked the MTUs (with a laptop directly attached to the ARIS) and it's 1500.  Note, no errors or drops.

root@ FW:~ # ./speedTest
Name    Mtu Network       Address              Ipkts Ierrs Idrop     Ibytes    Opkts Oerrs     Obytes  Coll  Drop
igb0   1500 <Link#1>      00:0e:c4:ce:da:87  5793428     0     0 8302223793  1816291     0  558912816     0     0
igb0      - fe80::%igb0/6 fe80::20e:c4ff:fe        0     -     -          0       16     -        916     -     -
igb0      - 66.215.208.0/ 66-215-214-225.dh   340638     -     -  427209195       63     -       3984     -     -
Retrieving speedtest.net configuration...
Testing from Spectrum (66.215.214.225)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Ultimate Internet Access (Ontario, CA) [21.91 km]: 25.683 ms
Testing download speed................................................................................
Download: 75.60 Mbit/s
Testing upload speed................................................................................................
Upload: 20.87 Mbit/s
Name    Mtu Network       Address              Ipkts Ierrs Idrop     Ibytes    Opkts Oerrs     Obytes  Coll  Drop
igb0   1500 <Link#1>      00:0e:c4:ce:da:87  5883463     0     0 8423019739  1913060     0  590716650     0     0
igb0      - fe80::%igb0/6 fe80::20e:c4ff:fe        0     -     -          0       16     -        916     -     -
igb0      - 66.215.208.0/ 66-215-214-225.dh   430687     -     -  545606637       63     -       3984     -     -

I just bought an ARRIS SB6190 and TPC-LINK TC-7620 (marked down at Target) from the spectrum list of supported cable modems.  Called Spectrum tried SB6190, tested with direct connecting laptop, getting 300 mpbs down.  Insert my 17.1.10 opnsense FW (rebooted fresh). getting 300 mps down, retesting gives me 80-100 mps download.  Swapped it out for TP-7620, direct connected laptop is 300 mp down.  But when I insert opnsense FW, I cannot get an IP via DHCP.  Had to install SB6190 back in.

BTW, spectrum bridges the connection to opnsense because it's an routable IP.  The laptop is running windows, but I think I;m going to boot a live linux off the opnsense FW to see if it does thew same thing with linux.  Anyone know if there's a liveCD for FreeBSD 11.1?

[angel2040]

7/25/2017:
OK.  So I was reading up the on SB6190 which I left in BTW.  Seems the ARRIS modems have the PUMA6 chipset which has issues with it (latency).  So I decided to buy another Spectrum supported cable modem: netgear CM600, called spectrum "support", installed it, same issue.   300Mps down first time, then it drops down to 100.

At this point between the screaming match that ensued between myself and "tech support" guy, I not-so-kindly asked to speak with his supervisor whom I asked politely to send someone out.

7/26/2017:
This morning spectrum tech knocks on the door, power goes out on in the whole hood.  Can it get any better?  So it gave me the opportunity to explain to him my setup, what I tried, etc.  He politely said Spectrum will NOT care nor do anything about anything that leaves the wan port of the cable modem.  It's beyond their hands.  I asked him what router/FW devices work in this area and consistently are getting the service level download speeds thinking maybe 300 is not yet solid here?  He left me a netgear R6300v2 router.  With this router, I'm consistently getting 300 Mps down.  One thing that I find very disturbing is, I had to call spectrum support to get the modem WAN (DHCP) IP stick because I just hooked it up and it wasn't assigning an IP to it. 

I'm not running opnsense now.  I have a i3-based DS81 I can re-purpose.  I will probably wind up moving the opnsense config to that and retest to see if that solves the issue.

[opnfwb]

It does seem as though something isn't working quite right on the J1900 platform. Can you post speedtest results at 300mbps to compare? Is the same speedtest.net server being used for the 300mbps result and the 75mbps result?

I have a 1gb synchronous connection, and OPNsense has no trouble pushing this 100%. I'm using a Core i3 4130 and 8GB of RAM, and a quad port Intel i340 server NIC. You should easily be able to push 300mbps with your setup.

One more thing to check would be any traffic shaping enabled? It seems odd that the speed is dropping back to the same speed that ATT Uverse had. Did you have a QOS limiter setup on your old service and its still using the old bandwidth values?

[angel2040]

I didn't have any traffic shaping or QoS configured.  But now you got me thinking. I did turn on suricata.  Every rule was on.  It was fine CPU-wise on my 75 Mps.  But I don't think I ever checked its affect on the 300 Mps connection?  Maybe, it wasn't fully up when I was running my very first test.  Crap.  I was so sure packets were getting dropped and that's what I was focusing on.

[opnfwb]

There's too many variables to be sure at this point.

Those J1900 platform boxes seem quite capable, I would be surprised if the CPU was maxed. However, I also don't use suricata and I'm not fully familiar with it. If it isn't multithreaded, I suppose it's possible that a single thread is maxing out and possibly limiting throughput. However, if that was the case, I would expect all of your speedtest results to be the same. The fact that the 1st test is 300mbit, and subsequent tests are not, seems to indicate that there's something else causing the issue. It wouldn't hurt to disable suricata anyway and rerun the tests on a "stock" config and then slowly start adding features to see where the issue beings to happen.

Without setting OPNsense back up and running more tests, we're just guessing. Did you use a tool like WinMTR to test packet loss and latency while the bandwidth tests were running? That's another good indicator of an issue if you're able to generate packet loss with a fully loaded connection. I've seen NIC driver settings (like TCP offloading and CRC offloading) cause issues too. At this point we have too much to cover without being able to simulate more tests.

[angel2040]

It's been a while.  I had an i5 with 16 gigs of ram (ds81) I just now got installed with brand spanking new OPNsense 17.7.7_1-amd64. 

I can confirm it's definitely suricata.  I did about 6 speed tests all giving me desired speed.  I enabled suricata and my connection was cut in half.  Gonna have to do more research on the ruleset.  I really want an inline IDS. 

[xinnan]

you can't compare that i3 to the j1900

The i3 per core performance is stunningly better.  2 cores on that will handle gigabit networks with ease and Suricata.  The j1900 is going to be overwhelmed on a very fast network.  I think replace your mobo and get an i3 with the highest per-core performance available.  They are cheap and fast.