Concerning Speedtest Results

[fakebizprez]

Code Select Expand

   Speedtest by Ookla

      Server: Nitel - Chicago, IL (id: 12187)
         ISP: AT&T Internet
Idle Latency:     2.54 ms   (jitter: 0.65ms, low: 2.00ms, high: 3.01ms)
    Download:  5180.79 Mbps (data used: 8.8 GB)
                  3.41 ms   (jitter: 0.63ms, low: 2.02ms, high: 6.15ms)
      Upload:  1870.71 Mbps (data used: 2.2 GB)
                  2.16 ms   (jitter: 0.33ms, low: 1.77ms, high: 6.51ms)
 Packet Loss:     0.0%

Plan: AT&T Fiber 5Gbps UP/DOWN

Any ideas on where I should begin to troubleshoot this?

[pfry]

Have you tried different servers or sites, e.g. https://fast.com/ or https://speed.cloudflare.com/?

[fakebizprez]

Have you tried different servers or sites, e.g. https://fast.com/ or https://speed.cloudflare.com/?

Hello,

This test was done using the OPNsense Shell - it's the only speedtest I know of that can be performed directly from the firewall. Are there other options?

[JamesFrisch]

PowerD disabled or set max performance?

[meyergru]

The CLI version of speedtest is notorious to auto-select non-optimal servers. Many servers are not optimized for upload speed, even less so at your expected bandwidth. I get less than half of my real upstream bandwidth from many speedtest servers as well. They are often good to up to 500 Mbit/s, but not more.

Try to list servers near you by calling "speedtest -L", then try any of them using "speedtest -s <serverid>".

[pfry]

[...]
This test was done using the OPNsense Shell - it's the only speedtest I know of that can be performed directly from the firewall. Are there other options?

Bad assumption on my part. But likely OPNsense would need some (server-type) optimization when testing against the firewall itself. Aside from the speedtest server choice issue.

I have to say, I'd plug in a workstation for (what I would expect to be... another bad assumption?) a more relevant test.

[fakebizprez]

Thanks, everyone.

I just tested every available server and the results were all about the same 5.2Gbps Down, 2.5Gbps Up.

OPNsense is running bare metal on a powerful PowerEdge R730. I think this is an issue with my ISP.

[fakebizprez]

PowerD disabled or set max performance?

Interesting question...

I am in the process of upgrading my home from a 100 AMP electrical panel to dual 200 AMP electrical panels, so all servers are set to "minimum" power settings, including the PowerEdge R730 that is running OPNsense baremetal.

How big of an impact do you believe that has on speed?


Specs:

Code Select Expand

CPU
Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz (12 cores, 24 threads)


[fakebizprez]

First test is Minimum Power, second test is Maximum Power:

Code Select Expand

      Server: Nitel - Chicago, IL (id: 12187)
         ISP: AT&T Internet
Idle Latency:     2.65 ms   (jitter: 0.38ms, low: 2.11ms, high: 3.25ms)
    Download:  5165.98 Mbps (data used: 8.2 GB)                                                   
                  3.79 ms   (jitter: 0.78ms, low: 2.04ms, high: 8.65ms)
      Upload:  3402.49 Mbps (data used: 5.0 GB)                                                   
                  2.02 ms   (jitter: 0.19ms, low: 1.71ms, high: 3.47ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/ca77d016-b22a-478a-b7c2-94355576cb5a
root@opnsense:~ # speedtest

   Speedtest by Ookla

      Server: Nitel - Chicago, IL (id: 12187)
         ISP: AT&T Internet
Idle Latency:     2.44 ms   (jitter: 0.45ms, low: 2.03ms, high: 2.73ms)
    Download:  4971.78 Mbps (data used: 7.7 GB)                                                   
                  3.62 ms   (jitter: 2.13ms, low: 1.77ms, high: 213.98ms)
      Upload:  1849.12 Mbps (data used: 925.1 MB)                                                   
                  2.17 ms   (jitter: 0.30ms, low: 1.77ms, high: 3.22ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/18bae36b-1870-4bc2-9cc6-8ae4f005a93b

I'm more confused than ever...

[pfry]

[...]
I'm more confused than ever...

I did some testing with powerd, and it made no difference in idle power on my systems (my main concern, as they are idle most of the time). Power usage was a fair bit higher than with Linux until I loaded the (onboard AMD) video driver, oddly enough. (Can't say if this would apply to discrete video, if you have such on your Dell.) This was with FreeBSD - I never checked OPNsense specifically. It would be nice if FreeBSD had an equivalent to "cat /proc/cpuinfo | grep MHz" in Linux.

It could be some sort of ramp rate oddity with the Haswell E. It shouldn't be a turbo issue, as the difference between base and max clock is all of 300Mhz.

I wouldn't call a 6 core Haswell E particularly "powerful" by modern standards in compute, but I'd expect it to be enough for a firewall. It does have lots of PCI-e I/O and a decent ECC memory subsystem (I assume you have four sticks installed). (I can't say about the Dell, but you could upgrade a standard 2011v3 to a flock of 10 or 20 chickens for cheep.)

[JamesFrisch]

How big of an impact do you believe that has on speed?

I don't know about real life impact, I can only tell you what I measured with speedtest.

When having it on hiadaptive, first run would be 6Gbit/s. If I ran a second one immediately afterwards (so the system could not enter any power savings) it was 10GBit/s.

When disabled I got 10GBit/s in the first run.


Also please make sure to reboot your system after setting it. I seem to remember that it would not apply otherwise.

[Greg_E]

Get powerful Windows/Mac/Linux computer, connect directly to the handoff point, do speed tests. That's what the ISP will do if you pay them for a service call. And they will probably test to an internal server that is on the same network as your supply, gives them and easy edge to hit the goals where their backbone might have some congestion. Time Warner was notorious for this, internal server always hit the speeds, external server almost never.

[fakebizprez]

How big of an impact do you believe that has on speed?

I don't know about real life impact, I can only tell you what I measured with speedtest.

When having it on hiadaptive, first run would be 6Gbit/s. If I ran a second one immediately afterwards (so the system could not enter any power savings) it was 10GBit/s.

When disabled I got 10GBit/s in the first run.


Also please make sure to reboot your system after setting it. I seem to remember that it would not apply otherwise.

Interesting. I did not reboot after changing the power settings. I will try this shortly and report back.

[fakebizprez]

Get powerful Windows/Mac/Linux computer, connect directly to the handoff point, do speed tests. That's what the ISP will do if you pay them for a service call. And they will probably test to an internal server that is on the same network as your supply, gives them and easy edge to hit the goals where their backbone might have some congestion. Time Warner was notorious for this, internal server always hit the speeds, external server almost never.

I didn't want to down this rabbit hole explaining, but here goes..

AT&T is notorious for making their fiber users operate a gateway named, the BGW-320 (https://fcc.report/FCC-ID/O6ZBGW320/4522478.pdf), which is essentially a firewall, a router, and a Wi-Fi device rolled into one. It has no "true" pass-through mode, and is essentially a piece of spyware.

After I purchased the 5Gbe fiber I had to call a tech out because I couldn't get upload speeds above 3000Gbe. Long story short, after hours f testing, the technician disclosed that there is an issue affecting the BGW-320, isolated to my neighborhood that is capping upload speeds at 3000Gbe.

I circumvented this BGW-320 gateway by installing a "hacked" SFP module that runs Linux and lets me plug directly into the Dell PowerEdge R730 that has OPNsense installed - baremetal.

Right off the bat, this fixed the issue. Months have gone by, which have included several changes in configuration, upstream updates, etc etc; and here I am.

This is why I'm starting to believe the issue is the ISP and not my device.

[pfry]

[...]
AT&T is notorious [...]

Points for consistency? It could be worse: U-Verse VDSL initially used the 2Wire RG, which had a non-RFC-compliant wannabe "IP" stack, and could not easily be circumvented as it used (IIRC) 802.1x authentication with an embedded certificate. I had DSL with static IPs; AT&T replaced it with U-Verse - ostensibly an upgrade - but the 2Wire could only associate 1 IP to 1 MAC, rendering the service effectively worthless. I ended up getting a TWC cable service.

Right off the bat, this fixed the issue. Months have gone by, which have included several changes in configuration, upstream updates, etc etc; and here I am.

This is why I'm starting to believe the issue is the ISP and not my device.

You started with full 5Gb/s symmetric performance from your described setup, and it took a dive at some point? That's kind of an important data point. Yeah, I'd start to believe it's the link, too. I believe I get it now - you wanted to be thorough, which is wise. I'd try the Humax just to see what it will do now - another data point. But I imagine you'll need to get a tech out again to "prove it out". Good luck.

I assumed you were troubleshooting a new setup that had never performed to spec.