10Gbps performance issues iperf benchmarking at 1Gbps

Started by ngr2001, January 22, 2025, 02:26:01 AM

Previous topic - Next topic
Print
Go Down Pages1
User actions
January 22, 2025, 02:26:01 AM
I added a 10Gb NIC to my OPNsense box and its negotiating the 10Gb link just fine, however when I run an iperf3 test from another client to the lan nic I only achieve 1Gb speeds, (931 Mbits/sec). I was expecting to at least hit 6000 Mbits/sec, not 931. I have IPS disabled for this test, I am not using Jumbo Frames. What am I missing here, even without Jumbo Frames I should be getting at least double or 5x what I am seeing. Details and pics below, open to trying anything. Even weirder is that I have 2Gb internet and my windows client hits 2Gb downloads on a speedtest. How could the WAN connected at 2.5Gb be faster than an internal speed test using iperf on the 10Gb lan nic.


Current OPNsense box Specs:

Intel D-1541 XEON 8 Core + Hyperthreading
32GB Ram
128GB M.2 SSD
Intel X550-T2 PCIe 4X 10Gb NIC

Client:
Windows 11
11th Gen i9-11900k
32GB ram
Intel X550-T2

Switch:
Brocade ICX-7250 (8x SFP+ 10Gb ports)
January 22, 2025, 09:52:23 AM #1 Last Edit: January 22, 2025, 09:55:45 AM by meyergru
Try iperf with "-P8". Speedtest uses multiple streams per default. And look up this, if you did not already.
Intel N100, 4 x I226-V, 16 GByte, 256 GByte NVME, ZTE F6005

1100 down / 440 up, Bufferbloat A+
January 23, 2025, 12:37:20 AM #2
Unless I am reading the output wrong, it seems like adding the switch iperf switch -P8 did not improve the performance, output below, seems like im still capped at 1Gb.

I will try some of the tweaks from that article and report back.



Connecting to host 10.0.0.1, port 45277
[  5] local 10.0.0.240 port 61183 connected to 10.0.0.1 port 45277
[  7] local 10.0.0.240 port 61184 connected to 10.0.0.1 port 45277
[  9] local 10.0.0.240 port 61185 connected to 10.0.0.1 port 45277
[ 11] local 10.0.0.240 port 61186 connected to 10.0.0.1 port 45277
[ 13] local 10.0.0.240 port 61187 connected to 10.0.0.1 port 45277
[ 15] local 10.0.0.240 port 61188 connected to 10.0.0.1 port 45277
[ 17] local 10.0.0.240 port 61189 connected to 10.0.0.1 port 45277
[ 19] local 10.0.0.240 port 61190 connected to 10.0.0.1 port 45277
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.50 MBytes  46.0 Mbits/sec
[  7]   0.00-1.00   sec  7.00 MBytes  58.5 Mbits/sec
[  9]   0.00-1.00   sec  13.0 MBytes   109 Mbits/sec
[ 11]   0.00-1.00   sec  12.9 MBytes   108 Mbits/sec
[ 13]   0.00-1.00   sec  19.9 MBytes   166 Mbits/sec
[ 15]   0.00-1.00   sec  24.9 MBytes   208 Mbits/sec
[ 17]   0.00-1.00   sec  16.0 MBytes   134 Mbits/sec
[ 19]   0.00-1.00   sec  11.6 MBytes  97.2 Mbits/sec
[SUM]   0.00-1.00   sec   111 MBytes   926 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  11.8 MBytes  98.9 Mbits/sec
[  7]   1.00-2.00   sec  14.0 MBytes   118 Mbits/sec
[  9]   1.00-2.00   sec  12.0 MBytes   101 Mbits/sec
[ 11]   1.00-2.00   sec  20.0 MBytes   168 Mbits/sec
[ 13]   1.00-2.00   sec  16.1 MBytes   136 Mbits/sec
[ 15]   1.00-2.00   sec  22.4 MBytes   188 Mbits/sec
[ 17]   1.00-2.00   sec  14.1 MBytes   119 Mbits/sec
[ 19]   1.00-2.00   sec  17.9 MBytes   150 Mbits/sec
[SUM]   1.00-2.00   sec   128 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.01   sec  16.0 MBytes   133 Mbits/sec
[  7]   2.00-3.01   sec  22.4 MBytes   186 Mbits/sec
[  9]   2.00-3.01   sec  13.4 MBytes   111 Mbits/sec
[ 11]   2.00-3.01   sec  21.2 MBytes   177 Mbits/sec
[ 13]   2.00-3.01   sec  11.1 MBytes  92.4 Mbits/sec
[ 15]   2.00-3.01   sec  11.0 MBytes  91.4 Mbits/sec
[ 17]   2.00-3.01   sec  10.1 MBytes  84.1 Mbits/sec
[ 19]   2.00-3.01   sec  25.2 MBytes   210 Mbits/sec
[SUM]   2.00-3.01   sec   130 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.01-4.00   sec  23.5 MBytes   198 Mbits/sec
[  7]   3.01-4.00   sec  16.2 MBytes   137 Mbits/sec
[  9]   3.01-4.00   sec  12.5 MBytes   105 Mbits/sec
[ 11]   3.01-4.00   sec  16.5 MBytes   139 Mbits/sec
[ 13]   3.01-4.00   sec  19.4 MBytes   163 Mbits/sec
[ 15]   3.01-4.00   sec  8.62 MBytes  72.8 Mbits/sec
[ 17]   3.01-4.00   sec  18.9 MBytes   159 Mbits/sec
[ 19]   3.01-4.00   sec  13.0 MBytes   110 Mbits/sec
[SUM]   3.01-4.00   sec   129 MBytes  1.09 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  20.4 MBytes   171 Mbits/sec
[  7]   4.00-5.00   sec  11.9 MBytes  99.6 Mbits/sec
[  9]   4.00-5.00   sec  11.6 MBytes  97.5 Mbits/sec
[ 11]   4.00-5.00   sec  14.8 MBytes   124 Mbits/sec
[ 13]   4.00-5.00   sec  25.6 MBytes   215 Mbits/sec
[ 15]   4.00-5.00   sec  12.8 MBytes   107 Mbits/sec
[ 17]   4.00-5.00   sec  21.1 MBytes   177 Mbits/sec
[ 19]   4.00-5.00   sec  9.88 MBytes  82.8 Mbits/sec
[SUM]   4.00-5.00   sec   128 MBytes  1.07 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.01   sec  19.6 MBytes   164 Mbits/sec
[  7]   5.00-6.01   sec  13.1 MBytes   109 Mbits/sec
[  9]   5.00-6.01   sec  14.9 MBytes   124 Mbits/sec
[ 11]   5.00-6.01   sec  12.2 MBytes   102 Mbits/sec
[ 13]   5.00-6.01   sec  13.1 MBytes   109 Mbits/sec
[ 15]   5.00-6.01   sec  14.9 MBytes   124 Mbits/sec
[ 17]   5.00-6.01   sec  26.4 MBytes   220 Mbits/sec
[ 19]   5.00-6.01   sec  15.5 MBytes   129 Mbits/sec
[SUM]   5.00-6.01   sec   130 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.01-7.01   sec  26.1 MBytes   219 Mbits/sec
[  7]   6.01-7.01   sec  13.5 MBytes   113 Mbits/sec
[  9]   6.01-7.01   sec  15.8 MBytes   132 Mbits/sec
[ 11]   6.01-7.01   sec  11.0 MBytes  92.2 Mbits/sec
[ 13]   6.01-7.01   sec  13.9 MBytes   116 Mbits/sec
[ 15]   6.01-7.01   sec  11.0 MBytes  92.2 Mbits/sec
[ 17]   6.01-7.01   sec  19.1 MBytes   160 Mbits/sec
[ 19]   6.01-7.01   sec  18.9 MBytes   158 Mbits/sec
[SUM]   6.01-7.01   sec   129 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.01-8.01   sec  11.8 MBytes  98.3 Mbits/sec
[  7]   7.01-8.01   sec  20.9 MBytes   175 Mbits/sec
[  9]   7.01-8.01   sec  15.8 MBytes   132 Mbits/sec
[ 11]   7.01-8.01   sec  17.9 MBytes   150 Mbits/sec
[ 13]   7.01-8.01   sec  18.0 MBytes   151 Mbits/sec
[ 15]   7.01-8.01   sec  13.1 MBytes   110 Mbits/sec
[ 17]   7.01-8.01   sec  17.6 MBytes   148 Mbits/sec
[ 19]   7.01-8.01   sec  14.0 MBytes   117 Mbits/sec
[SUM]   7.01-8.01   sec   129 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.01-9.01   sec  8.00 MBytes  67.1 Mbits/sec
[  7]   8.01-9.01   sec  32.1 MBytes   269 Mbits/sec
[  9]   8.01-9.01   sec  12.8 MBytes   107 Mbits/sec
[ 11]   8.01-9.01   sec  9.88 MBytes  82.8 Mbits/sec
[ 13]   8.01-9.01   sec  22.5 MBytes   189 Mbits/sec
[ 15]   8.01-9.01   sec  19.0 MBytes   159 Mbits/sec
[ 17]   8.01-9.01   sec  10.5 MBytes  88.0 Mbits/sec
[ 19]   8.01-9.01   sec  14.1 MBytes   118 Mbits/sec
[SUM]   8.01-9.01   sec   129 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.01-10.01  sec  18.0 MBytes   152 Mbits/sec
[  7]   9.01-10.01  sec  17.8 MBytes   150 Mbits/sec
[  9]   9.01-10.01  sec  13.4 MBytes   113 Mbits/sec
[ 11]   9.01-10.01  sec  8.25 MBytes  69.5 Mbits/sec
[ 13]   9.01-10.01  sec  25.4 MBytes   214 Mbits/sec
[ 15]   9.01-10.01  sec  15.2 MBytes   129 Mbits/sec
[ 17]   9.01-10.01  sec  12.8 MBytes   107 Mbits/sec
[ 19]   9.01-10.01  sec  17.4 MBytes   146 Mbits/sec
[SUM]   9.01-10.01  sec   128 MBytes  1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   161 MBytes   135 Mbits/sec                  sender
[  5]   0.00-10.02  sec   160 MBytes   134 Mbits/sec                  receiver
[  7]   0.00-10.01  sec   169 MBytes   142 Mbits/sec                  sender
[  7]   0.00-10.02  sec   168 MBytes   141 Mbits/sec                  receiver
[  9]   0.00-10.01  sec   135 MBytes   113 Mbits/sec                  sender
[  9]   0.00-10.02  sec   135 MBytes   113 Mbits/sec                  receiver
[ 11]   0.00-10.01  sec   145 MBytes   121 Mbits/sec                  sender
[ 11]   0.00-10.02  sec   144 MBytes   121 Mbits/sec                  receiver
[ 13]   0.00-10.01  sec   185 MBytes   155 Mbits/sec                  sender
[ 13]   0.00-10.02  sec   185 MBytes   155 Mbits/sec                  receiver
[ 15]   0.00-10.01  sec   153 MBytes   128 Mbits/sec                  sender
[ 15]   0.00-10.02  sec   152 MBytes   128 Mbits/sec                  receiver
[ 17]   0.00-10.01  sec   167 MBytes   140 Mbits/sec                  sender
[ 17]   0.00-10.02  sec   166 MBytes   139 Mbits/sec                  receiver
[ 19]   0.00-10.01  sec   158 MBytes   132 Mbits/sec                  sender
[ 19]   0.00-10.02  sec   157 MBytes   131 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  1.24 GBytes  1.07 Gbits/sec                  sender
[SUM]   0.00-10.02  sec  1.24 GBytes  1.06 Gbits/sec                  receiver

iperf Done.

January 23, 2025, 12:42:50 AM #3
The reported speeds are so exactly what would be expected from a 1 Gbps connection - are you really sure that there is no 1 Gbps limit anywhere between client and server?

You can also test the reverse direction with "-R", BTW.
Intel N100, 4 x I226-V, 16 GByte, 256 GByte NVME, ZTE F6005

1100 down / 440 up, Bufferbloat A+
January 23, 2025, 02:42:04 PM #4
Earlier I started messing around with the hardware offloading settings, which made no difference. However, after reverting back to a full stock config and rebooting my iperf scores are looking better in threaded mode. The numbers below should prove all nics are at 10Gb. However I still feel like the scores are low. The netgate hardware store sells a box running a D-1541 Xeon cpu and its rated for 18Gb, im sure I can do better than the below. I'm gonna have to try some more tweaks, its just hard to determine which are really making an impact.

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   535 MBytes   449 Mbits/sec                  sender
[  5]   0.00-10.01  sec   535 MBytes   449 Mbits/sec                  receiver
[  7]   0.00-10.01  sec   550 MBytes   461 Mbits/sec                  sender
[  7]   0.00-10.01  sec   549 MBytes   461 Mbits/sec                  receiver
[  9]   0.00-10.01  sec   529 MBytes   443 Mbits/sec                  sender
[  9]   0.00-10.01  sec   529 MBytes   443 Mbits/sec                  receiver
[ 11]   0.00-10.01  sec   562 MBytes   471 Mbits/sec                  sender
[ 11]   0.00-10.01  sec   562 MBytes   471 Mbits/sec                  receiver
[ 13]   0.00-10.01  sec   526 MBytes   441 Mbits/sec                  sender
[ 13]   0.00-10.01  sec   526 MBytes   441 Mbits/sec                  receiver
[ 15]   0.00-10.01  sec   552 MBytes   463 Mbits/sec                  sender
[ 15]   0.00-10.01  sec   552 MBytes   463 Mbits/sec                  receiver
[ 17]   0.00-10.01  sec   534 MBytes   448 Mbits/sec                  sender
[ 17]   0.00-10.01  sec   534 MBytes   448 Mbits/sec                  receiver
[ 19]   0.00-10.01  sec   558 MBytes   468 Mbits/sec                  sender
[ 19]   0.00-10.01  sec   558 MBytes   468 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  4.24 GBytes  3.64 Gbits/sec                  sender
[SUM]   0.00-10.01  sec  4.24 GBytes  3.64 Gbits/sec                  receiver

January 23, 2025, 04:04:16 PM #5
Well, the D-1541 is not the fastest single-threading CPU in the world, so you will probably need multiple streams in order to squeeze out more. Also, those 18 Gbps advertised routing speed may be under optimal circumstances (and probably not under FreeBSD).
Intel N100, 4 x I226-V, 16 GByte, 256 GByte NVME, ZTE F6005

1100 down / 440 up, Bufferbloat A+
January 23, 2025, 07:24:26 PM #6
Quote from: meyergru on January 23, 2025, 04:04:16 PMWell, the D-1541 is not the fastest single-threading CPU in the world, so you will probably need multiple streams in order to squeeze out more. Also, those 18 Gbps advertised routing speed may be under optimal circumstances (and probably not under FreeBSD).

That sounds like a spec from the Netgate 1541, which was advertised to offer 19Gb/s of routing/firewall. It would be interesting to test through it as opposed to to it. (I can't criticize, as I'm not set up to test properly yet either. Eventually...)
January 23, 2025, 08:49:24 PM #7
I think the DEC740 was quoted at something like that, too, and I never got anything like it under real-life conditions.
Intel N100, 4 x I226-V, 16 GByte, 256 GByte NVME, ZTE F6005

1100 down / 440 up, Bufferbloat A+
January 24, 2025, 06:54:53 PM #8
I'm averaging around 5Gb now, I tried each tweak below, I'll leave them in place for now, I feel like adding RSS helped reduce my latency according to some other speed tests. I am still only averaging around 10-12ms latency on bufferbloat tests, before I upgraded everything I was a consistent 2-4ms scoring A+, now with 2Gb service and 10Gb nics various tweaks etc I cant seem to get below 10ms of latency. That said though my clients on 1Gb nics are scoring A+, it seems like my client running at 10Gb are having the issue scoring A+, if that makes any sense ?



iperf benchmarks -P8 (8 Threads)

Stock Tunables:
[SUM]   0.00-10.01  sec  5.26 GBytes  4.52 Gbits/sec                  receiver

Added RSS Enabled:
[SUM]   0.00-10.02  sec  5.74 GBytes  4.92 Gbits/sec                  receiver

Added kern.ipc.maxsockbuf 16777216
[SUM]   0.00-10.01  sec  5.56 GBytes  4.77 Gbits/sec                  receiver

Added net.inet.tcp.soreceive_stream 1
[SUM]   0.00-10.01  sec  5.71 GBytes  4.90 Gbits/sec                  receiver
January 27, 2025, 10:46:20 PM #9
when shopping for a new router OS moving from clearOS. i ended up here with opnsense. opnsense was not tuned very well for 10GBe traffic. on my lan side. knowing that my system has more than enough power to handle it. hp z840 with 2x 2690v4 cpus x 128gb of mem. 4x intel x710-da2 !)GBe & cards 1 Silicom PE2G6I35-CX 6-Port 1GBe. i already knew it would max out on clearOS.

so i followed the setup here for multi-gigabit internet. this managed to get my full speed across my network

Print
Go Up Pages1
User actions