I added a 10Gb NIC to my OPNsense box and its negotiating the 10Gb link just fine, however when I run an iperf3 test from another client to the lan nic I only achieve 1Gb speeds, (931 Mbits/sec). I was expecting to at least hit 6000 Mbits/sec, not 931. I have IPS disabled for this test, I am not using Jumbo Frames. What am I missing here, even without Jumbo Frames I should be getting at least double or 5x what I am seeing. Details and pics below, open to trying anything. Even weirder is that I have 2Gb internet and my windows client hits 2Gb downloads on a speedtest. How could the WAN connected at 2.5Gb be faster than an internal speed test using iperf on the 10Gb lan nic.
Current OPNsense box Specs:
Intel D-1541 XEON 8 Core + Hyperthreading
32GB Ram
128GB M.2 SSD
Intel X550-T2 PCIe 4X 10Gb NIC
Client:
Windows 11
11th Gen i9-11900k
32GB ram
Intel X550-T2
Switch:
Brocade ICX-7250 (8x SFP+ 10Gb ports)
Try iperf with "-P8". Speedtest uses multiple streams per default. And look up this (https://docs.opnsense.org/troubleshooting/performance.html), if you did not already.
Unless I am reading the output wrong, it seems like adding the switch iperf switch -P8 did not improve the performance, output below, seems like im still capped at 1Gb.
I will try some of the tweaks from that article and report back.
Connecting to host 10.0.0.1, port 45277
[ 5] local 10.0.0.240 port 61183 connected to 10.0.0.1 port 45277
[ 7] local 10.0.0.240 port 61184 connected to 10.0.0.1 port 45277
[ 9] local 10.0.0.240 port 61185 connected to 10.0.0.1 port 45277
[ 11] local 10.0.0.240 port 61186 connected to 10.0.0.1 port 45277
[ 13] local 10.0.0.240 port 61187 connected to 10.0.0.1 port 45277
[ 15] local 10.0.0.240 port 61188 connected to 10.0.0.1 port 45277
[ 17] local 10.0.0.240 port 61189 connected to 10.0.0.1 port 45277
[ 19] local 10.0.0.240 port 61190 connected to 10.0.0.1 port 45277
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 5.50 MBytes 46.0 Mbits/sec
[ 7] 0.00-1.00 sec 7.00 MBytes 58.5 Mbits/sec
[ 9] 0.00-1.00 sec 13.0 MBytes 109 Mbits/sec
[ 11] 0.00-1.00 sec 12.9 MBytes 108 Mbits/sec
[ 13] 0.00-1.00 sec 19.9 MBytes 166 Mbits/sec
[ 15] 0.00-1.00 sec 24.9 MBytes 208 Mbits/sec
[ 17] 0.00-1.00 sec 16.0 MBytes 134 Mbits/sec
[ 19] 0.00-1.00 sec 11.6 MBytes 97.2 Mbits/sec
[SUM] 0.00-1.00 sec 111 MBytes 926 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 11.8 MBytes 98.9 Mbits/sec
[ 7] 1.00-2.00 sec 14.0 MBytes 118 Mbits/sec
[ 9] 1.00-2.00 sec 12.0 MBytes 101 Mbits/sec
[ 11] 1.00-2.00 sec 20.0 MBytes 168 Mbits/sec
[ 13] 1.00-2.00 sec 16.1 MBytes 136 Mbits/sec
[ 15] 1.00-2.00 sec 22.4 MBytes 188 Mbits/sec
[ 17] 1.00-2.00 sec 14.1 MBytes 119 Mbits/sec
[ 19] 1.00-2.00 sec 17.9 MBytes 150 Mbits/sec
[SUM] 1.00-2.00 sec 128 MBytes 1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.01 sec 16.0 MBytes 133 Mbits/sec
[ 7] 2.00-3.01 sec 22.4 MBytes 186 Mbits/sec
[ 9] 2.00-3.01 sec 13.4 MBytes 111 Mbits/sec
[ 11] 2.00-3.01 sec 21.2 MBytes 177 Mbits/sec
[ 13] 2.00-3.01 sec 11.1 MBytes 92.4 Mbits/sec
[ 15] 2.00-3.01 sec 11.0 MBytes 91.4 Mbits/sec
[ 17] 2.00-3.01 sec 10.1 MBytes 84.1 Mbits/sec
[ 19] 2.00-3.01 sec 25.2 MBytes 210 Mbits/sec
[SUM] 2.00-3.01 sec 130 MBytes 1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.01-4.00 sec 23.5 MBytes 198 Mbits/sec
[ 7] 3.01-4.00 sec 16.2 MBytes 137 Mbits/sec
[ 9] 3.01-4.00 sec 12.5 MBytes 105 Mbits/sec
[ 11] 3.01-4.00 sec 16.5 MBytes 139 Mbits/sec
[ 13] 3.01-4.00 sec 19.4 MBytes 163 Mbits/sec
[ 15] 3.01-4.00 sec 8.62 MBytes 72.8 Mbits/sec
[ 17] 3.01-4.00 sec 18.9 MBytes 159 Mbits/sec
[ 19] 3.01-4.00 sec 13.0 MBytes 110 Mbits/sec
[SUM] 3.01-4.00 sec 129 MBytes 1.09 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 20.4 MBytes 171 Mbits/sec
[ 7] 4.00-5.00 sec 11.9 MBytes 99.6 Mbits/sec
[ 9] 4.00-5.00 sec 11.6 MBytes 97.5 Mbits/sec
[ 11] 4.00-5.00 sec 14.8 MBytes 124 Mbits/sec
[ 13] 4.00-5.00 sec 25.6 MBytes 215 Mbits/sec
[ 15] 4.00-5.00 sec 12.8 MBytes 107 Mbits/sec
[ 17] 4.00-5.00 sec 21.1 MBytes 177 Mbits/sec
[ 19] 4.00-5.00 sec 9.88 MBytes 82.8 Mbits/sec
[SUM] 4.00-5.00 sec 128 MBytes 1.07 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.01 sec 19.6 MBytes 164 Mbits/sec
[ 7] 5.00-6.01 sec 13.1 MBytes 109 Mbits/sec
[ 9] 5.00-6.01 sec 14.9 MBytes 124 Mbits/sec
[ 11] 5.00-6.01 sec 12.2 MBytes 102 Mbits/sec
[ 13] 5.00-6.01 sec 13.1 MBytes 109 Mbits/sec
[ 15] 5.00-6.01 sec 14.9 MBytes 124 Mbits/sec
[ 17] 5.00-6.01 sec 26.4 MBytes 220 Mbits/sec
[ 19] 5.00-6.01 sec 15.5 MBytes 129 Mbits/sec
[SUM] 5.00-6.01 sec 130 MBytes 1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.01-7.01 sec 26.1 MBytes 219 Mbits/sec
[ 7] 6.01-7.01 sec 13.5 MBytes 113 Mbits/sec
[ 9] 6.01-7.01 sec 15.8 MBytes 132 Mbits/sec
[ 11] 6.01-7.01 sec 11.0 MBytes 92.2 Mbits/sec
[ 13] 6.01-7.01 sec 13.9 MBytes 116 Mbits/sec
[ 15] 6.01-7.01 sec 11.0 MBytes 92.2 Mbits/sec
[ 17] 6.01-7.01 sec 19.1 MBytes 160 Mbits/sec
[ 19] 6.01-7.01 sec 18.9 MBytes 158 Mbits/sec
[SUM] 6.01-7.01 sec 129 MBytes 1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.01-8.01 sec 11.8 MBytes 98.3 Mbits/sec
[ 7] 7.01-8.01 sec 20.9 MBytes 175 Mbits/sec
[ 9] 7.01-8.01 sec 15.8 MBytes 132 Mbits/sec
[ 11] 7.01-8.01 sec 17.9 MBytes 150 Mbits/sec
[ 13] 7.01-8.01 sec 18.0 MBytes 151 Mbits/sec
[ 15] 7.01-8.01 sec 13.1 MBytes 110 Mbits/sec
[ 17] 7.01-8.01 sec 17.6 MBytes 148 Mbits/sec
[ 19] 7.01-8.01 sec 14.0 MBytes 117 Mbits/sec
[SUM] 7.01-8.01 sec 129 MBytes 1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.01-9.01 sec 8.00 MBytes 67.1 Mbits/sec
[ 7] 8.01-9.01 sec 32.1 MBytes 269 Mbits/sec
[ 9] 8.01-9.01 sec 12.8 MBytes 107 Mbits/sec
[ 11] 8.01-9.01 sec 9.88 MBytes 82.8 Mbits/sec
[ 13] 8.01-9.01 sec 22.5 MBytes 189 Mbits/sec
[ 15] 8.01-9.01 sec 19.0 MBytes 159 Mbits/sec
[ 17] 8.01-9.01 sec 10.5 MBytes 88.0 Mbits/sec
[ 19] 8.01-9.01 sec 14.1 MBytes 118 Mbits/sec
[SUM] 8.01-9.01 sec 129 MBytes 1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.01-10.01 sec 18.0 MBytes 152 Mbits/sec
[ 7] 9.01-10.01 sec 17.8 MBytes 150 Mbits/sec
[ 9] 9.01-10.01 sec 13.4 MBytes 113 Mbits/sec
[ 11] 9.01-10.01 sec 8.25 MBytes 69.5 Mbits/sec
[ 13] 9.01-10.01 sec 25.4 MBytes 214 Mbits/sec
[ 15] 9.01-10.01 sec 15.2 MBytes 129 Mbits/sec
[ 17] 9.01-10.01 sec 12.8 MBytes 107 Mbits/sec
[ 19] 9.01-10.01 sec 17.4 MBytes 146 Mbits/sec
[SUM] 9.01-10.01 sec 128 MBytes 1.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 161 MBytes 135 Mbits/sec sender
[ 5] 0.00-10.02 sec 160 MBytes 134 Mbits/sec receiver
[ 7] 0.00-10.01 sec 169 MBytes 142 Mbits/sec sender
[ 7] 0.00-10.02 sec 168 MBytes 141 Mbits/sec receiver
[ 9] 0.00-10.01 sec 135 MBytes 113 Mbits/sec sender
[ 9] 0.00-10.02 sec 135 MBytes 113 Mbits/sec receiver
[ 11] 0.00-10.01 sec 145 MBytes 121 Mbits/sec sender
[ 11] 0.00-10.02 sec 144 MBytes 121 Mbits/sec receiver
[ 13] 0.00-10.01 sec 185 MBytes 155 Mbits/sec sender
[ 13] 0.00-10.02 sec 185 MBytes 155 Mbits/sec receiver
[ 15] 0.00-10.01 sec 153 MBytes 128 Mbits/sec sender
[ 15] 0.00-10.02 sec 152 MBytes 128 Mbits/sec receiver
[ 17] 0.00-10.01 sec 167 MBytes 140 Mbits/sec sender
[ 17] 0.00-10.02 sec 166 MBytes 139 Mbits/sec receiver
[ 19] 0.00-10.01 sec 158 MBytes 132 Mbits/sec sender
[ 19] 0.00-10.02 sec 157 MBytes 131 Mbits/sec receiver
[SUM] 0.00-10.01 sec 1.24 GBytes 1.07 Gbits/sec sender
[SUM] 0.00-10.02 sec 1.24 GBytes 1.06 Gbits/sec receiver
iperf Done.
The reported speeds are so exactly what would be expected from a 1 Gbps connection - are you really sure that there is no 1 Gbps limit anywhere between client and server?
You can also test the reverse direction with "-R", BTW.
Earlier I started messing around with the hardware offloading settings, which made no difference. However, after reverting back to a full stock config and rebooting my iperf scores are looking better in threaded mode. The numbers below should prove all nics are at 10Gb. However I still feel like the scores are low. The netgate hardware store sells a box running a D-1541 Xeon cpu and its rated for 18Gb, im sure I can do better than the below. I'm gonna have to try some more tweaks, its just hard to determine which are really making an impact.
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 535 MBytes 449 Mbits/sec sender
[ 5] 0.00-10.01 sec 535 MBytes 449 Mbits/sec receiver
[ 7] 0.00-10.01 sec 550 MBytes 461 Mbits/sec sender
[ 7] 0.00-10.01 sec 549 MBytes 461 Mbits/sec receiver
[ 9] 0.00-10.01 sec 529 MBytes 443 Mbits/sec sender
[ 9] 0.00-10.01 sec 529 MBytes 443 Mbits/sec receiver
[ 11] 0.00-10.01 sec 562 MBytes 471 Mbits/sec sender
[ 11] 0.00-10.01 sec 562 MBytes 471 Mbits/sec receiver
[ 13] 0.00-10.01 sec 526 MBytes 441 Mbits/sec sender
[ 13] 0.00-10.01 sec 526 MBytes 441 Mbits/sec receiver
[ 15] 0.00-10.01 sec 552 MBytes 463 Mbits/sec sender
[ 15] 0.00-10.01 sec 552 MBytes 463 Mbits/sec receiver
[ 17] 0.00-10.01 sec 534 MBytes 448 Mbits/sec sender
[ 17] 0.00-10.01 sec 534 MBytes 448 Mbits/sec receiver
[ 19] 0.00-10.01 sec 558 MBytes 468 Mbits/sec sender
[ 19] 0.00-10.01 sec 558 MBytes 468 Mbits/sec receiver
[SUM] 0.00-10.01 sec 4.24 GBytes 3.64 Gbits/sec sender
[SUM] 0.00-10.01 sec 4.24 GBytes 3.64 Gbits/sec receiver
Well, the D-1541 is not the fastest single-threading CPU in the world, so you will probably need multiple streams in order to squeeze out more. Also, those 18 Gbps advertised routing speed may be under optimal circumstances (and probably not under FreeBSD).
Quote from: meyergru on January 23, 2025, 04:04:16 PMWell, the D-1541 is not the fastest single-threading CPU in the world, so you will probably need multiple streams in order to squeeze out more. Also, those 18 Gbps advertised routing speed may be under optimal circumstances (and probably not under FreeBSD).
That sounds like a spec from the Netgate 1541, which was advertised to offer 19Gb/s of routing/firewall. It would be interesting to test through it as opposed to to it. (I can't criticize, as I'm not set up to test properly yet either. Eventually...)
I think the DEC740 was quoted at something like that, too, and I never got anything like it under real-life conditions.
I'm averaging around 5Gb now, I tried each tweak below, I'll leave them in place for now, I feel like adding RSS helped reduce my latency according to some other speed tests. I am still only averaging around 10-12ms latency on bufferbloat tests, before I upgraded everything I was a consistent 2-4ms scoring A+, now with 2Gb service and 10Gb nics various tweaks etc I cant seem to get below 10ms of latency. That said though my clients on 1Gb nics are scoring A+, it seems like my client running at 10Gb are having the issue scoring A+, if that makes any sense ?
iperf benchmarks -P8 (8 Threads)
Stock Tunables:
[SUM] 0.00-10.01 sec 5.26 GBytes 4.52 Gbits/sec receiver
Added RSS Enabled:
[SUM] 0.00-10.02 sec 5.74 GBytes 4.92 Gbits/sec receiver
Added kern.ipc.maxsockbuf 16777216
[SUM] 0.00-10.01 sec 5.56 GBytes 4.77 Gbits/sec receiver
Added net.inet.tcp.soreceive_stream 1
[SUM] 0.00-10.01 sec 5.71 GBytes 4.90 Gbits/sec receiver
when shopping for a new router OS moving from clearOS. i ended up here with opnsense. opnsense was not tuned very well for 10GBe traffic. on my lan side. knowing that my system has more than enough power to handle it. hp z840 with 2x 2690v4 cpus x 128gb of mem. 4x intel x710-da2 !)GBe & cards 1 Silicom PE2G6I35-CX 6-Port 1GBe. i already knew it would max out on clearOS.
so i followed the setup here (https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/) for multi-gigabit internet. this managed to get my full speed across my network