10G but reaching only 5-5.5gbps

[power13]

Hi guys

What I have:

• DEC750
• AMD Ryzen Embedded V1500B (4 cores, 8 threads) 
• OPNsense 26.7.3_8-amd64, FreeBSD 15.1-RELEASE-p3
• Ubiquiti SFP+ Modul UACC-CM-RJ45-MG (compatible according to https://docs.opnsense.org/hardware/sfp_compatibility.html)
• Intel X550-T1 in computer for testing (used driver installed from Windows 11 or newest driver from Intel 04.09.2025, 4.1.267.0)
• Cat6 cables

What I tuned according to https://docs.opnsense.org/troubleshooting/performance.html

• net.isr.bindthreads = 1
• net.isr.maxthreads = -1
• net.inet.rss.enabled = 1
• net.inet.rss.bits = 2

However, with https://www.speedtest.net/, iperf3 or with 32G filedownload from https://speedtest.init7.net/en.html, I only get 5-5.5gbps ...

I tested from computer to opnsense.
But also from computer through opnsense to speedtest.init7.net.

I tested with

Code Select Expand

iperf3 -c speedtest.init7.net
I tested in WSL but also in Windows 11 with iperf3, same result.

I doubt its on the performance of my internet provider ...

What am I missing that I do not get the 8.5-10gbps?

Regards
power13

[meyergru]

See: https://forum.opnsense.org/index.php?topic=42985.0, point 10.

That being said: You probabaly will not see more than 8 GBit/s even when you actually use more than one stream.

[power13]

I tested with

Code Select Expand

iperf3 -P 16 -c speedtest.init7.net -R -t 60
and got

Code Select Expand

[SUM]   0.00-62.36  sec  42.1 GBytes  5.80 Gbits/sec  106677             sender
[SUM]   0.00-60.00  sec  42.1 GBytes  6.03 Gbits/sec                  receiver
I found someone in a blog got about 7.5gbps https://blog.shade.sh/index.php/archive/2116

During iperf test, only 3-4 CPUs / Threads have load, the other do not have load.

What I think is, I have only 3 queues per interface, so load will not distribute over more CPUs / threads?

Code Select Expand

sysctl dev.ax.1.channels_info
dev.ax.1.channels_info:
Pre-set maximums:
RX:             2
TX:             2
Other:          0
Combined:       3

Current hardware settings:
RX:             0
TX:             0
Other:          0
Combined:       3

Code Select Expand

vmstat -i
...
irq82: ax0:dev_irq                   730          0
irq84: ax0                         46169         29
irq86: ax0:rxq0                  3726781       2321
irq87: ax0:rxq1                  4808525       2995
irq88: ax0:rxq2                  2124617       1323
irq89: ax1:dev_irq                     6          0
irq91: ax1                         43441         27
irq93: ax1:rxq0                  4480174       2790
irq94: ax1:rxq1                  3908571       2434
irq95: ax1:rxq2                  3606745       2246
...
Can someone confirm their DEC750 has only 3 queues too?

However, these 3 CPUs / threads are about 40-50%, not 100% during measure.
So it might not be the bottleneck.

[BrandyWine]

Several things come to mind.
1) iperf3 (on the router) is not a tool for testing bandwidth of a router. We dont run apps on the fw, we use the device as a fw/router.
2) run two instances of iperf3 using --affinity switch
3) scheduling of threads can be an issue, the kernel wants to move threads around cpu's (scheduler attempt to balance). As-is, manipulating the OS so that the cpu's wanted for just iperf3 have only iperf3 threads on those cpu's is not easy doing. Threads that move take time because the cache on new cpu needs to fetch from memory. One trick here is, if you want 8 threads and eliminate the cache timing issue, run 8 instances of iperf3 where each is bound to a unique cpu. You can also tune the Time Quantum settings for the default ULE scheduler.

I bet you can get above 8G in testing, but this will not be indicative of actual fw/router bandwidth. The next step I would do, test the device as just a  router (tuned iperf3 server and client on each side of the router, dec in middle).

Run the same test as you did, note the results, then just run "sysctl net.inet.tcp.sendbuf_max=16777216" then re-run the testing again, any difference?

All that said, there's tcp RTT issues not accounted for, which impacts measured bandwidth.
https://www.pinglabz.com/how-to-use-iperf3/
https://www.pinglabz.com/iperf3-parallel-streams-tcp-window/

[pfry]

[...]1) iperf3 (on the router) is not a tool for testing bandwidth of a router. We dont run apps on the fw, we use the device as a fw/router.
[...]
Run the same test as you did, note the results, then just run "sysctl net.inet.tcp.sendbuf_max=16777216" then re-run the testing again, any difference?[...]

Good point. Folks can tune the heck out of OPNsense as a session terminator, but it generally serves little purpose (improved performance for benchmarks against the firewall itself, and some services, perhaps?) and any (permanent) tuneables should be checked/maintained across (kernel) upgrades to avoid obsolete/potentially detrimental cruft.