Hi guys
What I have:
- DEC750
- AMD Ryzen Embedded V1500B (4 cores, 8 threads)
- OPNsense 26.7.3_8-amd64, FreeBSD 15.1-RELEASE-p3
- Ubiquiti SFP+ Modul UACC-CM-RJ45-MG (compatible according to https://docs.opnsense.org/hardware/sfp_compatibility.html)
- Intel X550-T1 in computer for testing (used driver installed from Windows 11 or newest driver from Intel 04.09.2025, 4.1.267.0)
- Cat6 cables
What I tuned according to https://docs.opnsense.org/troubleshooting/performance.html
- net.isr.bindthreads = 1
- net.isr.maxthreads = -1
- net.inet.rss.enabled = 1
- net.inet.rss.bits = 2
However, with https://www.speedtest.net/, iperf3 or with 32G filedownload from https://speedtest.init7.net/en.html, I only get 5-5.5gbps ...
I tested from computer to opnsense.
But also from computer through opnsense to speedtest.init7.net.
I tested with
iperf3 -c speedtest.init7.net I tested in WSL but also in Windows 11 with iperf3, same result.
I doubt its on the performance of my internet provider ...
What am I missing that I do not get the 8.5-10gbps?
Regards
power13
See: https://forum.opnsense.org/index.php?topic=42985.0, point 10.
That being said: You probabaly will not see more than 8 GBit/s even when you actually use more than one stream.
I tested with
iperf3 -P 16 -c speedtest.init7.net -R -t 60
and got
[SUM] 0.00-62.36 sec 42.1 GBytes 5.80 Gbits/sec 106677 sender
[SUM] 0.00-60.00 sec 42.1 GBytes 6.03 Gbits/sec receiver
I found someone in a blog got about 7.5gbps https://blog.shade.sh/index.php/archive/2116
During iperf test, only 3-4 CPUs / Threads have load, the other do not have load.
What I think is, I have only 3 queues per interface, so load will not distribute over more CPUs / threads?
sysctl dev.ax.1.channels_info
dev.ax.1.channels_info:
Pre-set maximums:
RX: 2
TX: 2
Other: 0
Combined: 3
Current hardware settings:
RX: 0
TX: 0
Other: 0
Combined: 3
vmstat -i
...
irq82: ax0:dev_irq 730 0
irq84: ax0 46169 29
irq86: ax0:rxq0 3726781 2321
irq87: ax0:rxq1 4808525 2995
irq88: ax0:rxq2 2124617 1323
irq89: ax1:dev_irq 6 0
irq91: ax1 43441 27
irq93: ax1:rxq0 4480174 2790
irq94: ax1:rxq1 3908571 2434
irq95: ax1:rxq2 3606745 2246
...
Can someone confirm their DEC750 has only 3 queues too?
However, these 3 CPUs / threads are about 40-50%, not 100% during measure.
So it might not be the bottleneck.
Several things come to mind.
1) iperf3 (on the router) is not a tool for testing bandwidth of a router. We dont run apps on the fw, we use the device as a fw/router.
2) run two instances of iperf3 using --affinity switch
3) scheduling of threads can be an issue, the kernel wants to move threads around cpu's (scheduler attempt to balance). As-is, manipulating the OS so that the cpu's wanted for just iperf3 have only iperf3 threads on those cpu's is not easy doing. Threads that move take time because the cache on new cpu needs to fetch from memory. One trick here is, if you want 8 threads and eliminate the cache timing issue, run 8 instances of iperf3 where each is bound to a unique cpu. You can also tune the Time Quantum settings for the default ULE scheduler.
I bet you can get above 8G in testing, but this will not be indicative of actual fw/router bandwidth. The next step I would do, test the device as just a router (tuned iperf3 server and client on each side of the router, dec in middle).
Run the same test as you did, note the results, then just run "sysctl net.inet.tcp.sendbuf_max=16777216" then re-run the testing again, any difference?
All that said, there's tcp RTT issues not accounted for, which impacts measured bandwidth.
https://www.pinglabz.com/how-to-use-iperf3/
https://www.pinglabz.com/iperf3-parallel-streams-tcp-window/
Quote from: BrandyWine on August 30, 2026, 05:57:40 PM[...]1) iperf3 (on the router) is not a tool for testing bandwidth of a router. We dont run apps on the fw, we use the device as a fw/router.
[...]
Run the same test as you did, note the results, then just run "sysctl net.inet.tcp.sendbuf_max=16777216" then re-run the testing again, any difference?[...]
Good point. Folks can tune the heck out of OPNsense as a session terminator, but it generally serves little purpose (improved performance for benchmarks against the firewall itself, and some services, perhaps?) and any (permanent) tuneables should be checked/maintained across (kernel) upgrades to avoid obsolete/potentially detrimental cruft.