Hi guys
What I have:
- DEC750
- AMD Ryzen Embedded V1500B (4 cores, 8 threads)
- OPNsense 26.7.3_8-amd64, FreeBSD 15.1-RELEASE-p3
- Ubiquiti SFP+ Modul UACC-CM-RJ45-MG (compatible according to https://docs.opnsense.org/hardware/sfp_compatibility.html)
- Intel X550-T1 in computer for testing (used driver installed from Windows 11 or newest driver from Intel 04.09.2025, 4.1.267.0)
- Cat6 cables
What I tuned according to https://docs.opnsense.org/troubleshooting/performance.html
- net.isr.bindthreads = 1
- net.isr.maxthreads = -1
- net.inet.rss.enabled = 1
- net.inet.rss.bits = 2
However, with https://www.speedtest.net/, iperf3 or with 32G filedownload from https://speedtest.init7.net/en.html, I only get 5-5.5gbps ...
I tested from computer to opnsense.
But also from computer through opnsense to speedtest.init7.net.
I tested with
iperf3 -c speedtest.init7.net I tested in WSL but also in Windows 11 with iperf3, same result.
I doubt its on the performance of my internet provider ...
What am I missing that I do not get the 8.5-10gbps?
Regards
power13
See: https://forum.opnsense.org/index.php?topic=42985.0, point 10.
That being said: You probably will not see more than 8 GBit/s even when you actually use more than one stream.
I tested with
iperf3 -P 16 -c speedtest.init7.net -R -t 60
and got
[SUM] 0.00-62.36 sec 42.1 GBytes 5.80 Gbits/sec 106677 sender
[SUM] 0.00-60.00 sec 42.1 GBytes 6.03 Gbits/sec receiver
I found someone in a blog got about 7.5gbps https://blog.shade.sh/index.php/archive/2116
During iperf test, only 3-4 CPUs / Threads have load, the other do not have load.
What I think is, I have only 3 queues per interface, so load will not distribute over more CPUs / threads?
sysctl dev.ax.1.channels_info
dev.ax.1.channels_info:
Pre-set maximums:
RX: 2
TX: 2
Other: 0
Combined: 3
Current hardware settings:
RX: 0
TX: 0
Other: 0
Combined: 3
vmstat -i
...
irq82: ax0:dev_irq 730 0
irq84: ax0 46169 29
irq86: ax0:rxq0 3726781 2321
irq87: ax0:rxq1 4808525 2995
irq88: ax0:rxq2 2124617 1323
irq89: ax1:dev_irq 6 0
irq91: ax1 43441 27
irq93: ax1:rxq0 4480174 2790
irq94: ax1:rxq1 3908571 2434
irq95: ax1:rxq2 3606745 2246
...
Can someone confirm their DEC750 has only 3 queues too?
However, these 3 CPUs / threads are about 40-50%, not 100% during measure.
So it might not be the bottleneck.
Several things come to mind.
1) iperf3 (on the router) is not a tool for testing bandwidth of a router. We dont run apps on the fw, we use the device as a fw/router.
2) run two instances of iperf3 using --affinity switch
3) scheduling of threads can be an issue, the kernel wants to move threads around cpu's (scheduler attempt to balance). As-is, manipulating the OS so that the cpu's wanted for just iperf3 have only iperf3 threads on those cpu's is not easy doing. Threads that move take time because the cache on new cpu needs to fetch from memory. One trick here is, if you want 8 threads and eliminate the cache timing issue, run 8 instances of iperf3 where each is bound to a unique cpu. You can also tune the Time Quantum settings for the default ULE scheduler.
I bet you can get above 8G in testing, but this will not be indicative of actual fw/router bandwidth. The next step I would do, test the device as just a router (tuned iperf3 server and client on each side of the router, dec in middle).
Run the same test as you did, note the results, then just run "sysctl net.inet.tcp.sendbuf_max=16777216" then re-run the testing again, any difference?
All that said, there's tcp RTT issues not accounted for, which impacts measured bandwidth.
https://www.pinglabz.com/how-to-use-iperf3/
https://www.pinglabz.com/iperf3-parallel-streams-tcp-window/
Quote from: BrandyWine on August 30, 2026, 05:57:40 PM[...]1) iperf3 (on the router) is not a tool for testing bandwidth of a router. We dont run apps on the fw, we use the device as a fw/router.
[...]
Run the same test as you did, note the results, then just run "sysctl net.inet.tcp.sendbuf_max=16777216" then re-run the testing again, any difference?[...]
Good point. Folks can tune the heck out of OPNsense as a session terminator, but it generally serves little purpose (improved performance for benchmarks against the firewall itself, and some services, perhaps?) and any (permanent) tuneables should be checked/maintained across (kernel) upgrades to avoid obsolete/potentially detrimental cruft.
I tested with iperf with the router, but through the router too. No difference, same results.
Tried with the --affinity switch on client side, the one core had about 30% load, seems not to be a bottleneck.
Cannot use --affinity on the server side, as its not my iperf server :-). But server side is from Init7, they have a very good reputation and also provide 25GBit/s internet.
Tested with "sysctl net.inet.tcp.sendbuf_max=16777216", unfortunately same result.
I have a bridge configured between two 1G LAN interfaces. 10G interfaces are not part of the bridge. Disabled the bridge and the two 1G LAN interfaces too. Did not make any difference.
I cannot get more than 5.63-6.0 Gbits/sec.
Could it depend on the SFP+ module I chose?
What else may it be?
Quote from: power13 on September 15, 2026, 09:58:03 PM[...]Could it depend on the SFP+ module I chose?[...]
So long as the interface clocks are sync'd there should not be a silent condition that would affect speed (and async clocking seems unlikely, particularly at 10Gb). You could fall down a component testing rabbit hole... which might be enlightening, but would certainly be expensive.
The "1.9W" spec (for the UACC-CM-RJ45-MG) looks decent. I wonder what chippie they use (guessing RealTek). I don't think "ifconfig -v" would give anything more than the product ID.
Speaking of "silent", how does netstat look? Anything untoward?
Quote from: power13 on September 15, 2026, 09:58:03 PMI cannot get more than 5.63-6.0 Gbits/sec.
For what's it worth :
I use to build 2 x 10 Gbps LACP setups for Hosting/Co-location related customers and those use to perform like a 12 to 13 Gbps connection in pretty much all cases and never reached anything near 20 Gbps in all cases.
So if you can hit 6 Gbps with a single 10 Gbps link I would say you are doing good despite the fact that I see more and more 8 Gbps reports with newer hardware over the last 5 years or so than the stuff that I was working with at the time...
As already mentioned by others,
DO NOT test iperf using the FW as a client or a server, test THRU it but not TO it.
As well post output of command on FW as root.
netstat -Q
If Dispatch policy & IP is not hybrid you have it set wrongly. In that case set a tunable
net.isr.dispatch with value
hybrid
Restart your FW and check the command again.
You should see something like this
# netstat -Q
Configuration:
Setting Current Limit
Thread count 8 8
Default queue limit 2048 10240
Dispatch policy hybrid n/a
Threads bound to CPUs enabled n/a
Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 cpu hybrid C--
igmp 2 2048 source default ---
rtsock 3 2048 source default ---
arp 4 2048 source default ---
ether 5 2048 cpu direct C--
ip6 6 1000 cpu hybrid C--
ip_direct 9 2048 cpu hybrid C--
ip6_direct 10 2048 cpu hybrid C--
Regards,
S.
I'm curious to know if you ever end up getting better performance out of the DEC750. Here is my thread about the same topic.
https://forum.opnsense.org/index.php?topic=50846.0
And the only way I managed to max out my internet connection:
https://forum.opnsense.org/index.php?topic=50846.msg268263#msg268263
I don't see that anyone has recommended this, and it may not be feasible but....you could plug a machine directly in to you ISP's endpoint and do a speed test from there. And then plug directly in to your OPNSense fw and check from there.
You might not have full bandwidth on the public side of your fw. It may not even be an OPNSense issue.
Quote from: stanps on September 25, 2026, 03:04:55 AMI don't see that anyone has recommended this, and it may not be feasible but....you could plug a machine directly in to you ISP's endpoint and do a speed test from there. And then plug directly in to your OPNSense fw and check from there.
You might not have full bandwidth on the public side of your fw. It may not even be an OPNSense issue.
It's definitely not an ISP issue, i get full speed with different machine, same opnsense config and sfp+ modules. CPU on DEC 750 is too slow for 10G, unless there are some config options that we've missed.
Quote from: ou1 on September 25, 2026, 07:53:57 AMCPU on DEC 750 is too slow for 10G
There was another topic where the DEC850 seemed to be the better choice indeed :)
Quote from: nero355 on September 25, 2026, 09:53:36 PMThere was another topic where the DEC850 seemed to be the better choice indeed :)
Yes, unfortunately at 1650 EUR (current price) and 45W typical power usage, this puts it out of reach for me as a home user. My M920q setup was under 400EUR including the X710 card and typically uses <20W. Not really comparable, an ancient mini PC with a hole cut into it and a fan over the hole, but it's silent and it gets the job done.
Maybe they didn't anticipate 10G internet to be so readily available when they designed the DEC750.
Quote from: ou1 on September 26, 2026, 10:04:57 AMYes, unfortunately at 1650 EUR (current price) and 45W typical power usage, this puts it out of reach for me as a home user.
My whole network was something like € 1300 (Router/Switches/Accesspoints/Cables/etc.) years ago so paying more than that for just one appliance is a bit too much indeed :)
QuoteMy M920q setup was under 400EUR including the X710 card and typically uses <20W.
Not really comparable, an ancient mini PC with a hole cut into it and a fan over the hole, but it's silent and it gets the job done.
That was this : https://forum.opnsense.org/index.php?topic=50846.msg268263#msg268263
Right ?
Looks good to me! :)
QuoteMaybe they didn't anticipate 10G internet to be so readily available when they designed the DEC750.
Whatever the reason was : I am a big fan of simply using OPNsense-like software on some DIY-ish hardware setup anyway :)
At least from a Home User point of view...
I tested with a "Ubiquiti UniFi Cloud Gateway UCG-Fiber" and I get 6Gbps up and down. So looks like, it was not necessarily the DEC750 :-). I tested through the Unifi device and from / to the Unifi device.
Have you tried to disable powerD?
Quote from: power13 on October 05, 2026, 12:51:47 PMI tested with a "Ubiquiti UniFi Cloud Gateway UCG-Fiber" and I get 6Gbps up and down. So looks like, it was not necessarily the DEC750 :-). I tested through the Unifi device and from / to the Unifi device.
Too bad I have not bookmarked all those horror stories about Ubiquiti UniFi Routers and 10 Gbps connections for moments like these... ^_^
Anyway...
Do yourself a favor and test it like this : https://forum.opnsense.org/index.php?topic=50846.msg268263#msg268263
IMHO the way to go in such cases :)