I've just upgraded from a PPPoE based 900 / 110 full fibre contract to a DHCP based 2000 / 2000 with YouFibre.
I get 2000 / 2000 with the supplied router using Windows 11 and Ookla speedtest.
I get 1400 / 1400 with my N100 i226 based router both from bare metal OPNsense cli and Windows 11 (direct connection to the router).
I get similar 1400 / 1400 iperf3 numbers to and from Win11 to OPNsense (and over 2000 both ways to Unraid on the network)
I was hoping that my router would run at full 2000 / 2000 speed. Am I being unrealistic?
System:
CWWK / Topton CW-ADLN-5L Ver: 1.0 Fengshang Edition
Bios Build Date and Time - 04/26/2023 14:02:13
Bios settings:
• ASPM disabled on all ports
• PCIe speed set to Auto on all ports
N100
16GB DDR5 Ram
1TB Samsung 980 NVMe
5 port Intel I226-V
EEPROM V2.17-0 eTrack 0x80000303
####
No line shaping or IDS/IPS.
Interfaces:
• Disable hardware checksum offload - checked
• Disable hardware TCP segmentation offload - checked
• Disable hardware large receive offload - checked
• VLAN Hardware Filtering - Disable VLAN Hardware Filtering
Tunables:
dev.igc.*.fc = 0
net.inet.rss.enabled = 1
net.inet.rss.bits = 2
net.isr.dispatch = deferred
net.isr.bindthreads = 1
net.isr.maxthreads = -1
/boot/loader.conf.local
hw.pci.enable_aspm = 0
Quote from: Ozymandias on June 06, 2026, 06:56:40 PMI was hoping that my router would run at full 2000 / 2000 speed.
Am I being unrealistic?
What is the CPU LOAD like during those benchmarks ??
Have you tried simply downloading something instead of Ookla Speedtest.net benchmarks ?
Or any Public iPerf3 Servers ?
Peaks at about 40%.
iperf3:
1.48 Gbits/sec from the router to a public server.
1.28 Gbits/sec from Win11 to the same public server.
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.
when RSS is enabled,
net.inet.rss.enabled = 1
the kernel will override dispatch and always make it hybrid (so you can leave that tunable out if you are enabling RSS), you can verify my claim with netstat -Q
net.isr.dispatch = hybrid
---
i tested a protectli VP2440 with N150 with i226 with Zenarmor and i was able to get 2.5 gbps
using the x710 10g interface without Zenarmor, i was able to get 9.46 gbps
so i would think that N100 could easily do 2 Gbps without IDS/IPS.
the tunables i used were (including installing the Intel Microcode Plugin):
dev.hwpstate_intel.0.epp = 10
dev.hwpstate_intel.1.epp = 10
dev.hwpstate_intel.2.epp = 10
dev.hwpstate_intel.3.epp = 10
dev.igc.0.fc = 0
dev.igc.1.fc = 0
hw.acpi.cpu.cx_lowest = C2
hw.ibrs_disable = 1
net.inet.tcp.soreceive_stream = 1
net.isr.bindthreads = 1
net.isr.maxthreads = -1
vm.pmap.pcid_enabled = 0
vm.pmap.pti = 0
note that i think the intel microcode plugin + vm.pmap.pcid_enabled = 0 is pretty much the gold standard with the N100/N150/N305 cpus, as they have a bug in the freebsd kernel otherwise...
i did update my i226 firmwares to v2.32 tho i dont think that mattered.
If your connection was still on PPPoE, it would blame that, but it is not.
I have tested the pure routing speed of N100-based OpnSense systems between VLANs to ~6 Gbps, so that should work.
W/R to the Windows 11 system, I have seen network drivers that needed tuning to exceed 1.5 Gbps on some network adapters, like disabling interrupt moderation, enlarge buffer sizes or changing flow of control settings.
Quote from: Ozymandias on June 06, 2026, 11:39:10 PMPeaks at about 40%.
iperf3:
1.48 Gbits/sec from the router to a public server.
1.28 Gbits/sec from Win11 to the same public server.
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.
I forgot to mention this :
Quote from: dirtyfreebooter on June 07, 2026, 03:28:26 AMI did update my i226 firmwares to v2.32 tho i dont think that mattered.
So have a look at this topic : https://forum.opnsense.org/index.php?topic=48695.0
Maybe that's the issue here...
Quote from: meyergru on June 07, 2026, 08:35:27 AMI have tested the pure routing speed of N100-based OpnSense systems between VLANs to ~6 Gbps, so that should work.
NICE! :)
So far I only knew that a N100 can satisfy the 2,5 Gbps NICs out there as long as you don't use any IDS/IPS stuff.
QuoteW/R to the Windows 11 system, I have seen network drivers that needed tuning to exceed 1.5 Gbps on some network adapters, like disabling interrupt moderation, enlarge buffer sizes or changing flow of control settings.
He got good speeds to his DIY NAS :
Quote2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.
So I am guessing that's not the issue...
Quote from: nero355 on June 07, 2026, 06:07:27 PMQuote from: Ozymandias on June 06, 2026, 11:39:10 PMPeaks at about 40%.
iperf3:
1.48 Gbits/sec from the router to a public server.
1.28 Gbits/sec from Win11 to the same public server.
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.
I forgot to mention this :
Quote from: dirtyfreebooter on June 07, 2026, 03:28:26 AMI did update my i226 firmwares to v2.32 tho i dont think that mattered.
So have a look at this topic : https://forum.opnsense.org/index.php?topic=48695.0
Maybe that's the issue here...
Quote from: meyergru on June 07, 2026, 08:35:27 AMI have tested the pure routing speed of N100-based OpnSense systems between VLANs to ~6 Gbps, so that should work.
NICE! :)
So far I only knew that a N100 can satisfy the 2,5 Gbps NICs out there as long as you don't use any IDS/IPS stuff.
QuoteW/R to the Windows 11 system, I have seen network drivers that needed tuning to exceed 1.5 Gbps on some network adapters, like disabling interrupt moderation, enlarge buffer sizes or changing flow of control settings.
He got good speeds to his DIY NAS :
Quote2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.
So I am guessing that's not the issue...
There are big timing differences between internet and local connections and also, the endpoints can behave differently, thus there can be any number of problems w/r to timing and/or buffering. The latter depends on which TCP algorithms are in use, such that flow control and buffering or interrupt coalescing can very well play a role.
Quote from: meyergru on June 07, 2026, 06:15:06 PMThere are big timing differences between internet and local connections and also, the endpoints can behave differently, thus there can be any number of problems w/r to timing and/or buffering.
The latter depends on which TCP algorithms are in use, such that flow control and buffering or interrupt coalescing can very well play a role.
OK, but this clearly shows there is something wrong with the OPNsense Mini PC itself :
Quote from: Ozymandias on June 06, 2026, 11:39:10 PMiperf3:
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.
These two should always be equal IMO :)
To rule out FreeBSD specific issues he could also test with a random Linux based Live Boot Environment now that I think of it...
I'll get a Linux based Live Boot up over the next few days and give it a test, plus maybe a fresh OPNsense basic install on a spare drive.
Quote from: nero355 on June 07, 2026, 06:30:17 PMOK, but this clearly shows there is something wrong with the OPNsense Mini PC itself :
Quoteiperf3:
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.
These two should always be equal IMO :)
No:
1. There is a big difference between OpnSense routing sessions between different partners and OpnSense being the endpoint (the latter one is slower).
2. Since this is iperf, I also like to point out this article (https://forum.opnsense.org/index.php?topic=42985.0), point 10. I totally depends on how many TCP sessions you use.
Pulling these together, I see ~1.87 Gbps with iperf -P1 vs. 3.56 Gbps with iperf -P4 vs. 6 Gbps when OpnSense routes only.
Quote from: Ozymandias on June 07, 2026, 06:42:28 PMI'll get a Linux based Live Boot up over the next few days and give it a test, plus maybe a fresh OPNsense basic install on a spare drive.
Good plan! :)
Quote from: meyergru on June 07, 2026, 06:51:47 PM1. There is a big difference between OpnSense routing sessions between different partners and OpnSense being the endpoint (the latter one is slower).
So you are saying that when I measure iPerf3 speeds between OPNsense and a Client that the speed will always be lower than between two any other type of Clients on the same subnet ?!
For example :Windows or Linux iPerf3 Server/Client <----> Windows or Linux iPerf3 Server/Client = 2,37 Gbps
OPNsense iPerf3 Server/Client <----> Windows or Linux iPerf3 Server/Client = Always less than the speed above ?!
Quote2. Since this is iperf, I also like to point out this article (https://forum.opnsense.org/index.php?topic=42985.0), point 10. I totally depends on how many TCP sessions you use.
Pulling these together, I see ~1.87 Gbps with iperf -P1 vs. 3.56 Gbps with iperf -P4 vs. 6 Gbps when OpnSense routes only.
Ofcourse you will always have to measure with as much
threads as possible and sometimes even raise the
window size and stuff like that... :)
Quote from: nero355 on June 07, 2026, 11:29:15 PMSo you are saying that when I measure iPerf3 speeds between OPNsense and a Client that the speed will always be lower than between two any other type of Clients on the same subnet ?!
Yes, of course, just because when OpnSense is itself involved, iperf3 runs on top of the actual routing. Well, "on the same subnet" is incorrect, of course, because OpnSense would not be involved at all if the traffic were between two clients on layer 2.
Quote from: nero355 on June 07, 2026, 11:29:15 PMOfcourse you will always have to measure with as much threads as possible and sometimes even raise the window size and stuff like that... :)
You may know, but the OP did not state how exactly the measurements were taken, which is why I pointed it out again. RSS will not do any good with just one connection.
iperf3 was done with -P 4
Quote from: meyergru on June 08, 2026, 07:02:58 AMYes, of course, just because when OpnSense is itself involved, iperf3 runs on top of the actual routing.
But I still don't get it :
A connection between OPNsense and a Client talking directly to it involves 0,0 routing ?!
Why would that be slower than the speed between two random Clients that are let's say connected to the same Default LAN network of OPNsense ?
QuoteWell, "on the same subnet" is incorrect, of course, because OpnSense would not be involved at all if the traffic were between two clients on layer 2.
Again :
OPNsense Default LAN network : 192.168.1.0/24
The speeds between :
OPNsense 192.168.1.1 <----> Random Client using Windows/Linux/MacOS/*BSD 192.168.1.22
And :
Random Client using Windows/Linux/MacOS/*BSD 192.168.1.22 <----> Random Client using Windows/Linux/MacOS/*BSD 192.168.1.42
Should always be the same in my opinion.
If that's not the case I would like to know why they are not/why you personally think they are not ?!
QuoteYou may know, but the OP did not state how exactly the measurements were taken, which is why I pointed it out again. RSS will not do any good with just one connection.
It looked like he knows what he is doing so I decided to have a little trust in my fellow OPNsense user :)
Quote from: nero355 on June 08, 2026, 06:47:47 PM[...]Why would that be slower than the speed between two random Clients that are let's say connected to the same Default LAN network of OPNsense?[...]
Additional CPU load. Also, OPNsense likely compares well to FreeBSD (and perhaps less well to Windows or Linux), but I tune FreeBSD extensively for network/TCP performance, while I don't bother so much with OPNsense, as I terminate few sessions on the firewall. But then, I'm only speculating, as I haven't tested this scenario.
Stop OPNsense FW services and run your windows test through the router, what speed does it get.
I think for anything above 1Gb you need to tune the device as a fast router. Anything beyond just router and it will slow things down.
You can also test through the fw from a host on port A to a host on port B. A ping test with various sizes, ping -s or ping -l, and record the response time, from there it's just math (use AI bot). You can also load up bandwidth by sending large sums of UDP from A to B.
Quote from: pfry on June 09, 2026, 02:38:57 AMAdditional CPU load.
I think that's not it to be honest...
I have also never seen such claims/issues when people use to build their DIY NAS based on FreeBSD with ZFS before there was OpenZFS for Linux for example.
And those sometimes even had 10 Gbps NICs onboard !! :)
Quote from: nero355 on June 09, 2026, 02:14:24 PMI think that's not it to be honest...[...]
Heh. I agree. But it should be eliminated as a possibility.
After dabbling with Alpine and going back into the bios I realised that PCIe 1 and 2 were set to Gen1 (I did think I had changed them to Auto a number of years ago). After amending to both to Auto I'm now getting 2.10 Gbits/sec to a public server from OPNsense, 2.12 Gbits/sec from Win11, and 2.35 Gbits/sec OPNsense to Unraid. Thanks for all the input.
Quote from: Ozymandias on June 09, 2026, 06:31:49 PMAfter dabbling with Alpine and going back into the bios I realised that PCIe 1 and 2 were set to Gen1 (I did think I had changed them to Auto a number of years ago). After amending to both to Auto I'm now getting 2.10 Gbits/sec to a public server from OPNsense, 2.12 Gbits/sec from Win11, and 2.35 Gbits/sec OPNsense to Unraid. Thanks for all the input.
LOL! I totally forgot that part! NICE! :)
I didn't have PCI-e clock on my bingo card. I probably should have, as I have an AMD B650 system where its e810 card will not run at 16Gt/s (v4) in a CPU slot, but will in the chipset slot (the two available slots are both x4). So I habitually look at pciconf when I put together FreeBSD/OPNsense systems.
Quote from: pfry on June 10, 2026, 02:03:29 AMI have an AMD B650 system
IMO those things should not exist in the first place :
The Mainstream CPUs and Chipsets are soo crippled compared to HEDT systems that there should only be the X models (for AMD) and Z models (for Intel) and nothing else, but sadly the world does not agree with me :'(
I know things have shifted slightly in the meantime, but the sweetness of something like my good old i7 3930K and the AsRock X79 Extreme11 was just too good to not to buy at the time! ^_^
Quote from: nero355 on June 10, 2026, 05:20:28 PM[...]The Mainstream CPUs and Chipsets are soo crippled compared to HEDT systems that there should only be the X models (for AMD) and Z models (for Intel)[...]
I don't need more than basic chipset features, and there are few boards with ECC support and a good PCI-e slot layout. For Intel, I would only use what is now the Wx80 chipset line, as I only use ECC RAM, but the premium price is unattractive. I'd prefer a bit more CPU I/O, but actually differentiated HEDT/server/whatever setups (like your old Sandy Bridge E) also cost more than I'm willing to pay. (I have a Haswell E very similar to your Sandy E: an E5-1650v3 on a Supermicro motherboard with 32GB of RAM, sitting in a cabinet. Prices were... less differentiated back then. Funny, I'd expect an Ivy Bridge with that 3-series model number. Intel and AMD - gotta love 'em.)