Sanity check for N100 / i226 on a 2000 / 2000 line

[Ozymandias]

I've just upgraded from a PPPoE based 900 / 110 full fibre contract to a DHCP based 2000 / 2000 with YouFibre.

I get 2000 / 2000 with the supplied router using Windows 11 and Ookla speedtest.

I get 1400 / 1400 with my N100 i226 based router both from bare metal OPNsense cli and Windows 11 (direct connection to the router).

I get similar 1400 / 1400 iperf3 numbers to and from Win11 to OPNsense (and over 2000 both ways to Unraid on the network)

I was hoping that my router would run at full 2000 / 2000 speed. Am I being unrealistic?


System:

CWWK / Topton CW-ADLN-5L Ver: 1.0 Fengshang Edition

Bios Build Date and Time - 04/26/2023 14:02:13
Bios settings:
• ASPM disabled on all ports
• PCIe speed set to Auto on all ports

N100

16GB DDR5 Ram
1TB Samsung 980 NVMe

5 port Intel I226-V
  EEPROM V2.17-0 eTrack 0x80000303

####
No line shaping or IDS/IPS.

Interfaces:
• Disable hardware checksum offload - checked
• Disable hardware TCP segmentation offload - checked
• Disable hardware large receive offload - checked
• VLAN Hardware Filtering - Disable VLAN Hardware Filtering

Tunables:
dev.igc.*.fc = 0
net.inet.rss.enabled = 1
net.inet.rss.bits = 2

net.isr.dispatch = deferred
net.isr.bindthreads = 1
net.isr.maxthreads = -1

/boot/loader.conf.local
hw.pci.enable_aspm = 0

[nero355]

I was hoping that my router would run at full 2000 / 2000 speed.

Am I being unrealistic?

What is the CPU LOAD like during those benchmarks ??


Have you tried simply downloading something instead of Ookla Speedtest.net benchmarks ?
Or any Public iPerf3 Servers ?

[Ozymandias]

Peaks at about 40%.

iperf3:
1.48 Gbits/sec from the router to a public server.
1.28 Gbits/sec from Win11 to the same public server.
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.

[dirtyfreebooter]

when RSS is enabled,

Code Select Expand

net.inet.rss.enabled = 1
the kernel will override dispatch and always make it hybrid (so you can leave that tunable out if you are enabling RSS), you can verify my claim with netstat -Q

Code Select Expand

net.isr.dispatch = hybrid
---

i tested a protectli VP2440 with N150 with i226 with Zenarmor and i was able to get 2.5 gbps

using the x710 10g interface without Zenarmor, i was able to get 9.46 gbps

so i would think that N100 could easily do 2 Gbps without IDS/IPS.

the tunables i used were (including installing the Intel Microcode Plugin):

Code Select Expand

dev.hwpstate_intel.0.epp = 10
dev.hwpstate_intel.1.epp = 10
dev.hwpstate_intel.2.epp = 10
dev.hwpstate_intel.3.epp = 10
dev.igc.0.fc = 0
dev.igc.1.fc = 0
hw.acpi.cpu.cx_lowest = C2
hw.ibrs_disable = 1
net.inet.tcp.soreceive_stream = 1
net.isr.bindthreads = 1
net.isr.maxthreads = -1
vm.pmap.pcid_enabled = 0
vm.pmap.pti = 0

note that i think the intel microcode plugin + vm.pmap.pcid_enabled = 0 is pretty much the gold standard with the N100/N150/N305 cpus, as they have a bug in the freebsd kernel otherwise...

i did update my i226 firmwares to v2.32 tho i dont think that mattered.

[meyergru]

If your connection was still on PPPoE, it would blame that, but it is not.

I have tested the pure routing speed of N100-based OpnSense systems between VLANs to ~6 Gbps, so that should work.

W/R to the Windows 11 system, I have seen network drivers that needed tuning to exceed 1.5 Gbps on some network adapters, like disabling interrupt moderation, enlarge buffer sizes or changing flow of control settings.

[nero355]

Peaks at about 40%.

iperf3:
1.48 Gbits/sec from the router to a public server.
1.28 Gbits/sec from Win11 to the same public server.
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.

I forgot to mention this :

I did update my i226 firmwares to v2.32 tho i dont think that mattered.

So have a look at this topic : https://forum.opnsense.org/index.php?topic=48695.0

Maybe that's the issue here...

I have tested the pure routing speed of N100-based OpnSense systems between VLANs to ~6 Gbps, so that should work.

NICE! :)

So far I only knew that a N100 can satisfy the 2,5 Gbps NICs out there as long as you don't use any IDS/IPS stuff.

W/R to the Windows 11 system, I have seen network drivers that needed tuning to exceed 1.5 Gbps on some network adapters, like disabling interrupt moderation, enlarge buffer sizes or changing flow of control settings.

He got good speeds to his DIY NAS :

2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.

So I am guessing that's not the issue...

[meyergru]

Peaks at about 40%.

iperf3:
1.48 Gbits/sec from the router to a public server.
1.28 Gbits/sec from Win11 to the same public server.
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.

I forgot to mention this :

I did update my i226 firmwares to v2.32 tho i dont think that mattered.

So have a look at this topic : https://forum.opnsense.org/index.php?topic=48695.0

Maybe that's the issue here...

I have tested the pure routing speed of N100-based OpnSense systems between VLANs to ~6 Gbps, so that should work.

NICE! :)

So far I only knew that a N100 can satisfy the 2,5 Gbps NICs out there as long as you don't use any IDS/IPS stuff.

W/R to the Windows 11 system, I have seen network drivers that needed tuning to exceed 1.5 Gbps on some network adapters, like disabling interrupt moderation, enlarge buffer sizes or changing flow of control settings.

He got good speeds to his DIY NAS :

2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.

So I am guessing that's not the issue...

There are big timing differences between internet and local connections and also, the endpoints can behave differently, thus there can be any number of problems w/r to timing and/or buffering. The latter depends on which TCP algorithms are in use, such that flow control and buffering or interrupt coalescing can very well play a role.

[nero355]

There are big timing differences between internet and local connections and also, the endpoints can behave differently, thus there can be any number of problems w/r to timing and/or buffering.
The latter depends on which TCP algorithms are in use, such that flow control and buffering or interrupt coalescing can very well play a role.

OK, but this clearly shows there is something wrong with the OPNsense Mini PC itself :

iperf3:
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.

These two should always be equal IMO :)

To rule out FreeBSD specific issues he could also test with a random Linux based Live Boot Environment now that I think of it...

[Ozymandias]

I'll get a Linux based Live Boot up over the next few days and give it a test, plus maybe a fresh OPNsense basic install on a spare drive.

[meyergru]

OK, but this clearly shows there is something wrong with the OPNsense Mini PC itself :

iperf3:
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.

These two should always be equal IMO :)

No:

1. There is a big difference between OpnSense routing sessions between different partners and OpnSense being the endpoint (the latter one is slower).

2. Since this is iperf, I also like to point out this article, point 10. I totally depends on how many TCP sessions you use.

Pulling these together, I see ~1.87 Gbps with iperf -P1 vs. 3.56 Gbps with iperf -P4 vs. 6 Gbps when OpnSense routes only.