From DEC850 to Minisforum MS-A2 - 10 Gbit throughput problems

[virtualdimension]

Hi all! I need your help.

I have a Deciso DEC850 (OPNsense Business, OPNsense 25.10.2_4-amd64) that limit the throughput at 1 Gbit with IDS/IPS. 1.2 without.
I use Zenarmor for the LAN, Suricata IDS/IPS for the WAN, Crowdsec, Q-Feeds (this has only been in place for 2 weeks), Unbound DNS, and the GeoIP rules included in the OPNsense Business package and those from MaxMind.
As long as I had a 200 Mbit internet connection I didn't have this problem, but now I've activated 10 Gbit FTTH (actual download speed 8 Gbit and upload 2 Gbit) I've found that I can't take full advantage of it.

My internet provider's modem (which also acts as an ONT) is connected via RJ45 cat.7 network cable from the 10Gbit output to the 10Gbit input of the DEC850 with the 10Gtek ASF-10G-T80 SFP+ module (I also used an Ubiquiti UACC-CM-RJ45-MG) and I created a DMZ for the IP address provided by the modem to the DEC850 firewall.
From the 10 Gbit output of the DEC850 I enter the 10 Gbit WAN port of a Ubiquiti UDM-PRO MAX with the 10G Direct Attach Cable (UACC-DAC-SFP10-1M).
I connected a Ubiquiti Pro Max 24 PoE switch to the UDM-PRO MAX, again using a direct 10 Gbit SFP+ cable.
All devices (access points, computers, printers, smartphones, etc.) are managed by the UDM-PRO MAX.
In the latter I do not use blocking rules, nor IDS/IPS module, but even if I activate it it has very little impact on the throughput (about 200 Mbit).

Well, if I run an iPerf3 test directly between the firewall and the Ubiquiti, the throughput rate is always around 1 Gbit.
If I disable IDS/IPS, Zenarmor, Crowdsec, and Q-Feeds on the DEC850, I only gain about 200 Mbit.

I also tried all the possible options through Tunables with the guides and advice I found in the forum and online, but the situation did not change.

So I bought a Minisforum MS-A2 with the AMD Ryzen 9 9955HX, 64GB (2x32) DDR5 5600MHz, Crucial T710 1TB SSD... and replaced the DEC850, connecting everything in the same way with the network card inside the new barebone (2 x 10G SFP+ Intel X710). I also updated the Intel network card's BIOS to the latest available version. Hardware offload is disabled. Tunables all done, but these make no difference.

Now, with the same configurations, compared to before I can reach 4 Gbit of transfer between UDM and Firewall, even if they are not constant and the speeds fluctuate a lot. Sometimes it is below 3 Gbit, other times it reaches 4-5 Gbit, but not more. I ran the iPerf tests both with the UDM set as a server and as a client (same thing with the OPNsense firewall).
Now, when I run a internet speed test directly from the UDM, I can actually reach peaks of 4 Gbps download and 1.2 Gbps upload. Still too far from the maximum values ��of my internet connection. If I connect directly to the 10 Gbit LAN port of the internet provider's modem, I can reach the maximum speed (about 7.8-8.5 Gbit download and 1.8-2 Gbit upload.

Even in this case, leaving Zenarmor and Suricata enabled/disable makes no difference. The processor (base clock 2.5 GHz and max boost clock up to 5.4 GHz) peaks at 6% utilization during use. The RAM, with all settings enabled, sits at around 16GB (out of 64 total).
Zenarmor doesn't yet support multicore, but even pinning a core to it doesn't make any difference, and the processor isn't being fully utilized. The processor literally yawns, and I don't understand what's creating this bottleneck that's preventing me from reaching 10 Gbit/s of throughput.

I'm asking for help in trying to understand and resolve what may be causing this problem and limitation.

Thank you in advance.

[JamesFrisch]

I am not quiet sure if I understand what you iperf or why you even have UDM and OPNsense. I would go with either one of them.
I also don't quiet get you setup or network topology, nor what speed exactly is your problem. And I can't give you good advise on why your Minisforum performs that bad. So I can just give you some general advise that applies to anyone. Maybe that helps.

- iperf3 is by default single core. Look into the multithread option
- make sure you have power savings disabled. I had mine on hiadaptive, got 5GB/s for the first thest, and when I ran the second test shortly after, I got my 9GBit/s because the CPU could not enter power saving yet. Disabling PowerD got me always 9GBit/s.
- Even my old 4-core i3-8100 is fast enough for 9GBit/s. But I don't run Zenarmor or Suricata, only Crowdsec.

[meyergru]

You did not say anything apart from the Zenarmor part about multithreading.

See this, point 10:

a. With speeds > 1 GBit, you need to enable multithreading for unimpeded measurements.
b. You also need to enable RSS to use all cores. It may also depend on how the FreeBSD NIC drivers are optimized, there may be special tuneables for yours.

That being said: With Zenarmor, you can only utilize one thread at this time. Period. Pinning a core does only make sure that this one core does not get utilized by anything else, but you will still be limited by it.

[nero355]

My internet provider's modem (which also acts as an ONT) is connected via RJ45 cat.7 network cable

I am not sure what the current status is, but if you want 10 Gbps speed via RJ45 then CAT6a is actually the only Certified and Officially Acknowledged cable type, while everything above it still awaits official recognition so to speak.

from the 10Gbit output to the 10Gbit input of the DEC850 with the 10Gtek ASF-10G-T80 SFP+ module (I also used an Ubiquiti UACC-CM-RJ45-MG) and I created a DMZ for the IP address provided by the modem to the DEC850 firewall.

Since you are using a DMZ IP Address my first question would be : Private or Public range ?

Also please note that most modules that convert SFP+ to RJ45 or the other way around work up to 30 meters of cable !!

From the 10 Gbit output of the DEC850 I enter the 10 Gbit WAN port of a Ubiquiti UDM-PRO MAX with the 10G Direct Attach Cable (UACC-DAC-SFP10-1M).

Why ?!

The whole Ubiquiti UniFi UDM range is a mess and especially when messing around with 10 Gbps connections and even worse when there is buffering to 1 Gbps connections involved and the WAN uses PPPoE on top of that...
If you need the UniFi Controller part of it then just connect it as a Client to your Management VLAN and leave it like that in the future or replace it with a simple dedicated UniFi Controller device.

If I connect directly to the 10 Gbit LAN port of the internet provider's modem, I can reach the maximum speed (about 7.8-8.5 Gbit download and 1.8-2 Gbit upload.

This is with a PC connected to the ISP's ONT/Router combo device you mentioned in the DMZ setup part above I assume ?

Which OS and what kind of hardware does it have ?


Whatever you do from this point on please stick to testing with :
- DEC850
- Minisforum MS-A2
- The PC you used for the full speed 10 Gbps test.

There is no point in involving the Ubiquiti UniFi UDM Pro in this whole thing!

- iperf3 is by default single core. Look into the multithread option

Larger Window Sizes can improve the speed too! :)