From DEC850 to Minisforum MS-A2 - 10 Gbit throughput problems

[virtualdimension]

Hi all! I need your help.

I have a Deciso DEC850 (OPNsense Business, OPNsense 25.10.2_4-amd64) that limit the throughput at 1 Gbit with IDS/IPS. 1.2 without.
I use Zenarmor for the LAN, Suricata IDS/IPS for the WAN, Crowdsec, Q-Feeds (this has only been in place for 2 weeks), Unbound DNS, and the GeoIP rules included in the OPNsense Business package and those from MaxMind.
As long as I had a 200 Mbit internet connection I didn't have this problem, but now I've activated 10 Gbit FTTH (actual download speed 8 Gbit and upload 2 Gbit) I've found that I can't take full advantage of it.

My internet provider's modem (which also acts as an ONT) is connected via RJ45 cat.7 network cable from the 10Gbit output to the 10Gbit input of the DEC850 with the 10Gtek ASF-10G-T80 SFP+ module (I also used an Ubiquiti UACC-CM-RJ45-MG) and I created a DMZ for the IP address provided by the modem to the DEC850 firewall.
From the 10 Gbit output of the DEC850 I enter the 10 Gbit WAN port of a Ubiquiti UDM-PRO MAX with the 10G Direct Attach Cable (UACC-DAC-SFP10-1M).
I connected a Ubiquiti Pro Max 24 PoE switch to the UDM-PRO MAX, again using a direct 10 Gbit SFP+ cable.
All devices (access points, computers, printers, smartphones, etc.) are managed by the UDM-PRO MAX.
In the latter I do not use blocking rules, nor IDS/IPS module, but even if I activate it it has very little impact on the throughput (about 200 Mbit).

Well, if I run an iPerf3 test directly between the firewall and the Ubiquiti, the throughput rate is always around 1 Gbit.
If I disable IDS/IPS, Zenarmor, Crowdsec, and Q-Feeds on the DEC850, I only gain about 200 Mbit.

I also tried all the possible options through Tunables with the guides and advice I found in the forum and online, but the situation did not change.

So I bought a Minisforum MS-A2 with the AMD Ryzen 9 9955HX, 64GB (2x32) DDR5 5600MHz, Crucial T710 1TB SSD... and replaced the DEC850, connecting everything in the same way with the network card inside the new barebone (2 x 10G SFP+ Intel X710). I also updated the Intel network card's BIOS to the latest available version. Hardware offload is disabled. Tunables all done, but these make no difference.

Now, with the same configurations, compared to before I can reach 4 Gbit of transfer between UDM and Firewall, even if they are not constant and the speeds fluctuate a lot. Sometimes it is below 3 Gbit, other times it reaches 4-5 Gbit, but not more. I ran the iPerf tests both with the UDM set as a server and as a client (same thing with the OPNsense firewall).
Now, when I run a internet speed test directly from the UDM, I can actually reach peaks of 4 Gbps download and 1.2 Gbps upload. Still too far from the maximum values ��of my internet connection. If I connect directly to the 10 Gbit LAN port of the internet provider's modem, I can reach the maximum speed (about 7.8-8.5 Gbit download and 1.8-2 Gbit upload.

Even in this case, leaving Zenarmor and Suricata enabled/disable makes no difference. The processor (base clock 2.5 GHz and max boost clock up to 5.4 GHz) peaks at 6% utilization during use. The RAM, with all settings enabled, sits at around 16GB (out of 64 total).
Zenarmor doesn't yet support multicore, but even pinning a core to it doesn't make any difference, and the processor isn't being fully utilized. The processor literally yawns, and I don't understand what's creating this bottleneck that's preventing me from reaching 10 Gbit/s of throughput.

I'm asking for help in trying to understand and resolve what may be causing this problem and limitation.

Thank you in advance.

[JamesFrisch]

I am not quiet sure if I understand what you iperf or why you even have UDM and OPNsense. I would go with either one of them.
I also don't quiet get you setup or network topology, nor what speed exactly is your problem. And I can't give you good advise on why your Minisforum performs that bad. So I can just give you some general advise that applies to anyone. Maybe that helps.

- iperf3 is by default single core. Look into the multithread option
- make sure you have power savings disabled. I had mine on hiadaptive, got 5GB/s for the first thest, and when I ran the second test shortly after, I got my 9GBit/s because the CPU could not enter power saving yet. Disabling PowerD got me always 9GBit/s.
- Even my old 4-core i3-8100 is fast enough for 9GBit/s. But I don't run Zenarmor or Suricata, only Crowdsec.

[meyergru]

You did not say anything apart from the Zenarmor part about multithreading.

See this, point 10:

a. With speeds > 1 GBit, you need to enable multithreading for unimpeded measurements.
b. You also need to enable RSS to use all cores. It may also depend on how the FreeBSD NIC drivers are optimized, there may be special tuneables for yours.

That being said: With Zenarmor, you can only utilize one thread at this time. Period. Pinning a core does only make sure that this one core does not get utilized by anything else, but you will still be limited by it.

[nero355]

My internet provider's modem (which also acts as an ONT) is connected via RJ45 cat.7 network cable

I am not sure what the current status is, but if you want 10 Gbps speed via RJ45 then CAT6a is actually the only Certified and Officially Acknowledged cable type, while everything above it still awaits official recognition so to speak.

from the 10Gbit output to the 10Gbit input of the DEC850 with the 10Gtek ASF-10G-T80 SFP+ module (I also used an Ubiquiti UACC-CM-RJ45-MG) and I created a DMZ for the IP address provided by the modem to the DEC850 firewall.

Since you are using a DMZ IP Address my first question would be : Private or Public range ?

Also please note that most modules that convert SFP+ to RJ45 or the other way around work up to 30 meters of cable !!

From the 10 Gbit output of the DEC850 I enter the 10 Gbit WAN port of a Ubiquiti UDM-PRO MAX with the 10G Direct Attach Cable (UACC-DAC-SFP10-1M).

Why ?!

The whole Ubiquiti UniFi UDM range is a mess and especially when messing around with 10 Gbps connections and even worse when there is buffering to 1 Gbps connections involved and the WAN uses PPPoE on top of that...
If you need the UniFi Controller part of it then just connect it as a Client to your Management VLAN and leave it like that in the future or replace it with a simple dedicated UniFi Controller device.

If I connect directly to the 10 Gbit LAN port of the internet provider's modem, I can reach the maximum speed (about 7.8-8.5 Gbit download and 1.8-2 Gbit upload.

This is with a PC connected to the ISP's ONT/Router combo device you mentioned in the DMZ setup part above I assume ?

Which OS and what kind of hardware does it have ?


Whatever you do from this point on please stick to testing with :
- DEC850
- Minisforum MS-A2
- The PC you used for the full speed 10 Gbps test.

There is no point in involving the Ubiquiti UniFi UDM Pro in this whole thing!

- iperf3 is by default single core. Look into the multithread option

Larger Window Sizes can improve the speed too! :)

[virtualdimension]

I am not quiet sure if I understand what you iperf or why you even have UDM and OPNsense. I would go with either one of them.
I also don't quiet get you setup or network topology, nor what speed exactly is your problem. And I can't give you good advise on why your Minisforum performs that bad. So I can just give you some general advise that applies to anyone. Maybe that helps.

- iperf3 is by default single core. Look into the multithread option
- make sure you have power savings disabled. I had mine on hiadaptive, got 5GB/s for the first thest, and when I ran the second test shortly after, I got my 9GBit/s because the CPU could not enter power saving yet. Disabling PowerD got me always 9GBit/s.
- Even my old 4-core i3-8100 is fast enough for 9GBit/s. But I don't run Zenarmor or Suricata, only Crowdsec.

I use OPNsense as a firewall because it has many more features and is much more secure than Ubiquiti, but I prefer Ubiquiti for its ecosystem (switches, access points, cameras, graphical interface, secure remote access, etc.), so I use both.
So, from the modem where the ONT is directly connected, I enter the OPNsense WAN port, and then the filtered connection passes from there to the Ubiquiti.

Yes, I am aware that the default iperf test is single core, in fact I have done several tests with the -P 4 (8, 16) commands

[virtualdimension]

You did not say anything apart from the Zenarmor part about multithreading.

See this, point 10:

a. With speeds > 1 GBit, you need to enable multithreading for unimpeded measurements.
b. You also need to enable RSS to use all cores. It may also depend on how the FreeBSD NIC drivers are optimized, there may be special tuneables for yours.

That being said: With Zenarmor, you can only utilize one thread at this time. Period. Pinning a core does only make sure that this one core does not get utilized by anything else, but you will still be limited by it.

Yes, I also activated RSS through tunables.

- net.inet.rss.enabled = 1
- net.isr.bindthreads = 1
- net.isr.maxthreads = -1
- net.inet.rss.bits = 4
- dev.ixl.0.rss_enabled = 1
- dev.ixl.1.rss_enabled = 1
- net.isr.dispatch = deferred (also tried hybrid)
- kern.ipc.maxsockbuf = 16777216

I tried using powerD (both with the Hyadaptive and Maximum options), but then I disabled it because it didn't make any difference. In the BIOS, I also disabled all power-saving options for both the CPU and network cards.
The promiscuous mode is on.
The "pin core" on Zenarmor is off.

[virtualdimension]

My internet provider's modem (which also acts as an ONT) is connected via RJ45 cat.7 network cable

I am not sure what the current status is, but if you want 10 Gbps speed via RJ45 then CAT6a is actually the only Certified and Officially Acknowledged cable type, while everything above it still awaits official recognition so to speak.

from the 10Gbit output to the 10Gbit input of the DEC850 with the 10Gtek ASF-10G-T80 SFP+ module (I also used an Ubiquiti UACC-CM-RJ45-MG) and I created a DMZ for the IP address provided by the modem to the DEC850 firewall.

Since you are using a DMZ IP Address my first question would be : Private or Public range ?

Also please note that most modules that convert SFP+ to RJ45 or the other way around work up to 30 meters of cable !!

From the 10 Gbit output of the DEC850 I enter the 10 Gbit WAN port of a Ubiquiti UDM-PRO MAX with the 10G Direct Attach Cable (UACC-DAC-SFP10-1M).

Why ?!

The whole Ubiquiti UniFi UDM range is a mess and especially when messing around with 10 Gbps connections and even worse when there is buffering to 1 Gbps connections involved and the WAN uses PPPoE on top of that...
If you need the UniFi Controller part of it then just connect it as a Client to your Management VLAN and leave it like that in the future or replace it with a simple dedicated UniFi Controller device.

If I connect directly to the 10 Gbit LAN port of the internet provider's modem, I can reach the maximum speed (about 7.8-8.5 Gbit download and 1.8-2 Gbit upload.

This is with a PC connected to the ISP's ONT/Router combo device you mentioned in the DMZ setup part above I assume ?

Which OS and what kind of hardware does it have ?


Whatever you do from this point on please stick to testing with :
- DEC850
- Minisforum MS-A2
- The PC you used for the full speed 10 Gbps test.

There is no point in involving the Ubiquiti UniFi UDM Pro in this whole thing!

- iperf3 is by default single core. Look into the multithread option

Larger Window Sizes can improve the speed too! :)

- I've tried also with cat.6A, but doesn't change nothing.

- The DMZ IP address is private. I've removed all Bogons network blocks in OPNsense.

- The connection cables between the modem/firewall/Ubiquiti are 1 meter long

- I know that PPPoE connections (especially with Ubiquiti) can cause problems, but that's not the case with me, since the UDM receives the connection already established by the modem through the firewall. However, if that doesn't solve the problem, I'll try what you suggested with the controller.

- Yes, you're correct. I used both a desktop computer (and a mobile workstation) with Windows 11 and 10 Gbit network cards (both internal and external).

- The Ubiquiti UniFi UDM PRO MAX is the device to which all the devices are connected, so it is the one that must ultimately guarantee the final performance, but to try to understand who is the bottleneck I have to do the tests between the UDM and OPNsense and between OPNsense and the main modem.

Modem IP: 192.168.10.1
Minisforum MS-A2 IP: 192.168.1.1
Ubiquiti IP: 10.0.0.1

I've tried iperf3 -c (and -s) from and to Minisforum and Ubiquiti, with -P 1 (and 8) and -t 30
I've tried disabling Zenarmor, Crowdsec, and Suricata, but it makes virtually no difference in terms of throughput. The speed is still around ~3.5-4 Gbps, and the processor is practically idle. I'll have to try connecting directly to the Minisforum, bypassing the UDM, and see what results I get.

[virtualdimension]

The network card in the Minisforum is an Intel X710-DA2.
I don't know which Intel driver is preinstalled in OPNsense. The latest driver available for FreeBSD and my network card is from late December 2025 on the Intel website.
FreeBSD driver for Intel X710
But I don't know how to install it...

[pfry]

[...]I don't know which Intel driver is preinstalled in OPNsense.[...]

It'd be the kernel driver, most likely. Version numbering may be different - I haven't looked at the driver on Intel's web site. I wouldn't expect the X710 to be your issue, but I suppose you never can tell. I haven't really tested mine yet - I need to get a couple new servers running and upgrade my Internet link. You might want to look at PCI-e speed and width, just to be sure.

Code Select Expand

root@fw:/home/user # dmesg | grep ixl
[1] ixl0: <Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k> mem 0xf5000000-0xf57fffff,0xf5a18000-0xf5a1ffff at device 0.0 on pci1
[1] ixl0: fw 9.152.77998 api 1.15 nvm 9.50 etid 8000f4ab oem 1.270.0
[1] ixl0: PF-ID[0]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
[1] ixl0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ixl0: Using 8 RX queues 8 TX queues
[1] ixl0: Using MSI-X interrupts with 9 vectors
[1] ixl0: Ethernet address: 3c:fd:fe:e7:2d:88
[1] ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
[1] ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
[1] ixl0: SR-IOV ready
[1] ixl0: netmap queues/slots: TX 8/1024, RX 8/1024