Sub-Gigabit speed on Intel J4125 with Intel X540 PCIe card

[TheDJ]

I recently switched to the Asrock J4125B-ITX (https://www.asrock.com/MB/Intel/J4125B-ITX/#Specification) board as the base for my Opnsense FW.
For switching, I use an Inter-Tech ST-7214 PCIe card (https://www.inter-tech.de/produktdetails-198/ST-7214_EN.html), which features 2x RJ-45 100/1000/10GBase-T based on an Intel X540 chipset.
The card is hooked up to an Zyxel XGS1250 10G port.
On the same switch, I am running my Unraid server, connected via a 2.5G NIC.

Here is my problem: when I run an iperf3 on the same (VLAN) subnet directly to the FW, I only achieve around 750-800Mbit/s throughput (even with -P 5 flag)
Opnsense and the switch both report 10G full-duplex for the connection. The switch and Unraid both report 2.5G full-duplex for their connection.
IPS/IDS is turned off, Crowdsec is running, but even turning it off does not improve performance.
CRC, TSO, and LRO offload are all turned off.

I am aware, that the board has a RTL8111H NIC and I know of the problems of FreeBSD with Realtek, but this NIC is only hooked up to WAN and should therefore not be part of the problem.
I am aware, that the CPU might not be powerful enough for full 10G, but at the same time, Topton sells a box with 4x2.5G with the same CPU (https://www.servethehome.com/topton-intel-j4125-4x-i225-fanless-virtualized-firewall-appliance-review-pfsense-opnsense-proxmox-ve/)

I am now wondering, how to investigate further.

[meyergru]

That is quite easy to explain:

1. Never measure speed to/from the OpnSense itself. This is known to give bad results. Instead, measure routed speeds through OpnSense.

2. The J4125 is a quite limited platform that has only 6 PCIe 2.0 lanes (see https://www.intel.de/content/www/de/de/products/sku/197305/intel-celeron-processor-j4125-4m-cache-up-to-2-70-ghz/specifications.html)

3. On your motherboard, most of these lanes are attached to peripherals, as the board is a general purpose board, leaving only 2 PCIe 2.0 lanes available on your (mechanical) PCIe x16 slot (see https://www.asrock.com/MB/Intel/J4125B-ITX/#Specification). This is different in specific firewall units where most of the lanes are assigned to the NICs.

4. Your NIC is a PCIe 2.1 x8 adapter which has two PHYs attached (see https://www.inter-tech.de/produktdetails-198/ST-7214_EN.html). While PCIe is capable of handling any combination of standards and number of lanes, matter-of-fact you are limiting both your PHYs to PCIe 2.0 x1, which is 500 MByte/s (see https://www.elektronik-kompendium.de/sites/com/0904051.htm).

So, with some added overhead, your observed results are to be expected. Your hardware is also not suited to the task at hand, mainly because not all J4125 platforms are architecturally equivalent. You will never get more speed than 5000 MBit/s.

That being said, J4125 is also limited by processing power plus newer CPUs even have more PCIe lanes and newer, faster standards (e.g. N5105: PCIe 3.0 x8). You should use X550 based adapters in that case, because X540 limits you to PCIe 2.0 anyway.

[TheDJ]

That is a great rundown at totally makes sense!

I was coming from an Intel NUC Ghost Canyon i5, that I got for cheap and wanted to decrease power draw.
Thank you for the thorough investigation. Maybe, I am still in the return window. Then I will upgrade my hardware.
Apparently I was just fooled by the PCIe x16 and did not bother to check, how many of them are actually wired.

[vpx23]

The cheapest replacement board probably is the Biostar B450NH with the cheapest boxed AM4 socket CPU AMD Ryzen 5 4500.

That's ~160 EUR.

[meyergru]

And by that, needing an additional graphics card? Buy a 4600G instead or even a 5600G. For the board, I would look for something with an Intel NIC because of better support.

However, you also need DDR-DIMM instead of SODIMM, so cannot repurpose that. All in all you are looking at a price where a quiet chinese specialized firewall box with 4x I226V costs the same at 1/3 the power consumption. If your server has only 2.5 GBit/s anyway, I would do just that.

[TheDJ]

Thank you again for your input!

I was still in the return window and I thought about futureproofing. So I went with the AMD 4350G Pro on a Fatal1ty B450 Gaming-ITX/ac (https://www.asrock.com/mb/AMD/Fatal1ty%20B450%20Gaming-ITXac/index.de.asp#Specification)

I installed it today and... I'm still kind of underwhelmed: I was running and iperf through the Firewall with 2 2.5G NICs on both ends and I maxed out at ~1.6/1.7G.
Better than before obviously, but far from wire speed.
What am I missing?
The board runs PCIe 3.0x16 with Renoir chips and no bifurcation is active, 10GbaseT is reported by the switch and the FW.

Am I missing something again?

EDIT: I still had DIMMs flying around and threw them in, so the upgrade was not too expensive.

[vpx23]

My best bet would be that it is the switch:

https://community.zyxel.com/en/discussion/14072/xgs1250-12-very-slow

https://community.zyxel.com/en/discussion/10854/xgs1250-12-very-very-very-slow

Also I assume the X540 not being able to negotiate down to 2.5G could be a problem by overwhelming the switch from converting 10G to 2.5G.

[TheDJ]

Wow, thanks again. I never thought of that.
This would definitely make sense - the switch is also placed in a rather airflow-constricted cabinet, so I guess that would contribute to it.

Great that basically my whole setup has some issues. At least I think I can stay with the new board + CPU now.
I will monitor it a bit and then see how it goes.

Thanks again. If I find anything new, I'll update.

[vpx23]

This switch looks very promising and is about the same price range: https://mikrotik.com/product/crs310_8g_2s_in

But I can't guarantee it will solve all the problems. You could connect the X540 with DAC.