Only half speed on 2.5 GBE LAN

[Rhaegar79]

Hello everyone,
I'm new to the forum and hope I'm in the right place with my question / problem.
I have installed OPNsense 24.1 on a fanless mini PC with the following features

• Intel Alder Lake-N 12th Gen N100
• 4x 2.5 GBE Intel I226-V
• 16 GB RAM
• 500 GB NVMe SSD

I have initially only connected the PC to the network via the LAN connection (igc1 in my case); later it will be the first device after the internet modem on a fibre optic connection that has not yet been switched. Until the connection is there, I only want to set up the OPNsense so far and use it as a DNS server.

In principle, everything works as desired, but when measuring with iperf3 I only ever get exactly half of the maximum possible speed on the LAN port. Here is some more information:

• In the dashboard, the port is correctly displayed as "2500Base-T <full-duplex>".
• All components are designed for 2.5 GBit, i.e. the switch, the cable and the remote end (desktop PC).
• If I start an Ubuntu Live system or a NomadBSD on the mini PC, iperf3 gives me a speed of 2.37 GBit/s.
• No special services are running, only unbound.
• CPU usage is at 28% max during iperf3
• RAM usage is not noticable

However, in OPNsense I only get 1.18 GBit/s, which is pretty much half of what is possible in other installations on the same device. I have tried the following settings/changes, but it always remains at the specified throughput:

• Hardware offloading enabled/disabled
• Tried some tuneables according to various recommendations for multi-gigabit connections
• Update to the latest version of OPNsense
• Deactivate unbound
• Tried different values of -P in iperf3

No matter what I do, OPNsense only ever measures a maximum of 1.18 GBit/s, although exactly the same hardware shows the full speed under other operating systems (including a FreeBSD variant).

Does anyone else have any ideas as to why this could be? Am I making a mistake with regard to the utilization of the ports?

*Additional info*
I just tried a live version of OPNsense 22.7. There I get 2.37 GBit/s. Haven't tried OPNsense 23, yet, but will do that maybe tomorrow.

[Rhaegar79]

Okay, I now know that it does not depend on the version of OPNsense, at all. There's an entirely different reason / cause.
When I tried different versions of OPNsense to find out if I get the low speeds with every version, I did not configure the interfaces in the exact same way every time. That led to sometimes getting the full speed and sometimes only getting half the possible speed.
Here's how I configured the LAN interface when I only got half speed with the live USB stick:

• After first boot I selected option 2 --> LAN interface --> DHCP. That led to a working connection to the LAN including DNS server and the local gateway address of my router, but the speed was reduced as described.
• I also tried option 2 --> LAN interface --> no DHCP --> static IP/netmask --> upstream gateway address set to my local router. That again led to a working connection to my LAN, but the speed was reduced, same as before.

When I got the full speed I configured the interface like this:
Option 2 --> LAN interface --> no DHCP --> static IP/netmask --> no upstream gateway address. I then set a default route to my local router via "sudo route add default router_ip" and I got the full speed. I then installed OPNsense permanently on my M.2 SSD. After the reboot the default gateway wasn't there anymore, of course, the above command is not a permanent setting, so I configured a gateway in the web UI via System --> Gateways --> Configuration. I didn't the the upstream option. I then configured a default route via System -->Routes --> Configuration where I set the target IP 0.0.0.0/0 to the gateway I just configured. I still got the full speed in iperf3.

I still don't know what exactly is causing the low speed, but that's probably just my lack of understanding how to correctly set up the interfaces. I'm new to OPNsense, so it's probably a stupid mistake. Would love for anyone to explain to me the reason for this / the mistake I made.

[audit13]

Hi,

Were you able to figure out the issue?

I'm a total noob and decided to try out a barebones Topton n100 with 4 x Intel i226v ports, added a 120 GB 2.5" SATA drive, and 8 GB of skhynix ram.

I started with bare metal 24.1 about a month ago and progressively updated to 24.7.1 as updates were released. Unit has been stable on all releases with no crashes.

I'm using eth0 for PPPoE (1.5/1.0 down/up) on VLAN 35 and eth1 for LAN. Not running any services other than Wireguard server and DDNS.

I connected my desktop directly to eth1 and can only get gigabit down and up as per Ookla tests on Chrome/Win10. With my asus gt-ax6000 and the same network cables, I can get 1.5/1.0.

Opnsense and Windows say the desktop is connected at 2.5 to eth1.

Maybe I have an incorrect bios setting for eth0 (WAN) port?

Any information would be appreciated.

Thanks for reading.

[doktornotor]

I'm using eth0 for PPPoE (1.5/1.0 down/up) on VLAN 35

PPPoE is single-threaded on FreeBSD. Your CPU won't do any better. Move on.

[audit13]

PPPoE is single-threaded on FreeBSD. Your CPU won't do any better. Move on.

Thank you for the reply.

I'm okay with gigabit for now since all of my equipment, with the exception of 1 desktop, has gigabit NICs.

[doktornotor]

Just adding some popular links...

https://docs.opnsense.org/troubleshooting/performance.html
https://www.neelc.org/posts/multicore-pppoe/

[meyergru]

I use pppoe with a slower CPU and it works quite well for more than 1 Gbps (see my signature) - at least if I do not use something fancy like Zenarmor or Suricata on top.

I would check first if the WAN interface speed is really at 2.5 Gbps, some ONTs do not work too well with autonegotiation. You can look for that in the details of the parent interface (eth0) under Interfaces: Overview, it should say "2500Base-T <full-duplex>". Enabling RSS would be the next thing.

[audit13]

I looked under assignment overview for PPPoE and see nothing to indicate link speed.

For Opt2 (igc0) which is the WAN, I do see 2500.

[doktornotor]

PPPoE does not show any link speed, because it does not make sense for PPP interfaces.