Slow iperf score over ethernet

[rama3124]

Not sure if this is the right forum to post this but i have just set up a mini pc running opnsense and ran iperf using a cat 6 cable connected from the LAN port of the mini PC directly into my macbook (through an ethernet to usb C adaptor) and it gave me speeds around 95Mbits/sec. I tried using macbook connected to 5ghz wifi from access point connected via cat 6 ethernet (to eliminate the USB-c adaptor as the cause of the problem) and still got the same speeds. Running speedtest on my laptop using Ookla also gives 95Mbits/sec using either wifi or ethernet cat6 cable direct to opnsense router. Yet speedtest directly from opnsense is 250Mbits/sec (the speed that i'm paying for). Any ideas why this might happen? I'm thoroughly confused

Thanks

[gctwnl]

I have had comparable experiences on OPNsense on the LAN interfaces. Basically, OPNsense needs good hardware. I use OPNsense hardware and the 600 series was so underpowered that my Gbit ethernet maxed out at something like 300Mbps when talking to the router via ethernet (iperf3).

Not having that much time, I bought beefier hardware (800 series) and now the LAN interface runs at 1Gbps as it should (iperf3 with -P, see below) with the occasional drop in performance.

Note: you might use multiple (3-5) parallel (-P or --parallel) in your iperf3 test. A single thread may not be able to fill the entire bandwidth all th etime (it should, but it doesn't). I.e. Mac-to-Mac, I get an easy 1Gbps on a single thread iperf3 test, but so far my experience with routers have been they are not as good as my Macs. I also have a Core i5 Intel NUC and it also easily does 1Gbps on a single threaded iperf3. My guess is that the router's work on interfaces with stuff like pf is a heavy burden for the network stack in the kernel.

In my experience, also with another router before (EdgeMax, Gbps router that was able to do max 300Mbps on the 1 Gbit LAN ethernet ports and much less when a bridge was added), having not enough power kinds of makes that you do not get the theoretically expected speeds. Hardware-wise, the ethernet port may be 1Gbps, but the software gets bogged down easily (e.g. when there is a VLAN and matters get complex in the networking stack). Another thing to make sure is that your minipc has AES in its chip so anything encryption doesn't get you a performance penalty for VPNs and such

[rama3124]

I don't think the hardware is the issue since it's a n5105 celeron and the CPU is only  at about 30% usage when I check the dashboard. I have only 4gb ram but even this is at about 50% usage at maximum. I have disabled all intrusion prevention (zenarmot and suricata) but still get the same 95MBps rate even when just running iperf between my Mac directly connected to the opnsense box with a cat6 ethernet cable. The fact that all network speed tests hover around 85 to 95Mbps make me think something is setting a speed limit in opnsense.

Has anyone experienced something similar?

[cookiemonster]

It sounds like either the adaptors or other hardware is limiting. Please check the LAN interface is getting set to the correct speed ie. "1000baseT <full-duplex>" or whatever is right for your hardware, but that is above 100. It sounds like it might be syncing at 100.

[lilsense]

it could also be zenarmor. You may want to tweak your system with RSS. I was getting 400mbps and now I am getting 900+... :)

[rama3124]

It sounds like either the adaptors or other hardware is limiting. Please check the LAN interface is getting set to the correct speed ie. "1000baseT <full-duplex>" or whatever is right for your hardware, but that is above 100. It sounds like it might be syncing at 100.

You are 100% correct! it is 100baseT in my LAN interface overview. I think this is some kind of issue with my aliexpress Topton PC since it has the newer i226 intel NIC. Anyone know of a workaround?

[cookiemonster]

I'm using i226 intel nics too without problem. I am virtualizing it though, so not directly comparable. But make sure the OPN version is based on freebsd 13, that has the updated drivers for i226.
We're going to have to go down to the details. You need to check dmesg, verify drivers used, etc.

[rama3124]

Fixed it, it turns out that it was a bad ethernet cable. Thanks for all the help

[cookiemonster]

Great!