Not getting gigabit throughput on iperf3

[pkazmir]

Hi,

I'm new to opnsense and freebsd but I do have experience on linux, so not a complete newbie.

I've installed the latest opnsense on an HP t740 running an AMD V1756B (4 cores/8 threads) w/8GB of memory. I also installed a dual-port intel i255 PCI card from QNAP, because I intend to eventually upgrade my ISP connection to 2GB fiber. (Right now it's 1GB.) My internal network is all 1GB but eventually I may upgrade it to 2.5 as well, at least for key systems.

Anyhow, I have this system set up only connected to my LAN at the moment so I can learn how to configure it. I installed iperf3 on it and when I run a test from my Windows PC on the same (1GB) switch, I get about about 94 Mbits/s, or 384 Mbits/s for 4 connections. But when I run the test from my Windows PC to my Windows Server on the same network, I get about 235 Mbits/s, or 931 Mbits/s for 4 connections.

I did some googling and found a number of results that talked about setting tunables especially for Intel cards (my interfaces are igc0 and igc1). I've set all of those and rebooted and it didn't make a difference. ifconfig looks normal to me. CPU usage never goes about 5% or so on any core. I've even checked the cable (which is a cat-6 cable) to see if that is the problem.

Other than the tunables I haven't done anything else to the system except set up the basic settings (hostname, gateway, DNS settings, and theme).

Can someone help me figure out what's going on?

Thanks!
Peter

[pkazmir]

I reinstalled the hardware with a clean install of Windows 10, and re-ran the iperf3 test, and got full gigabit throughput. So it's definitely a software configurating/tuning/driver issue with the OPNSense install.

Any ideas?

[pkazmir]

Also to try to narrow things down, I installed pfSense on the same hardware and I got full gigabit throughput. So it's definitely something in the OPNSense distribution/configuration...

[pkazmir]

Reinstalled OPNSense and iperf3, and test went back to the bad result again - in fact, even worse - 246 Mbits/sec.

Feeling frustrated, I really don't want to use pfSense.

[pkazmir]

And one more data point...when running iperf3 on OPNSense in *client* mode, I get full wire speed. I'm only getting the bad speeds when I run it in *server* mode (AND client mode in reverse).

Does that help? Any suggestions at all?

Thanks,
Peter

[BrakkeBezem]

I have exactly the same issue. I get 940+ speeds when opnsense is the server, when opnsense is the client, I only get speeds from 700-750. Anybody has any idea why this would happen?

[BrakkeBezem]

It seems that on the interface without vlans, the performance is normal. Do vlans have such a heavy performance penalty?

[glasi]

Thanks for you effort. I had already considered doing some performance tests with pfsense as well.

I am having the same performance issues. And we are not alone.

In the past I did compare the routing performance of OPNsense and Ubuntu (https://forum.opnsense.org/index.php?topic=18754.msg91547#msg91547). Pretty clear result: OPNsense crap, Ubuntu full wirespeed.

Surprisingly, a rollback to OPNsense 20.1 gives me full wirespeed.

Something seems to be broken in OPNsense.

[BrakkeBezem]

I have seen some similar posts around this, although I don't know if they're equal to my problem. They all seem to have lower download than upload speeds though. Never seen a solution to this. I am using one of the protectli boxes: VP2410 – 4 Port Intel ® J4125

[glasi]

There shouldn't be any problems with the CPU. With pfsense and Win10 you have already been able to achieve higher data rates.

By the way, I use a Supermicro A2SDi-4C-HLN4F motherboard with an Intel Atom C3558 CPU.

In case there is free time this weekend I will rebuild my complete setup on OPNsense 20.1 for testing purposes.

[BrakkeBezem]

Hi Glasi,

Have you had any chance of testing this out?

[Vilhonator]

iperf isn't exactly 100% accurate on Opnsense, so you can find out different results from time to time.

Also don't confuse Gb/s with GB/s (Gb/s = Gigabit per second which about 125 Megabytes per second), as long as you are getting around 800 Mb/s to 1Gb/s, there's nothing wrong, speeds are roughly 12-20% slower depending on network card and cables etc.

Way you are able to test out speeds in some accuracy, is to send huge files between 2 or more devices within same network.
As long as it takes longer than few seconds (infact longer it takes, the better), you get realistic results.