Testing bitrate of Gigabit ports of my PC Engines APU2E4

[hushcoden]

I'm playing around with iperf3 testing the bitrate of the three Intel i210AT Ethernet ports of my APU2E4 - https://pcengines.ch/apu2e4.htm - connected to my laptop (Intel i5-3380M CPU @2.90 GHz dual-core, 12 GB of RAM DDR3) and I'm not getting good results, an average of 230 Mb/s while I was expxecting something around 900+ Mb/s

Code: [Select]

Connecting to host 192.168.0.1, port 5201
[  5] local 192.168.0.10 port 53877 connected to 192.168.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  28.5 MBytes   239 Mbits/sec    0    351 KBytes
[  5]   1.00-2.00   sec  27.6 MBytes   231 Mbits/sec    0    351 KBytes
[  5]   2.00-3.00   sec  26.3 MBytes   220 Mbits/sec    0    352 KBytes
[  5]   3.00-4.00   sec  26.2 MBytes   219 Mbits/sec    0    352 KBytes
[  5]   4.00-5.00   sec  26.9 MBytes   226 Mbits/sec    0    352 KBytes
[  5]   5.00-6.00   sec  27.5 MBytes   230 Mbits/sec    0    352 KBytes
[  5]   6.00-7.00   sec  27.7 MBytes   232 Mbits/sec    0    352 KBytes
[  5]   7.00-8.00   sec  27.3 MBytes   229 Mbits/sec    0    352 KBytes
[  5]   8.00-9.00   sec  28.0 MBytes   234 Mbits/sec    0    352 KBytes
[  5]   9.00-10.00  sec  26.9 MBytes   226 Mbits/sec    0    352 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   273 MBytes   229 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   272 MBytes   228 Mbits/sec                  receiver

iperf Done.

In interface settings, I left the deafult values (pic. opn1) and I've just added the following tunables (pic. opn2)

Should I remove the tunetables or I'm missing something ?

Tia.

[FlightService]

I am having issues obtaining full speed as well.  I am maxing out at around 600 down.  I am surprised at how little support these speed posts get.

I have to say, from a support and performance point of view.  I am thinking of shelving my opnsense unit and moving to a COTS one.

[Gauss23]

I am having issues obtaining full speed as well.  I am maxing out at around 600 down.  I am surprised at how little support these speed posts get.

I have to say, from a support and performance point of view.  I am thinking of shelving my opnsense unit and moving to a COTS one.

Support? This forum is community driven. Did you pay something for the software or the support?

As I read the perfomance issues have something to do with the switch to the current BSD version. It will get better with improving drivers.

Do you think those COTS products hold what their shiny product sheets promise?

Both of you are not telling what you are testing and what features are enabled on your boxes.

Running iperf against the OPNsense is not a good idea. It's always better to have 2 clients running iperf through the OPNsense to tell the real routing speeds.

[FlightService]

Support? This forum is community driven. Did you pay something for the software or the support?

Do you think those COTS products hold what their shiny product sheets promise?

Both of you are not telling what you are testing and what features are enabled on your boxes.

Thanks for the reinforcement of my opinion of the support here.  No I do not expect COTS to match their spec sheets, but I do trust COTS third party reviewers.

For the record I have my own thread with all the information you asked.  I was just searching and came across this thread with similar issues.

Before you blast people looking for solutions and making a comment about their observation on support, do a search for gigabit and see how many posts there are with no/minimal responses.
The best post out there is almost a full page of the OPs own research before another person comments.


Sent from my iPhone using Tapatalk

[Gauss23]

What should other users tell you about performance issues in some drivers? They are not able to help, so there is just not so much of an answer right now.

I'm blasting no one.

I just checked your thread and found that you are using outdated hardware and compare throughput from a modem to the throughput of a firewall. Did you try completely disabling firewalling on the OPNsense? That would be a comparison.

Opnsense router consists of a Lenovo M91p USFF with a Intel Core i5-2400S CPU @ 2.50GHz (4 cores), 8 gb ram, 120 gb SSD, with a Jeirdus (Intel Chipset 82546) dual-port gigabit 8492MT PCI Server NIC which is connected to a TP-Link switch. (Yes, the M91p USFF is weird with a PCI instead of PCI-e, I know I am close to saturating the PCI bus with a gigabit connection but Fios doesn't seem to be able to deliver over 1.1 gigabit to my house and never consistently.)

[FlightService]

all features are disabled, Opnsense is doing nothing but routing.

Whats funny, is my "outdated hardware" would clock over 1Gbps on Windows 10 when connected directly to my ONT.  I get it isn't the newest, but that is one thing the opnsense is advertised for

Yeah, you blasted me, hence the quote. 

[Gauss23]

If you feel blasted I'm sorry for that, but I react in an allergic way if people believe that the have the natural right to be supported in an Open Source project. I honestly apologize for any bad feelings.

I just checked your hardware combination again. Did you ever tried another OS to use this box as a router and received full speed?

When you are using the box as a Windows client, you get of course 1Gbit.

But when using this box as a router all packets are coming in on one port, need to pass the CPU and go out on the other port. The numbers you presented are not bad for a PCI dual-NIC. In- and Output need to share the PCI bandwidth.

It would be interesting if you would try a maybe Linux based firewall/router on this box to see if your PCI bus is able to get more throughput.

[Gauss23]

And one more thing you could try:
via Console: pkg install speedtest-cli

And try the speedtest directly on the OPNsense. This would be a comparable result with your Windows installation on the same box.

[FlightService]

The dual NIC only is connected on one port.  The WAN is through the motherboard and the LAN is on one port of the dual NIC.  When I went to set this up I was warned that trying to run the WAN through the NIC would be an issue. 


Sent from my iPhone using Tapatalk

[Gauss23]

Result of speedtest-cli wou ld be interesting. As it would be the same as you were running it on Windows.

Here is a troubleshooting-guide, most of the hints are for OPNsense, too.
https://docs.netgate.com/pfsense/en/latest/troubleshooting/low-throughput.html

Especially the Hardware offloading stuff would be worth a try.

[FlightService]

I can give it a shot.  I tried https://teklager.se/en/knowledge-base/opnsense-performance-optimization/


Sent from my iPhone using Tapatalk

[Gauss23]

The most obvious test is to watch the firewall CPU load while transferring data. This can be observed from Diagnostics > System Activity or from the shell by running:

top -aSH
If an IRQ process for a network card is using a significant amount of CPU on a core, then either the hardware is being fully (or over) utilized, or the driver may need adjustments to work as expected. If the firewall is not under any stress whatsoever while transferring data, the problem likely lies elsewhere.

This is also an important check.

And enabling hardware offloading does not change anything?

[FlightService]

My CPU is practically at idle constantly


Sent from my iPhone using Tapatalk

[dave]

Couple of things you might want to do:

Update your BIOS:
https://pcengines.ch/howto.htm#bios
https://pcengines.github.io/

Teklager have some optimisations:
https://teklager.se/en/knowledge-base/opnsense-performance-optimization/

Also, if you're running a PPPoE based WAN interface, that can affect performance.

From what I understand this is a BSD thing.
APU's are capable of hitting gigabit throughput with Linux based OS's like IPFire (something to do with multi-threading), and BSD's PPPoE implementation kinda sucks apparently.

[FlightService]

So going through this thread of interest again I noticed a few hardware settings I will try this weekend.  (Work deadlines have been kicking in this week and the kids have been tele-schooling so I try to avoid down time during the week).  If that and what has been suggested here fails I will try loading IPFire.  If that fails I will probably just smash an easy button and install a Omada Cloud key and router (my WAP is already an Omada)

More to come over the next few days.


Sent from my iPhone using Tapatalk