Expected LAN throughput on APU 2E4?

[datenimperator]

Dear all,

I'm running current stable OpnSense on an APU 2E4 AMD GX-412TC SOC (4 cores) w/ 4GB RAM. No VPN, typical load average ~0.5, 20-40% CPU usage.

I started using this setup while I was on a 400 Mb WAN, and it was well capable to saturate the line. My ISP upgraded to 1Gb 6 months ago, and I was never able to see more than ~600 Mb throughput, whatever I tried. I installed iperf on the router and on a wired client, also nowhere near one gigabit.

I've read about router performance issues with the kernel that comes with 20.7. Are these expected to be mitigated any time soon? Will upgrading onto a stronger CPU help? Should I downgrade to 20.1?

Any help is appreciated. Regards

Christian

[Ricardo]

https://forum.opnsense.org/index.php?topic=9264.0

The absolute max WAN speed is anywhere between ~250Mbit/sec and ~650Mbit/sec, highly dependant on type of WAN==PPPoE (produces worst performance) and if NAT+pf enabled. If any other extra CPU-heavy services are enabled, like IPS/IDS, performance can go even under 250Mbit)sec very easily.

[datenimperator]

The speed increased dramatically when I turned off flow control in the NIC driver. In System->Settings->Tunables I added

Code: [Select]

dev.igb.0.fc = 0
dev.igb.1.fc = 0
dev.igb.2.fc = 0

Additional improvement was to disable the meltdown patches, also in tunables:

Code: [Select]

vm.pmap.pti = 0

Now I'm back to 920Mbit LAN->WAN throughput on the same hardware. Also, CPU usage reduced a lot. For the record, I'm using an external Technicolor TC4400 cable modem which doesn't require any PPPoE but simply hands out DHCP addresses. Regards

Christian

[hushcoden]

I have a pretty basic setup, modem <--> APU2E4 <--> access point and it works just fine.

I've run a few tests with iperf3 between my laptop (connected to the AP at 1 Gb/s with cat6) and the (main) LAN of my APU2E4 and I cannot go over 200 Mb/s - tried with FC on and off and I've seen a minimal difference: is it because my connection is PPPoE or should I consider to re-install OPNsense ?

Also, can the PPPoE type of connection affect the other LAN ports speed?

Tia.

[hushcoden]

Now I'm back to 920Mbit LAN->WAN throughput on the same hardware.

Can you share details on how you're performing the test?

[datenimperator]

That's what my ISP Unitymedia gives me. Having said that, I'm totally aware that this isn't anywhere near a scientific measurement. speedtest.unitymedia.de (now actually Vodafone) gives me between 600 and 950 Mbit/s, depending on weekday and time. I consider this like something of a "last mile bandwidth". On sunday mornings the line is pretty fast, they reported 940 MBit/s some minutes ago.

speedtest.net reports around 650 MBit/s now.

Running iperf on the firewall and a wired client reports a lot less, 276 MBit/s. I assume that directly relates to things like IO and CPU resources on the firewall?

[cmmh]

Thank you for your tunables settings.  They've helped to more than double my throughput from ~300mbit to at last 700 mbits/second.  Still wish I could squeeze out a little more from my 1 Gbps/symmetrical connection but this is highly helpful. 

Hmm, I may have spoken too soon as now I'm having trouble getting over 400 Mbits again.

[thowe]

I have an APU2c4. A very smart and rock solid solution. With the right settings and not too many services running, you can reach nearly 1GBit.

Currently, I have the problem that the frequency of the CPU sometimes hangs at 600MHZ and does not increase back to 1GHZ under load. This was a problem with a previous BIOS version and should actually be solved today. But maybe you should have a look at the frequency of your CPU.

Run Turbostat:

• Open shell (serial or ssh)
• su
• pkg add http://pkg0.isc.freebsd.org/FreeBSD:11:amd64/latest/All/turbostat-4.17_2.txz
• rehash
• kldload cpuctl
• turbostat --interval 3

Normally you see a busy frequency up to 1GHZ. If you are stuck you see 600MHz (599MHz).

Code: [Select]

turbostat version 17.06.23 - Len Brown <lenb@kernel.org>
CPUID(0): AuthenticAMD 13 CPUID levels; family:model:stepping 0xf:30:1 (15:48:1)
CPUID(1): SSE3 MONITOR - - - TSC MSR - -
CPUID(6): APERF, No-TURBO, No-DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, No-EPB
CPUID(7): No-SGX
NSFOD /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
-       -       69      11.58   599     998     218
0       0       109     18.13   599     998     17
1       1       53      8.82    599     998     138
2       2       56      9.36    599     998     23
3       3       60      10.03   599     998     40

The problem is documented here: https://github.com/pcengines/apu2-documentation/blob/master/docs/debug/cpu_frequency.md

This was the GitHub Issue before the fix (closed): https://github.com/pcengines/coreboot/issues/196

After e reboot the frequency is back to normal for some time - but after a few hours it's stuck at 600MHz again.

May be you are affected too?

[hushcoden]

I'm pretty sure with the latest firmware this has been fixed: have you tried the v4.13.0.2 ?

[thowe]

Thanks. I followed the recommendation to use the latest v4.11.x release. On production systems I am on the conservative side. But when you say that this version is fine for OPNsense and the problem is fixed, then I will give it a try.

[thowe]

v4.13.0.2 installed. So far frequency goes up to 1Ghz. I will report here, should the problem come at a later point. But currently this seems to be ok! Would be great. 

Thanks for the hint.

[rakete]

v4.13.0.2 installed. So far frequency goes up to 1Ghz. I will report here, should the problem come at a later point. But currently this seems to be ok! Would be great. 

Thanks for the hint.

And do you see any benefit in routing performance or LAN throughput?

[thowe]

I did not run specific benchmarks.

What I can say is that the CPU in my case idles more than before. So there should be more headroom for performance now.

But I think not all boards have been affected by the stuck low frequency issue. So I can not judge if anybody profits.

[thowe]

The problem is unfortunately not solved! 

After a few hours of operation, the problem with the limitation to a maximum of 600MHz unfortunately reappeared even with the latest firmware.

Hardware: APU2C4
BIOS: v4.13.0.2 (cold boot after flashrom)

The question is how the APU2 can be made to leave the economy mode of 600MHz again.

[Ricardo]

The problem is unfortunately not solved! 

After a few hours of operation, the problem with the limitation to a maximum of 600MHz unfortunately reappeared even with the latest firmware.

Hardware: APU2C4
BIOS: v4.13.0.2 (cold boot after flashrom)

The question is how the APU2 can be made to leave the economy mode of 600MHz again.

Better to re-open the ticket on github, and provide some troubleshooting results to 3mdeb.