PC Engines APU2 1Gbit traffic not achievable

[rmayr]

Indeed. This happens not only for LAN->WAN traffic, but also for traffic between two different internal (e.g. LAN and DMZ) segments with no NAT involved and only directly connected routes in use. I have not yet tried with VTI instead of policy based IPsec, but this issue may make OpnSense a non-starter for the intended production use at our university institute (that is the reason why I am now spending far too much time putting OpnSense through such tests).

You really want to run a university institute in production with a APU device??  :o

No, not on an APU - this is my test device to find some of the issues in parallel to a VM installation (which seems to have the same performance issues, actually). We would only put it in production on a faster hardware, but don't expect such bottlenecks to necessarily change. We are aiming for at least 2-3, better 5Gbps throughput between some of the segments, and definitely need IPsec and flow analysis and would like (but don't necessarily require) IDS/IPS on. Given our current experience, I am not sure how likely that is.

[Patrick M. Hausen]

Get a Deciso DEC38xx and you will definitely be able to match that requirement.

[rmayr]

Further tests on an 8-core ProxMox VM server with 4 cores assigned to a OpnSense test instance shows 1.6 Gbps throughput limit with the CPU not fully loaded (only 2 out of 4 cores in the VM being used). Putting traffic flow analysis and Surricata into the mix, I am not sure how a hardware like the one sold by Decisio would reach 5 Gbps with the current OpnSense version. What is the big difference we are missing?

[Ricardo]

The best solution is to get a written(!) assurance from Deciso, what traffic their hardware can do. That way you can demand the promised performance for your money, if it turns out thweir hardware underperforms. Otherwise any vendor on the planet can say literally anything they are not shy to say. As you cant depend on generic  marketing PDFs.

[rmayr]

The concerning bit is the heavy side effect of having IPsec enabled for completely unrelated traffic. It points to a general performance bottleneck in the kernel.

[rmayr]

Now with more detailed measurements: https://www.mayrhofer.eu.org/post/firewall-throughput-opnsense-openwrt/

[ProServ]

For my experience, with all my APU (4D4, 4C4, ...), I've 850Mb/s only if I use several ethernet packets simultaneously (option -P 2 or -P 4 with iperf)
Otherwise, performance is critical (450/500Mb/s) with 1 packet (iperf default mode test).

I've all upgraded... OPNSense 21.7.5, Coreboot 4.14.0.6
And I've added this parameters :
- hw.igb.rx_process_limit="-1"
- hw.igb.tx_process_limit="-1"
- legal.intel_igb.license_ack="1"
- net.inet.tcp.tso="1"
- net.inet.udp.checksum="1"
- hint.p4tcc.0.disabled=1
- hint.acpi_throttle.0.disabled=1
- hint.acpi_perf.0.disabled=1

[cookiemonster]

@ProServ thanks for sharing this.
Have you had to opportunity to retest with 21.7.5? I ask because unless I'm mistaken some of those tunables have gone from the kernel. So if the testing shows the same results then it would be intersting to see if they remain after commenting them out.
For instance I can't find sysctl tunables for hw.igb.rx_process_limit or hw.igb.{number}.rx_process.
The same for hint.p4tcc.0.disabled

[iam]

Has someone else tested 22.1 already. I'm now getting much higher speed values on PPPoE WAN:

https://forum.opnsense.org/index.php?topic=26162.msg128661#msg128661

[fireburner]

Most of the suggested tunables are not supported any more in 22.1.
I have however not yet tested network performance in 22.1 yet.

Tunables dev.igb.0.fc, dev.igb.1.fc,... are still shown as valid tunables.

These are shown as unsupported:
dev.igb.0.eee_disabled, dev.igb.1.eee_disabled, ...
hint.acpi_perf.0.disabled
hint.acpi_throttle.0.disabled
hint.p4tcc.0.disabled
hw.igb.0.fc, hw.igb.1.fc, ...
hw.igb.num_queues
hw.igb.rx_process_limit
hw.igb.tx_process_limit
legal.intel_igb.license_ack


I have removed the flow control tunables, as the network speed was minimally faster.
I got 350/210 MBit/s with ipferf in both directions (including the use of vlans) and IDS/IPS off.
One way got me 390 MBit/s.

With IDS (no IPS, because that is broken for me in 22.1):
one way: 300 MBit/s

[Northguy]

On LAN side 22.1 APU2D4, Gigabit network. All non-functional tunables removed as mentioned by @fireburner, no IDS/IPS.

I recall having measured higher values on 21.x (~800 - 900 MBit/s)

Code Select Expand

--------@DiskStation:/$ iperf3 -c 192.168.1.1 -p 19160 -P 30 -4 -R
Connecting to host 192.168.1.1, port 19160
Reverse mode, remote host 192.168.1.1 is sending
[  5] local 192.168.1.10 port 43300 connected to 192.168.1.1 port 19160
[  7] local 192.168.1.10 port 43302 connected to 192.168.1.1 port 19160
[  9] local 192.168.1.10 port 43304 connected to 192.168.1.1 port 19160
[ 11] local 192.168.1.10 port 43310 connected to 192.168.1.1 port 19160
[ 13] local 192.168.1.10 port 43312 connected to 192.168.1.1 port 19160
[ 15] local 192.168.1.10 port 43314 connected to 192.168.1.1 port 19160
[ 17] local 192.168.1.10 port 43316 connected to 192.168.1.1 port 19160
[ 19] local 192.168.1.10 port 43318 connected to 192.168.1.1 port 19160
[ 21] local 192.168.1.10 port 43320 connected to 192.168.1.1 port 19160
[ 23] local 192.168.1.10 port 43322 connected to 192.168.1.1 port 19160
[ 25] local 192.168.1.10 port 43324 connected to 192.168.1.1 port 19160
[ 27] local 192.168.1.10 port 43326 connected to 192.168.1.1 port 19160
[ 29] local 192.168.1.10 port 43328 connected to 192.168.1.1 port 19160
[ 31] local 192.168.1.10 port 43330 connected to 192.168.1.1 port 19160
[ 33] local 192.168.1.10 port 43332 connected to 192.168.1.1 port 19160
[ 35] local 192.168.1.10 port 43334 connected to 192.168.1.1 port 19160
[ 37] local 192.168.1.10 port 43336 connected to 192.168.1.1 port 19160
[ 39] local 192.168.1.10 port 43338 connected to 192.168.1.1 port 19160
[ 41] local 192.168.1.10 port 43344 connected to 192.168.1.1 port 19160
[ 43] local 192.168.1.10 port 43346 connected to 192.168.1.1 port 19160
[ 45] local 192.168.1.10 port 43352 connected to 192.168.1.1 port 19160
[ 47] local 192.168.1.10 port 43354 connected to 192.168.1.1 port 19160
[ 49] local 192.168.1.10 port 43356 connected to 192.168.1.1 port 19160
[ 51] local 192.168.1.10 port 43358 connected to 192.168.1.1 port 19160
[ 53] local 192.168.1.10 port 43360 connected to 192.168.1.1 port 19160
[ 55] local 192.168.1.10 port 43362 connected to 192.168.1.1 port 19160
[ 57] local 192.168.1.10 port 43364 connected to 192.168.1.1 port 19160
[ 59] local 192.168.1.10 port 43366 connected to 192.168.1.1 port 19160
[ 61] local 192.168.1.10 port 43368 connected to 192.168.1.1 port 19160
[ 63] local 192.168.1.10 port 43370 connected to 192.168.1.1 port 19160

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.66  sec  19.6 MBytes  15.4 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  18.9 MBytes  15.8 Mbits/sec                  receiver
[  7]   0.00-10.66  sec  18.0 MBytes  14.2 Mbits/sec    0             sender
[  7]   0.00-10.00  sec  17.2 MBytes  14.5 Mbits/sec                  receiver
[  9]   0.00-10.66  sec  21.6 MBytes  17.0 Mbits/sec    0             sender
[  9]   0.00-10.00  sec  20.9 MBytes  17.5 Mbits/sec                  receiver
[ 11]   0.00-10.66  sec  20.1 MBytes  15.8 Mbits/sec    0             sender
[ 11]   0.00-10.00  sec  19.4 MBytes  16.3 Mbits/sec                  receiver
[ 13]   0.00-10.66  sec  20.1 MBytes  15.8 Mbits/sec    0             sender
[ 13]   0.00-10.00  sec  19.4 MBytes  16.3 Mbits/sec                  receiver
[ 15]   0.00-10.66  sec  22.9 MBytes  18.0 Mbits/sec    0             sender
[ 15]   0.00-10.00  sec  22.1 MBytes  18.6 Mbits/sec                  receiver
[ 17]   0.00-10.66  sec  19.4 MBytes  15.2 Mbits/sec    0             sender
[ 17]   0.00-10.00  sec  18.6 MBytes  15.6 Mbits/sec                  receiver
[ 19]   0.00-10.66  sec  20.0 MBytes  15.7 Mbits/sec    0             sender
[ 19]   0.00-10.00  sec  19.1 MBytes  16.1 Mbits/sec                  receiver
[ 21]   0.00-10.66  sec  22.8 MBytes  17.9 Mbits/sec    0             sender
[ 21]   0.00-10.00  sec  21.9 MBytes  18.3 Mbits/sec                  receiver
[ 23]   0.00-10.66  sec  20.8 MBytes  16.3 Mbits/sec    0             sender
[ 23]   0.00-10.00  sec  19.9 MBytes  16.7 Mbits/sec                  receiver
[ 25]   0.00-10.66  sec  20.0 MBytes  15.7 Mbits/sec    0             sender
[ 25]   0.00-10.00  sec  19.1 MBytes  16.0 Mbits/sec                  receiver
[ 27]   0.00-10.66  sec  18.5 MBytes  14.6 Mbits/sec    0             sender
[ 27]   0.00-10.00  sec  17.6 MBytes  14.8 Mbits/sec                  receiver
[ 29]   0.00-10.66  sec  18.8 MBytes  14.8 Mbits/sec    0             sender
[ 29]   0.00-10.00  sec  17.9 MBytes  15.0 Mbits/sec                  receiver
[ 31]   0.00-10.66  sec  16.6 MBytes  13.1 Mbits/sec    0             sender
[ 31]   0.00-10.00  sec  15.8 MBytes  13.2 Mbits/sec                  receiver
[ 33]   0.00-10.66  sec  17.0 MBytes  13.4 Mbits/sec    0             sender
[ 33]   0.00-10.00  sec  16.1 MBytes  13.5 Mbits/sec                  receiver
[ 35]   0.00-10.66  sec  17.6 MBytes  13.9 Mbits/sec    0             sender
[ 35]   0.00-10.00  sec  16.8 MBytes  14.1 Mbits/sec                  receiver
[ 37]   0.00-10.66  sec  18.9 MBytes  14.9 Mbits/sec    0             sender
[ 37]   0.00-10.00  sec  18.0 MBytes  15.1 Mbits/sec                  receiver
[ 39]   0.00-10.66  sec  17.8 MBytes  14.0 Mbits/sec    0             sender
[ 39]   0.00-10.00  sec  16.9 MBytes  14.2 Mbits/sec                  receiver
[ 41]   0.00-10.66  sec  20.0 MBytes  15.7 Mbits/sec    0             sender
[ 41]   0.00-10.00  sec  19.1 MBytes  16.0 Mbits/sec                  receiver
[ 43]   0.00-10.66  sec  21.9 MBytes  17.2 Mbits/sec    0             sender
[ 43]   0.00-10.00  sec  21.0 MBytes  17.6 Mbits/sec                  receiver
[ 45]   0.00-10.66  sec  20.8 MBytes  16.3 Mbits/sec    0             sender
[ 45]   0.00-10.00  sec  19.9 MBytes  16.7 Mbits/sec                  receiver
[ 47]   0.00-10.66  sec  16.2 MBytes  12.8 Mbits/sec    0             sender
[ 47]   0.00-10.00  sec  15.4 MBytes  12.9 Mbits/sec                  receiver
[ 49]   0.00-10.66  sec  19.0 MBytes  15.0 Mbits/sec    0             sender
[ 49]   0.00-10.00  sec  18.1 MBytes  15.2 Mbits/sec                  receiver
[ 51]   0.00-10.66  sec  21.5 MBytes  16.9 Mbits/sec    0             sender
[ 51]   0.00-10.00  sec  20.6 MBytes  17.3 Mbits/sec                  receiver
[ 53]   0.00-10.66  sec  16.8 MBytes  13.2 Mbits/sec    0             sender
[ 53]   0.00-10.00  sec  15.9 MBytes  13.3 Mbits/sec                  receiver
[ 55]   0.00-10.66  sec  15.6 MBytes  12.3 Mbits/sec    0             sender
[ 55]   0.00-10.00  sec  14.8 MBytes  12.4 Mbits/sec                  receiver
[ 57]   0.00-10.66  sec  17.6 MBytes  13.9 Mbits/sec    0             sender
[ 57]   0.00-10.00  sec  16.8 MBytes  14.1 Mbits/sec                  receiver
[ 59]   0.00-10.66  sec  16.1 MBytes  12.7 Mbits/sec    0             sender
[ 59]   0.00-10.00  sec  15.2 MBytes  12.8 Mbits/sec                  receiver
[ 61]   0.00-10.66  sec  15.0 MBytes  11.8 Mbits/sec    1             sender
[ 61]   0.00-10.00  sec  14.1 MBytes  11.8 Mbits/sec                  receiver
[ 63]   0.00-10.66  sec  13.5 MBytes  10.6 Mbits/sec    0             sender
[ 63]   0.00-10.00  sec  12.6 MBytes  10.6 Mbits/sec                  receiver
[SUM]   0.00-10.66  sec   564 MBytes   444 Mbits/sec    1             sender
[SUM]   0.00-10.00  sec   539 MBytes   452 Mbits/sec                  receiver

iperf Done.
--------@DiskStation:/$ iperf3 -c 192.168.1.1 -p 3958 -P 30 -4
Connecting to host 192.168.1.1, port 3958
[  5] local 192.168.1.10 port 50816 connected to 192.168.1.1 port 3958
[  7] local 192.168.1.10 port 50818 connected to 192.168.1.1 port 3958
[  9] local 192.168.1.10 port 50820 connected to 192.168.1.1 port 3958
[ 11] local 192.168.1.10 port 50822 connected to 192.168.1.1 port 3958
[ 13] local 192.168.1.10 port 50824 connected to 192.168.1.1 port 3958
[ 15] local 192.168.1.10 port 50826 connected to 192.168.1.1 port 3958
[ 17] local 192.168.1.10 port 50828 connected to 192.168.1.1 port 3958
[ 19] local 192.168.1.10 port 50830 connected to 192.168.1.1 port 3958
[ 21] local 192.168.1.10 port 50836 connected to 192.168.1.1 port 3958
[ 23] local 192.168.1.10 port 50838 connected to 192.168.1.1 port 3958
[ 25] local 192.168.1.10 port 50840 connected to 192.168.1.1 port 3958
[ 27] local 192.168.1.10 port 50842 connected to 192.168.1.1 port 3958
[ 29] local 192.168.1.10 port 50844 connected to 192.168.1.1 port 3958
[ 31] local 192.168.1.10 port 50846 connected to 192.168.1.1 port 3958
[ 33] local 192.168.1.10 port 50848 connected to 192.168.1.1 port 3958
[ 35] local 192.168.1.10 port 50850 connected to 192.168.1.1 port 3958
[ 37] local 192.168.1.10 port 50852 connected to 192.168.1.1 port 3958
[ 39] local 192.168.1.10 port 50854 connected to 192.168.1.1 port 3958
[ 41] local 192.168.1.10 port 50856 connected to 192.168.1.1 port 3958
[ 43] local 192.168.1.10 port 50858 connected to 192.168.1.1 port 3958
[ 45] local 192.168.1.10 port 50860 connected to 192.168.1.1 port 3958
[ 47] local 192.168.1.10 port 50862 connected to 192.168.1.1 port 3958
[ 49] local 192.168.1.10 port 50864 connected to 192.168.1.1 port 3958
[ 51] local 192.168.1.10 port 50866 connected to 192.168.1.1 port 3958
[ 53] local 192.168.1.10 port 50868 connected to 192.168.1.1 port 3958
[ 55] local 192.168.1.10 port 50870 connected to 192.168.1.1 port 3958
[ 57] local 192.168.1.10 port 50872 connected to 192.168.1.1 port 3958
[ 59] local 192.168.1.10 port 50874 connected to 192.168.1.1 port 3958
[ 61] local 192.168.1.10 port 50876 connected to 192.168.1.1 port 3958
[ 63] local 192.168.1.10 port 50878 connected to 192.168.1.1 port 3958

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  15.3 MBytes  12.8 Mbits/sec    0             sender
[  5]   0.00-10.11  sec  15.2 MBytes  12.6 Mbits/sec                  receiver
[  7]   0.00-10.00  sec  15.4 MBytes  12.9 Mbits/sec    0             sender
[  7]   0.00-10.11  sec  15.3 MBytes  12.7 Mbits/sec                  receiver
[  9]   0.00-10.00  sec  22.7 MBytes  19.0 Mbits/sec    0             sender
[  9]   0.00-10.11  sec  22.5 MBytes  18.6 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec  15.2 MBytes  12.7 Mbits/sec    0             sender
[ 11]   0.00-10.11  sec  15.1 MBytes  12.5 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec  15.4 MBytes  12.9 Mbits/sec    0             sender
[ 13]   0.00-10.11  sec  15.3 MBytes  12.7 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec  15.3 MBytes  12.8 Mbits/sec    0             sender
[ 15]   0.00-10.11  sec  15.2 MBytes  12.6 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec  15.5 MBytes  13.0 Mbits/sec    0             sender
[ 17]   0.00-10.11  sec  15.4 MBytes  12.8 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec  15.7 MBytes  13.2 Mbits/sec    0             sender
[ 19]   0.00-10.11  sec  15.6 MBytes  13.0 Mbits/sec                  receiver
[ 21]   0.00-10.00  sec  15.5 MBytes  13.0 Mbits/sec    0             sender
[ 21]   0.00-10.11  sec  15.4 MBytes  12.8 Mbits/sec                  receiver
[ 23]   0.00-10.00  sec  15.0 MBytes  12.6 Mbits/sec    1             sender
[ 23]   0.00-10.11  sec  14.9 MBytes  12.4 Mbits/sec                  receiver
[ 25]   0.00-10.00  sec  15.4 MBytes  12.9 Mbits/sec    0             sender
[ 25]   0.00-10.11  sec  15.3 MBytes  12.7 Mbits/sec                  receiver
[ 27]   0.00-10.00  sec  15.2 MBytes  12.7 Mbits/sec    0             sender
[ 27]   0.00-10.11  sec  15.1 MBytes  12.5 Mbits/sec                  receiver
[ 29]   0.00-10.00  sec  15.4 MBytes  12.9 Mbits/sec    0             sender
[ 29]   0.00-10.11  sec  15.3 MBytes  12.7 Mbits/sec                  receiver
[ 31]   0.00-10.00  sec  22.7 MBytes  19.0 Mbits/sec    1             sender
[ 31]   0.00-10.11  sec  22.6 MBytes  18.7 Mbits/sec                  receiver
[ 33]   0.00-10.00  sec  34.2 MBytes  28.7 Mbits/sec    0             sender
[ 33]   0.00-10.11  sec  33.9 MBytes  28.1 Mbits/sec                  receiver
[ 35]   0.00-10.00  sec  15.2 MBytes  12.8 Mbits/sec    0             sender
[ 35]   0.00-10.11  sec  15.2 MBytes  12.6 Mbits/sec                  receiver
[ 37]   0.00-10.00  sec  23.2 MBytes  19.4 Mbits/sec    0             sender
[ 37]   0.00-10.11  sec  23.0 MBytes  19.1 Mbits/sec                  receiver
[ 39]   0.00-10.00  sec  15.3 MBytes  12.8 Mbits/sec    0             sender
[ 39]   0.00-10.11  sec  15.1 MBytes  12.6 Mbits/sec                  receiver
[ 41]   0.00-10.00  sec  15.5 MBytes  13.0 Mbits/sec    0             sender
[ 41]   0.00-10.11  sec  15.4 MBytes  12.8 Mbits/sec                  receiver
[ 43]   0.00-10.00  sec  17.3 MBytes  14.5 Mbits/sec    0             sender
[ 43]   0.00-10.11  sec  17.0 MBytes  14.1 Mbits/sec                  receiver
[ 45]   0.00-10.00  sec  15.1 MBytes  12.7 Mbits/sec    0             sender
[ 45]   0.00-10.11  sec  15.0 MBytes  12.5 Mbits/sec                  receiver
[ 47]   0.00-10.00  sec  15.2 MBytes  12.8 Mbits/sec    0             sender
[ 47]   0.00-10.11  sec  15.1 MBytes  12.6 Mbits/sec                  receiver
[ 49]   0.00-10.00  sec  15.3 MBytes  12.8 Mbits/sec    0             sender
[ 49]   0.00-10.11  sec  15.1 MBytes  12.6 Mbits/sec                  receiver
[ 51]   0.00-10.00  sec  15.1 MBytes  12.7 Mbits/sec    0             sender
[ 51]   0.00-10.11  sec  15.0 MBytes  12.5 Mbits/sec                  receiver
[ 53]   0.00-10.00  sec  15.4 MBytes  12.9 Mbits/sec    0             sender
[ 53]   0.00-10.11  sec  15.3 MBytes  12.7 Mbits/sec                  receiver
[ 55]   0.00-10.00  sec  15.4 MBytes  12.9 Mbits/sec    0             sender
[ 55]   0.00-10.11  sec  15.3 MBytes  12.7 Mbits/sec                  receiver
[ 57]   0.00-10.00  sec  15.1 MBytes  12.7 Mbits/sec    0             sender
[ 57]   0.00-10.11  sec  15.0 MBytes  12.5 Mbits/sec                  receiver
[ 59]   0.00-10.00  sec  23.0 MBytes  19.3 Mbits/sec    0             sender
[ 59]   0.00-10.11  sec  22.9 MBytes  19.0 Mbits/sec                  receiver
[ 61]   0.00-10.00  sec  22.2 MBytes  18.6 Mbits/sec    0             sender
[ 61]   0.00-10.11  sec  21.8 MBytes  18.1 Mbits/sec                  receiver
[ 63]   0.00-10.00  sec  22.5 MBytes  18.9 Mbits/sec    0             sender
[ 63]   0.00-10.11  sec  22.4 MBytes  18.6 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec   525 MBytes   440 Mbits/sec    2             sender
[SUM]   0.00-10.11  sec   521 MBytes   432 Mbits/sec                  receiver

iperf Done.


[hibby50]

Hey all,

I wanted to resurrect this thread. I'm a new convert to opnsense and I'm really impressed. Though I am having issues with single connection performance on my apu2e4 (i210 nic).

I know this horse has been beaten to death, the problem I'm having is a lot of the tunables that get posted around are not up to date, and I haven't seen anything for newer BSD versions. I have no problem getting gigabit with multiple streams in iperf, but singe stream tops out around 400mbps. I know the hardware is capable of it since it worked fine on linux (again I know, dead horse)

I'm more interested in understanding the technical reason behind the limitation, if there are tunables/settings in opnsense 22 that can improve the performance, and is there an upstream bug/effort to improve this.

[fireburner]

Does anyone know if the upcoming FreeBSD 13.1 (coming in OPNsense 22.7) brings some single core network improvements?

[Patrick M. Hausen]

OPNsense 22.1 is already running on FreeBSD 13-STABLE. That's closer to 13.1 than to 13.0. There won't be huge changes AFAIK.

See the FreeBSD release model for reference. -STABLE is a moving target that gets continuously updated independent of tagged release versions. The picture is slightly outdated, since 12.3 and 13.1 are the current release versions at the moment.

[franco]

It's easy to test 13.1 now, but no high hopes as Patrick explained... https://forum.opnsense.org/index.php?topic=28505.0


Cheers,
Franco