Massive performance drop with 21.7 on APU2

[lynix]

Does anybody run 21.7 on an APU2 with proper throughput?

After upgrading from 21.1.9 to 21.7 (and now 21.7.1) I noticed my NAT throughput dropped to ~330 Mbit/s. Before the upgrade I had ~980 Mbit/s.

When setting up OPNsense (21.1.3 back then) I had followed this guide and set the following tunables:

Code: [Select]

net.inet.tcp.tso=1
net.inet.udp.checksum=1
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
legal.intel_igb.license_ack="1"
Could it be the case that these tunables have no effect with 21.7 anymore?

I also noticed that all hardware offloading options in the webinterface (Interfaces -> Settings) are disabled (all checkboxes are checked). I didn't know they were there, so maybe I should uncheck them instead of using the tunables?

An iperf3 run from a LAN client to the LAN interface of the OPNsense box shows:

Code: [Select]

$ iperf3 -c 10.0.20.1 -p 14218
Connecting to host 10.0.20.1, port 14218
[  5] local 10.0.20.10 port 35220 connected to 10.0.20.1 port 14218
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  26.8 MBytes   225 Mbits/sec    0   67.9 KBytes
[  5]   1.00-2.00   sec  26.6 MBytes   223 Mbits/sec    0   67.9 KBytes
[  5]   2.00-3.00   sec  26.7 MBytes   224 Mbits/sec    0   67.9 KBytes
[  5]   3.00-4.00   sec  26.6 MBytes   223 Mbits/sec    0   67.9 KBytes
[  5]   4.00-5.00   sec  26.4 MBytes   222 Mbits/sec    0   67.9 KBytes
[  5]   5.00-6.00   sec  26.7 MBytes   224 Mbits/sec    0   67.9 KBytes
[  5]   6.00-7.00   sec  26.5 MBytes   222 Mbits/sec    0   67.9 KBytes
[  5]   7.00-8.00   sec  26.6 MBytes   223 Mbits/sec    0   67.9 KBytes
[  5]   8.00-9.00   sec  26.0 MBytes   218 Mbits/sec    0   67.9 KBytes
[  5]   9.00-10.00  sec  26.3 MBytes   221 Mbits/sec    0   67.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   265 MBytes   222 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   265 MBytes   222 Mbits/sec                  receiver
Thanks in advance for any recommendations!

[luketheduke]

Does throughput look any better if you run the iPerf traffic through the firewall instead of making the APU2 initiate/terminate the traffic?

[lynix]

Does throughput look any better if you run the iPerf traffic through the firewall instead of making the APU2 initiate/terminate the traffic?

That's what I just finished testing   I've added another Linux box on the WAN side of the OPNsense box and used that as an iperf3 target from LAN.

So I have the following with one connection:

Code: [Select]

$ iperf3 -p 51337 -c 192.168.1.66
Connecting to host 192.168.1.66, port 51337
[  5] local 10.0.20.50 port 43764 connected to 192.168.1.66 port 51337
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  51.1 MBytes   429 Mbits/sec   64   1.12 MBytes
[  5]   1.00-2.00   sec  50.0 MBytes   420 Mbits/sec    0   1.24 MBytes
[  5]   2.00-3.00   sec  50.0 MBytes   419 Mbits/sec    0   1.34 MBytes
[  5]   3.00-4.00   sec  48.8 MBytes   409 Mbits/sec    0   1.41 MBytes
[  5]   4.00-5.00   sec  50.0 MBytes   419 Mbits/sec    2   1.03 MBytes
[  5]   5.00-6.00   sec  48.8 MBytes   409 Mbits/sec    0   1.10 MBytes
[  5]   6.00-7.00   sec  48.8 MBytes   409 Mbits/sec    0   1.16 MBytes
[  5]   7.00-8.00   sec  48.8 MBytes   409 Mbits/sec    0   1.20 MBytes
[  5]   8.00-9.00   sec  48.8 MBytes   409 Mbits/sec    0   1.22 MBytes
[  5]   9.00-10.00  sec  47.5 MBytes   398 Mbits/sec    0   1.23 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   492 MBytes   413 Mbits/sec   66             sender
[  5]   0.00-10.00  sec   490 MBytes   411 Mbits/sec                  receiver
~400 Mbit/s throughput including NAT, actually not bad. Still no gigabit but I maybe my memories are wrong and it has never been full speed with just one stream.

But interesting: does this mean that when running iperf3 on the APU2 its measurement is bottle-necked by CPU?

Using 4 connections I finally have my gigabit throughput with NAT:

Code: [Select]

$ iperf3 -p 51337 -c 192.168.1.66 -P 4
Connecting to host 192.168.1.66, port 51337
[  5] local 10.0.20.50 port 43776 connected to 192.168.1.66 port 51337
[  7] local 10.0.20.50 port 43778 connected to 192.168.1.66 port 51337
[  9] local 10.0.20.50 port 43780 connected to 192.168.1.66 port 51337
[ 11] local 10.0.20.50 port 43782 connected to 192.168.1.66 port 51337
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  25.9 MBytes   218 Mbits/sec   22   77.8 KBytes
[  7]   0.00-1.00   sec  26.8 MBytes   225 Mbits/sec    5    228 KBytes
[  9]   0.00-1.00   sec  36.7 MBytes   308 Mbits/sec   11    160 KBytes
[ 11]   0.00-1.00   sec  28.0 MBytes   235 Mbits/sec   13    122 KBytes
[SUM]   0.00-1.00   sec   117 MBytes   984 Mbits/sec   51
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  27.4 MBytes   230 Mbits/sec   17   80.6 KBytes
[  7]   1.00-2.00   sec  25.0 MBytes   210 Mbits/sec    3    148 KBytes
[  9]   1.00-2.00   sec  28.5 MBytes   239 Mbits/sec   19    133 KBytes
[ 11]   1.00-2.00   sec  30.9 MBytes   259 Mbits/sec    9    141 KBytes
[SUM]   1.00-2.00   sec   112 MBytes   938 Mbits/sec   48
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  27.8 MBytes   234 Mbits/sec   18   70.7 KBytes
[  7]   2.00-3.00   sec  24.9 MBytes   209 Mbits/sec    4    146 KBytes
[  9]   2.00-3.00   sec  28.5 MBytes   239 Mbits/sec   21   93.3 KBytes
[ 11]   2.00-3.00   sec  30.9 MBytes   259 Mbits/sec   14   89.1 KBytes
[SUM]   2.00-3.00   sec   112 MBytes   941 Mbits/sec   57
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  24.9 MBytes   209 Mbits/sec   21   91.9 KBytes
[  7]   3.00-4.00   sec  25.7 MBytes   215 Mbits/sec    1    173 KBytes
[  9]   3.00-4.00   sec  37.1 MBytes   311 Mbits/sec   12    144 KBytes
[ 11]   3.00-4.00   sec  24.4 MBytes   204 Mbits/sec   17    102 KBytes
[SUM]   3.00-4.00   sec   112 MBytes   940 Mbits/sec   51
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  28.3 MBytes   237 Mbits/sec   12    170 KBytes
[  7]   4.00-5.00   sec  24.8 MBytes   208 Mbits/sec   12    120 KBytes
[  9]   4.00-5.00   sec  33.9 MBytes   284 Mbits/sec   14    112 KBytes
[ 11]   4.00-5.00   sec  24.4 MBytes   204 Mbits/sec   16    100 KBytes
[SUM]   4.00-5.00   sec   111 MBytes   934 Mbits/sec   54
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  36.7 MBytes   308 Mbits/sec   15   86.3 KBytes
[  7]   5.00-6.00   sec  23.4 MBytes   196 Mbits/sec   12    116 KBytes
[  9]   5.00-6.00   sec  26.4 MBytes   222 Mbits/sec   15    167 KBytes
[ 11]   5.00-6.00   sec  25.4 MBytes   213 Mbits/sec   15   80.6 KBytes
[SUM]   5.00-6.00   sec   112 MBytes   938 Mbits/sec   57
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  27.0 MBytes   226 Mbits/sec   15    123 KBytes
[  7]   6.00-7.00   sec  25.1 MBytes   211 Mbits/sec    3    154 KBytes
[  9]   6.00-7.00   sec  35.0 MBytes   294 Mbits/sec   21   74.9 KBytes
[ 11]   6.00-7.00   sec  24.9 MBytes   209 Mbits/sec   15    112 KBytes
[SUM]   6.00-7.00   sec   112 MBytes   939 Mbits/sec   54
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  34.0 MBytes   285 Mbits/sec   11    115 KBytes
[  7]   7.00-8.00   sec  24.9 MBytes   209 Mbits/sec    6   94.7 KBytes
[  9]   7.00-8.00   sec  28.5 MBytes   239 Mbits/sec   18   96.2 KBytes
[ 11]   7.00-8.00   sec  25.2 MBytes   212 Mbits/sec   19   67.9 KBytes
[SUM]   7.00-8.00   sec   113 MBytes   945 Mbits/sec   54
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  32.3 MBytes   271 Mbits/sec   17   74.9 KBytes
[  7]   8.00-9.00   sec  22.9 MBytes   192 Mbits/sec   12   86.3 KBytes
[  9]   8.00-9.00   sec  29.6 MBytes   248 Mbits/sec   18    127 KBytes
[ 11]   8.00-9.00   sec  26.5 MBytes   223 Mbits/sec   10    126 KBytes
[SUM]   8.00-9.00   sec   111 MBytes   933 Mbits/sec   57
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  28.7 MBytes   241 Mbits/sec   15   76.4 KBytes
[  7]   9.00-10.00  sec  24.3 MBytes   204 Mbits/sec    9    127 KBytes
[  9]   9.00-10.00  sec  35.0 MBytes   293 Mbits/sec   15    100 KBytes
[ 11]   9.00-10.00  sec  24.4 MBytes   204 Mbits/sec   15    140 KBytes
[SUM]   9.00-10.00  sec   112 MBytes   942 Mbits/sec   54
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   293 MBytes   246 Mbits/sec  163             sender
[  5]   0.00-10.00  sec   292 MBytes   245 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   248 MBytes   208 Mbits/sec   67             sender
[  7]   0.00-10.00  sec   246 MBytes   207 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   319 MBytes   268 Mbits/sec  164             sender
[  9]   0.00-10.00  sec   317 MBytes   266 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   265 MBytes   222 Mbits/sec  143             sender
[ 11]   0.00-10.00  sec   264 MBytes   221 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec  537             sender
[SUM]   0.00-10.00  sec  1.09 GBytes   938 Mbits/sec                  receiver
Interesting side note: I unchecked all "disable offloading" boxes and re-ran all tests with hardware offloading enabled. Didn't make any difference, neither in terms of throughput nor in load.

[Gary7]

With 21.7, some of the tunables have changed.

For instance, "hw.igb" doesn't exist anymore. Look at "hw.em"

Replace hw.igb.rx_process_limit="-1" with hw.em.rx_process_limit="-1"
Neither hw.igb.tx_process_limit or hw.em.tx_process_limit exist.

There are a lot of values for "dev.igb"

Unfortunately, you will need to check all your configured tunables to verify that the names haven't changed.
At the console, enter "sysctl tunable-name" to verify the name and value.

[lynix]

With 21.7, some of the tunables have changed.

For instance, "hw.igb" doesn't exist anymore. Look at "hw.em"

Ah thanks, that's exactly what I have been asking myself.

Unfortunately, you will need to check all your configured tunables to verify that the names haven't changed.
At the console, enter "sysctl tunable-name" to verify the name and value.

Will I only need to check the ones I have added manually after the 21.1.x installation or do I need to check all of them, even those that I have never touched and left at default value?

[Gary7]

All names of default tunables should be fine.

I would just look at the tunables that you've added.

In my attempts to improve throughput, I've added several tunables.
My internet speed is 200 Mbps (236 Mbps actual). I get full speed with no problem.
Maybe, I would get full 200 Mbps speed without any tuning.
If I ever go to Gigabit speed, I might need to upgrade from my APU2.

[lynix]

Okay I've gone through the list. It seems the only applicable tunable is the one you already mentioned:

Code: [Select]

hw.em.rx_process_limit="-1"
Unfortunately it didn't change anything, still only 300-400 Mbit/s NAT throughput to WAN.

My ISP contract gives me ~600 Mbit/s so I really would like to see that with a single connection... So if anyone else has performance metrics for an APU2 with 21.7, I'm all ears.

In the meantime I will probably scratch my head trying to remember if I really had more throughput on 21.1.9 on one connection. And I will jealously look at OpenWRT (or Linux in general) which can pump through 1 Gbit/s on the APU2 without breaking sweat 

[Patrick M. Hausen]

Are you using a bridge interface to turn your apu into a small switch? Or just one LAN, one WAN?

[lynix]

Are you using a bridge interface to turn your apu into a small switch? Or just one LAN, one WAN?

No bridges and no fancy things with WAN, but for the LAN side I'm using 802.1Q VLANs to split the physical interface into 4 different 'trust zones'. Could that be the issue?

I've read that upstream BSD has merged two drivers for the Intel NICs, so maybe VLAN filter offloading doesn't work anymore?

[Patrick M. Hausen]

The general recommendation is to disable VLAN tag offloading for routers/firewalls and only use it for hosts that are an endpoint with respect to their communication.

[lynix]

You mean 'VLAN Hardware Filtering'? That's currently disabled so it should be fine.