OPNsense Forum
Archive => 21.7 Legacy Series => Topic started by: lynix on August 05, 2021, 02:06:25 pm
-
Does anybody run 21.7 on an APU2 with proper throughput?
After upgrading from 21.1.9 to 21.7 (and now 21.7.1) I noticed my NAT throughput dropped to ~330 Mbit/s. Before the upgrade I had ~980 Mbit/s.
When setting up OPNsense (21.1.3 back then) I had followed this guide (https://teklager.se/en/knowledge-base/opnsense-performance-optimization/) and set the following tunables:
net.inet.tcp.tso=1
net.inet.udp.checksum=1
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
legal.intel_igb.license_ack="1"
Could it be the case that these tunables have no effect with 21.7 anymore?
I also noticed that all hardware offloading options in the webinterface (Interfaces -> Settings) are disabled (all checkboxes are checked). I didn't know they were there, so maybe I should uncheck them instead of using the tunables?
An iperf3 run from a LAN client to the LAN interface of the OPNsense box shows:
$ iperf3 -c 10.0.20.1 -p 14218
Connecting to host 10.0.20.1, port 14218
[ 5] local 10.0.20.10 port 35220 connected to 10.0.20.1 port 14218
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 26.8 MBytes 225 Mbits/sec 0 67.9 KBytes
[ 5] 1.00-2.00 sec 26.6 MBytes 223 Mbits/sec 0 67.9 KBytes
[ 5] 2.00-3.00 sec 26.7 MBytes 224 Mbits/sec 0 67.9 KBytes
[ 5] 3.00-4.00 sec 26.6 MBytes 223 Mbits/sec 0 67.9 KBytes
[ 5] 4.00-5.00 sec 26.4 MBytes 222 Mbits/sec 0 67.9 KBytes
[ 5] 5.00-6.00 sec 26.7 MBytes 224 Mbits/sec 0 67.9 KBytes
[ 5] 6.00-7.00 sec 26.5 MBytes 222 Mbits/sec 0 67.9 KBytes
[ 5] 7.00-8.00 sec 26.6 MBytes 223 Mbits/sec 0 67.9 KBytes
[ 5] 8.00-9.00 sec 26.0 MBytes 218 Mbits/sec 0 67.9 KBytes
[ 5] 9.00-10.00 sec 26.3 MBytes 221 Mbits/sec 0 67.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 265 MBytes 222 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 265 MBytes 222 Mbits/sec receiver
Thanks in advance for any recommendations!
-
Does throughput look any better if you run the iPerf traffic through the firewall instead of making the APU2 initiate/terminate the traffic?
-
Does throughput look any better if you run the iPerf traffic through the firewall instead of making the APU2 initiate/terminate the traffic?
That's what I just finished testing :) I've added another Linux box on the WAN side of the OPNsense box and used that as an iperf3 target from LAN.
So I have the following with one connection:
$ iperf3 -p 51337 -c 192.168.1.66
Connecting to host 192.168.1.66, port 51337
[ 5] local 10.0.20.50 port 43764 connected to 192.168.1.66 port 51337
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 51.1 MBytes 429 Mbits/sec 64 1.12 MBytes
[ 5] 1.00-2.00 sec 50.0 MBytes 420 Mbits/sec 0 1.24 MBytes
[ 5] 2.00-3.00 sec 50.0 MBytes 419 Mbits/sec 0 1.34 MBytes
[ 5] 3.00-4.00 sec 48.8 MBytes 409 Mbits/sec 0 1.41 MBytes
[ 5] 4.00-5.00 sec 50.0 MBytes 419 Mbits/sec 2 1.03 MBytes
[ 5] 5.00-6.00 sec 48.8 MBytes 409 Mbits/sec 0 1.10 MBytes
[ 5] 6.00-7.00 sec 48.8 MBytes 409 Mbits/sec 0 1.16 MBytes
[ 5] 7.00-8.00 sec 48.8 MBytes 409 Mbits/sec 0 1.20 MBytes
[ 5] 8.00-9.00 sec 48.8 MBytes 409 Mbits/sec 0 1.22 MBytes
[ 5] 9.00-10.00 sec 47.5 MBytes 398 Mbits/sec 0 1.23 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 492 MBytes 413 Mbits/sec 66 sender
[ 5] 0.00-10.00 sec 490 MBytes 411 Mbits/sec receiver
~400 Mbit/s throughput including NAT, actually not bad. Still no gigabit but I maybe my memories are wrong and it has never been full speed with just one stream.
But interesting: does this mean that when running iperf3 on the APU2 its measurement is bottle-necked by CPU?
Using 4 connections I finally have my gigabit throughput with NAT:
$ iperf3 -p 51337 -c 192.168.1.66 -P 4
Connecting to host 192.168.1.66, port 51337
[ 5] local 10.0.20.50 port 43776 connected to 192.168.1.66 port 51337
[ 7] local 10.0.20.50 port 43778 connected to 192.168.1.66 port 51337
[ 9] local 10.0.20.50 port 43780 connected to 192.168.1.66 port 51337
[ 11] local 10.0.20.50 port 43782 connected to 192.168.1.66 port 51337
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 25.9 MBytes 218 Mbits/sec 22 77.8 KBytes
[ 7] 0.00-1.00 sec 26.8 MBytes 225 Mbits/sec 5 228 KBytes
[ 9] 0.00-1.00 sec 36.7 MBytes 308 Mbits/sec 11 160 KBytes
[ 11] 0.00-1.00 sec 28.0 MBytes 235 Mbits/sec 13 122 KBytes
[SUM] 0.00-1.00 sec 117 MBytes 984 Mbits/sec 51
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 27.4 MBytes 230 Mbits/sec 17 80.6 KBytes
[ 7] 1.00-2.00 sec 25.0 MBytes 210 Mbits/sec 3 148 KBytes
[ 9] 1.00-2.00 sec 28.5 MBytes 239 Mbits/sec 19 133 KBytes
[ 11] 1.00-2.00 sec 30.9 MBytes 259 Mbits/sec 9 141 KBytes
[SUM] 1.00-2.00 sec 112 MBytes 938 Mbits/sec 48
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 27.8 MBytes 234 Mbits/sec 18 70.7 KBytes
[ 7] 2.00-3.00 sec 24.9 MBytes 209 Mbits/sec 4 146 KBytes
[ 9] 2.00-3.00 sec 28.5 MBytes 239 Mbits/sec 21 93.3 KBytes
[ 11] 2.00-3.00 sec 30.9 MBytes 259 Mbits/sec 14 89.1 KBytes
[SUM] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 57
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 24.9 MBytes 209 Mbits/sec 21 91.9 KBytes
[ 7] 3.00-4.00 sec 25.7 MBytes 215 Mbits/sec 1 173 KBytes
[ 9] 3.00-4.00 sec 37.1 MBytes 311 Mbits/sec 12 144 KBytes
[ 11] 3.00-4.00 sec 24.4 MBytes 204 Mbits/sec 17 102 KBytes
[SUM] 3.00-4.00 sec 112 MBytes 940 Mbits/sec 51
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 28.3 MBytes 237 Mbits/sec 12 170 KBytes
[ 7] 4.00-5.00 sec 24.8 MBytes 208 Mbits/sec 12 120 KBytes
[ 9] 4.00-5.00 sec 33.9 MBytes 284 Mbits/sec 14 112 KBytes
[ 11] 4.00-5.00 sec 24.4 MBytes 204 Mbits/sec 16 100 KBytes
[SUM] 4.00-5.00 sec 111 MBytes 934 Mbits/sec 54
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 36.7 MBytes 308 Mbits/sec 15 86.3 KBytes
[ 7] 5.00-6.00 sec 23.4 MBytes 196 Mbits/sec 12 116 KBytes
[ 9] 5.00-6.00 sec 26.4 MBytes 222 Mbits/sec 15 167 KBytes
[ 11] 5.00-6.00 sec 25.4 MBytes 213 Mbits/sec 15 80.6 KBytes
[SUM] 5.00-6.00 sec 112 MBytes 938 Mbits/sec 57
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 27.0 MBytes 226 Mbits/sec 15 123 KBytes
[ 7] 6.00-7.00 sec 25.1 MBytes 211 Mbits/sec 3 154 KBytes
[ 9] 6.00-7.00 sec 35.0 MBytes 294 Mbits/sec 21 74.9 KBytes
[ 11] 6.00-7.00 sec 24.9 MBytes 209 Mbits/sec 15 112 KBytes
[SUM] 6.00-7.00 sec 112 MBytes 939 Mbits/sec 54
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 34.0 MBytes 285 Mbits/sec 11 115 KBytes
[ 7] 7.00-8.00 sec 24.9 MBytes 209 Mbits/sec 6 94.7 KBytes
[ 9] 7.00-8.00 sec 28.5 MBytes 239 Mbits/sec 18 96.2 KBytes
[ 11] 7.00-8.00 sec 25.2 MBytes 212 Mbits/sec 19 67.9 KBytes
[SUM] 7.00-8.00 sec 113 MBytes 945 Mbits/sec 54
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 32.3 MBytes 271 Mbits/sec 17 74.9 KBytes
[ 7] 8.00-9.00 sec 22.9 MBytes 192 Mbits/sec 12 86.3 KBytes
[ 9] 8.00-9.00 sec 29.6 MBytes 248 Mbits/sec 18 127 KBytes
[ 11] 8.00-9.00 sec 26.5 MBytes 223 Mbits/sec 10 126 KBytes
[SUM] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 57
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 28.7 MBytes 241 Mbits/sec 15 76.4 KBytes
[ 7] 9.00-10.00 sec 24.3 MBytes 204 Mbits/sec 9 127 KBytes
[ 9] 9.00-10.00 sec 35.0 MBytes 293 Mbits/sec 15 100 KBytes
[ 11] 9.00-10.00 sec 24.4 MBytes 204 Mbits/sec 15 140 KBytes
[SUM] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 54
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 293 MBytes 246 Mbits/sec 163 sender
[ 5] 0.00-10.00 sec 292 MBytes 245 Mbits/sec receiver
[ 7] 0.00-10.00 sec 248 MBytes 208 Mbits/sec 67 sender
[ 7] 0.00-10.00 sec 246 MBytes 207 Mbits/sec receiver
[ 9] 0.00-10.00 sec 319 MBytes 268 Mbits/sec 164 sender
[ 9] 0.00-10.00 sec 317 MBytes 266 Mbits/sec receiver
[ 11] 0.00-10.00 sec 265 MBytes 222 Mbits/sec 143 sender
[ 11] 0.00-10.00 sec 264 MBytes 221 Mbits/sec receiver
[SUM] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 537 sender
[SUM] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec receiver
Interesting side note: I unchecked all "disable offloading" boxes and re-ran all tests with hardware offloading enabled. Didn't make any difference, neither in terms of throughput nor in load.
-
With 21.7, some of the tunables have changed.
For instance, "hw.igb" doesn't exist anymore. Look at "hw.em"
Replace hw.igb.rx_process_limit="-1" with hw.em.rx_process_limit="-1"
Neither hw.igb.tx_process_limit or hw.em.tx_process_limit exist.
There are a lot of values for "dev.igb"
Unfortunately, you will need to check all your configured tunables to verify that the names haven't changed.
At the console, enter "sysctl tunable-name" to verify the name and value.
-
With 21.7, some of the tunables have changed.
For instance, "hw.igb" doesn't exist anymore. Look at "hw.em"
Ah thanks, that's exactly what I have been asking myself.
Unfortunately, you will need to check all your configured tunables to verify that the names haven't changed.
At the console, enter "sysctl tunable-name" to verify the name and value.
Will I only need to check the ones I have added manually after the 21.1.x installation or do I need to check all of them, even those that I have never touched and left at default value?
-
All names of default tunables should be fine.
I would just look at the tunables that you've added.
In my attempts to improve throughput, I've added several tunables.
My internet speed is 200 Mbps (236 Mbps actual). I get full speed with no problem.
Maybe, I would get full 200 Mbps speed without any tuning.
If I ever go to Gigabit speed, I might need to upgrade from my APU2.
-
Okay I've gone through the list. It seems the only applicable tunable is the one you already mentioned:
hw.em.rx_process_limit="-1"
Unfortunately it didn't change anything, still only 300-400 Mbit/s NAT throughput to WAN.
My ISP contract gives me ~600 Mbit/s so I really would like to see that with a single connection... So if anyone else has performance metrics for an APU2 with 21.7, I'm all ears.
In the meantime I will probably scratch my head trying to remember if I really had more throughput on 21.1.9 on one connection. And I will jealously look at OpenWRT (or Linux in general) which can pump through 1 Gbit/s on the APU2 without breaking sweat ;D
-
Are you using a bridge interface to turn your apu into a small switch? Or just one LAN, one WAN?
-
Are you using a bridge interface to turn your apu into a small switch? Or just one LAN, one WAN?
No bridges and no fancy things with WAN, but for the LAN side I'm using 802.1Q VLANs to split the physical interface into 4 different 'trust zones'. Could that be the issue?
I've read that upstream BSD has merged two drivers for the Intel NICs, so maybe VLAN filter offloading doesn't work anymore?
-
The general recommendation is to disable VLAN tag offloading for routers/firewalls and only use it for hosts that are an endpoint with respect to their communication.
-
You mean 'VLAN Hardware Filtering'? That's currently disabled so it should be fine.