1
21.7 Legacy Series / Re: Massive performance drop with 21.7 on APU2
« on: August 06, 2021, 02:57:23 pm »
You mean 'VLAN Hardware Filtering'? That's currently disabled so it should be fine.
Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
Pages: [1]
2
21.7 Legacy Series / Re: igb on APU4 with OPNS 21.7
« on: August 06, 2021, 09:18:28 am »
Most of the tunables have changed with 21.7. Only one of the three you have mentioned is still there but under a different name:
Credit goes to Gary7 in this thread for clarifying this. For me that tunable doesn't seem to have an effect at all anymore.
With my APU2 I'm reaching ~300 Mbit/s NAT throughput when using a single connection and ~938 Mbit/s with four connections. An important thing to keep in mind is that if you run 'iperf3' on the box itself it will be CPU-bottlenecked, so always measure throughput between two machines with the APU in between.
My ISP gives me ~600 Mbit/s downstream so I really would like to reach that with a single connection. But I have to admit I'm not absolutely sure I had that with 21.1.9. Maybe I just never noticed.
Code: [Select]
hw.igb.rx_process_limit --> hw.em.rx_process_limitCredit goes to Gary7 in this thread for clarifying this. For me that tunable doesn't seem to have an effect at all anymore.
With my APU2 I'm reaching ~300 Mbit/s NAT throughput when using a single connection and ~938 Mbit/s with four connections. An important thing to keep in mind is that if you run 'iperf3' on the box itself it will be CPU-bottlenecked, so always measure throughput between two machines with the APU in between.
My ISP gives me ~600 Mbit/s downstream so I really would like to reach that with a single connection. But I have to admit I'm not absolutely sure I had that with 21.1.9. Maybe I just never noticed.
3
21.7 Legacy Series / Re: Massive performance drop with 21.7 on APU2
« on: August 06, 2021, 08:01:02 am »Are you using a bridge interface to turn your apu into a small switch? Or just one LAN, one WAN?
No bridges and no fancy things with WAN, but for the LAN side I'm using 802.1Q VLANs to split the physical interface into 4 different 'trust zones'. Could that be the issue?
I've read that upstream BSD has merged two drivers for the Intel NICs, so maybe VLAN filter offloading doesn't work anymore?
4
21.7 Legacy Series / Re: Massive performance drop with 21.7 on APU2
« on: August 05, 2021, 07:52:49 pm »
Okay I've gone through the list. It seems the only applicable tunable is the one you already mentioned:
Unfortunately it didn't change anything, still only 300-400 Mbit/s NAT throughput to WAN.
My ISP contract gives me ~600 Mbit/s so I really would like to see that with a single connection... So if anyone else has performance metrics for an APU2 with 21.7, I'm all ears.
In the meantime I will probably scratch my head trying to remember if I really had more throughput on 21.1.9 on one connection. And I will jealously look at OpenWRT (or Linux in general) which can pump through 1 Gbit/s on the APU2 without breaking sweat
Code: [Select]
hw.em.rx_process_limit="-1"Unfortunately it didn't change anything, still only 300-400 Mbit/s NAT throughput to WAN.
My ISP contract gives me ~600 Mbit/s so I really would like to see that with a single connection... So if anyone else has performance metrics for an APU2 with 21.7, I'm all ears.
In the meantime I will probably scratch my head trying to remember if I really had more throughput on 21.1.9 on one connection. And I will jealously look at OpenWRT (or Linux in general) which can pump through 1 Gbit/s on the APU2 without breaking sweat
5
21.7 Legacy Series / Re: Massive performance drop with 21.7 on APU2
« on: August 05, 2021, 07:29:31 pm »With 21.7, some of the tunables have changed.
For instance, "hw.igb" doesn't exist anymore. Look at "hw.em"
Ah thanks, that's exactly what I have been asking myself.
Unfortunately, you will need to check all your configured tunables to verify that the names haven't changed.
At the console, enter "sysctl tunable-name" to verify the name and value.
Will I only need to check the ones I have added manually after the 21.1.x installation or do I need to check all of them, even those that I have never touched and left at default value?
6
21.7 Legacy Series / Re: igb on APU4 with OPNS 21.7
« on: August 05, 2021, 07:21:41 pm »
@cookiemonster: Any changes regarding throughput? And did you keep those tunables?
My APU2 delivers ~400 Mbit/s iperf3 NAT throughput with a single connection and ~938 Mbit/s with 4 concurrent streams. I could swear I had that almost-gigabit throughput on 21.1.9 with one connection but I've got no data to prove it.
My APU2 delivers ~400 Mbit/s iperf3 NAT throughput with a single connection and ~938 Mbit/s with 4 concurrent streams. I could swear I had that almost-gigabit throughput on 21.1.9 with one connection but I've got no data to prove it.
7
21.7 Legacy Series / Re: Massive performance drop with 21.7 on APU2
« on: August 05, 2021, 06:49:14 pm »Does throughput look any better if you run the iPerf traffic through the firewall instead of making the APU2 initiate/terminate the traffic?
That's what I just finished testing
I've added another Linux box on the WAN side of the OPNsense box and used that as an iperf3 target from LAN.So I have the following with one connection:
Code: [Select]
$ iperf3 -p 51337 -c 192.168.1.66
Connecting to host 192.168.1.66, port 51337
[ 5] local 10.0.20.50 port 43764 connected to 192.168.1.66 port 51337
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 51.1 MBytes 429 Mbits/sec 64 1.12 MBytes
[ 5] 1.00-2.00 sec 50.0 MBytes 420 Mbits/sec 0 1.24 MBytes
[ 5] 2.00-3.00 sec 50.0 MBytes 419 Mbits/sec 0 1.34 MBytes
[ 5] 3.00-4.00 sec 48.8 MBytes 409 Mbits/sec 0 1.41 MBytes
[ 5] 4.00-5.00 sec 50.0 MBytes 419 Mbits/sec 2 1.03 MBytes
[ 5] 5.00-6.00 sec 48.8 MBytes 409 Mbits/sec 0 1.10 MBytes
[ 5] 6.00-7.00 sec 48.8 MBytes 409 Mbits/sec 0 1.16 MBytes
[ 5] 7.00-8.00 sec 48.8 MBytes 409 Mbits/sec 0 1.20 MBytes
[ 5] 8.00-9.00 sec 48.8 MBytes 409 Mbits/sec 0 1.22 MBytes
[ 5] 9.00-10.00 sec 47.5 MBytes 398 Mbits/sec 0 1.23 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 492 MBytes 413 Mbits/sec 66 sender
[ 5] 0.00-10.00 sec 490 MBytes 411 Mbits/sec receiver~400 Mbit/s throughput including NAT, actually not bad. Still no gigabit but I maybe my memories are wrong and it has never been full speed with just one stream.
But interesting: does this mean that when running iperf3 on the APU2 its measurement is bottle-necked by CPU?
Using 4 connections I finally have my gigabit throughput with NAT:
Code: [Select]
$ iperf3 -p 51337 -c 192.168.1.66 -P 4
Connecting to host 192.168.1.66, port 51337
[ 5] local 10.0.20.50 port 43776 connected to 192.168.1.66 port 51337
[ 7] local 10.0.20.50 port 43778 connected to 192.168.1.66 port 51337
[ 9] local 10.0.20.50 port 43780 connected to 192.168.1.66 port 51337
[ 11] local 10.0.20.50 port 43782 connected to 192.168.1.66 port 51337
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 25.9 MBytes 218 Mbits/sec 22 77.8 KBytes
[ 7] 0.00-1.00 sec 26.8 MBytes 225 Mbits/sec 5 228 KBytes
[ 9] 0.00-1.00 sec 36.7 MBytes 308 Mbits/sec 11 160 KBytes
[ 11] 0.00-1.00 sec 28.0 MBytes 235 Mbits/sec 13 122 KBytes
[SUM] 0.00-1.00 sec 117 MBytes 984 Mbits/sec 51
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 27.4 MBytes 230 Mbits/sec 17 80.6 KBytes
[ 7] 1.00-2.00 sec 25.0 MBytes 210 Mbits/sec 3 148 KBytes
[ 9] 1.00-2.00 sec 28.5 MBytes 239 Mbits/sec 19 133 KBytes
[ 11] 1.00-2.00 sec 30.9 MBytes 259 Mbits/sec 9 141 KBytes
[SUM] 1.00-2.00 sec 112 MBytes 938 Mbits/sec 48
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 27.8 MBytes 234 Mbits/sec 18 70.7 KBytes
[ 7] 2.00-3.00 sec 24.9 MBytes 209 Mbits/sec 4 146 KBytes
[ 9] 2.00-3.00 sec 28.5 MBytes 239 Mbits/sec 21 93.3 KBytes
[ 11] 2.00-3.00 sec 30.9 MBytes 259 Mbits/sec 14 89.1 KBytes
[SUM] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 57
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 24.9 MBytes 209 Mbits/sec 21 91.9 KBytes
[ 7] 3.00-4.00 sec 25.7 MBytes 215 Mbits/sec 1 173 KBytes
[ 9] 3.00-4.00 sec 37.1 MBytes 311 Mbits/sec 12 144 KBytes
[ 11] 3.00-4.00 sec 24.4 MBytes 204 Mbits/sec 17 102 KBytes
[SUM] 3.00-4.00 sec 112 MBytes 940 Mbits/sec 51
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 28.3 MBytes 237 Mbits/sec 12 170 KBytes
[ 7] 4.00-5.00 sec 24.8 MBytes 208 Mbits/sec 12 120 KBytes
[ 9] 4.00-5.00 sec 33.9 MBytes 284 Mbits/sec 14 112 KBytes
[ 11] 4.00-5.00 sec 24.4 MBytes 204 Mbits/sec 16 100 KBytes
[SUM] 4.00-5.00 sec 111 MBytes 934 Mbits/sec 54
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 36.7 MBytes 308 Mbits/sec 15 86.3 KBytes
[ 7] 5.00-6.00 sec 23.4 MBytes 196 Mbits/sec 12 116 KBytes
[ 9] 5.00-6.00 sec 26.4 MBytes 222 Mbits/sec 15 167 KBytes
[ 11] 5.00-6.00 sec 25.4 MBytes 213 Mbits/sec 15 80.6 KBytes
[SUM] 5.00-6.00 sec 112 MBytes 938 Mbits/sec 57
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 27.0 MBytes 226 Mbits/sec 15 123 KBytes
[ 7] 6.00-7.00 sec 25.1 MBytes 211 Mbits/sec 3 154 KBytes
[ 9] 6.00-7.00 sec 35.0 MBytes 294 Mbits/sec 21 74.9 KBytes
[ 11] 6.00-7.00 sec 24.9 MBytes 209 Mbits/sec 15 112 KBytes
[SUM] 6.00-7.00 sec 112 MBytes 939 Mbits/sec 54
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 34.0 MBytes 285 Mbits/sec 11 115 KBytes
[ 7] 7.00-8.00 sec 24.9 MBytes 209 Mbits/sec 6 94.7 KBytes
[ 9] 7.00-8.00 sec 28.5 MBytes 239 Mbits/sec 18 96.2 KBytes
[ 11] 7.00-8.00 sec 25.2 MBytes 212 Mbits/sec 19 67.9 KBytes
[SUM] 7.00-8.00 sec 113 MBytes 945 Mbits/sec 54
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 32.3 MBytes 271 Mbits/sec 17 74.9 KBytes
[ 7] 8.00-9.00 sec 22.9 MBytes 192 Mbits/sec 12 86.3 KBytes
[ 9] 8.00-9.00 sec 29.6 MBytes 248 Mbits/sec 18 127 KBytes
[ 11] 8.00-9.00 sec 26.5 MBytes 223 Mbits/sec 10 126 KBytes
[SUM] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 57
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 28.7 MBytes 241 Mbits/sec 15 76.4 KBytes
[ 7] 9.00-10.00 sec 24.3 MBytes 204 Mbits/sec 9 127 KBytes
[ 9] 9.00-10.00 sec 35.0 MBytes 293 Mbits/sec 15 100 KBytes
[ 11] 9.00-10.00 sec 24.4 MBytes 204 Mbits/sec 15 140 KBytes
[SUM] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 54
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 293 MBytes 246 Mbits/sec 163 sender
[ 5] 0.00-10.00 sec 292 MBytes 245 Mbits/sec receiver
[ 7] 0.00-10.00 sec 248 MBytes 208 Mbits/sec 67 sender
[ 7] 0.00-10.00 sec 246 MBytes 207 Mbits/sec receiver
[ 9] 0.00-10.00 sec 319 MBytes 268 Mbits/sec 164 sender
[ 9] 0.00-10.00 sec 317 MBytes 266 Mbits/sec receiver
[ 11] 0.00-10.00 sec 265 MBytes 222 Mbits/sec 143 sender
[ 11] 0.00-10.00 sec 264 MBytes 221 Mbits/sec receiver
[SUM] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 537 sender
[SUM] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec receiverInteresting side note: I unchecked all "disable offloading" boxes and re-ran all tests with hardware offloading enabled. Didn't make any difference, neither in terms of throughput nor in load.
8
21.7 Legacy Series / Massive performance drop with 21.7 on APU2
« on: August 05, 2021, 02:06:25 pm »
Does anybody run 21.7 on an APU2 with proper throughput?
After upgrading from 21.1.9 to 21.7 (and now 21.7.1) I noticed my NAT throughput dropped to ~330 Mbit/s. Before the upgrade I had ~980 Mbit/s.
When setting up OPNsense (21.1.3 back then) I had followed this guide and set the following tunables:
Could it be the case that these tunables have no effect with 21.7 anymore?
I also noticed that all hardware offloading options in the webinterface (Interfaces -> Settings) are disabled (all checkboxes are checked). I didn't know they were there, so maybe I should uncheck them instead of using the tunables?
An iperf3 run from a LAN client to the LAN interface of the OPNsense box shows:
Thanks in advance for any recommendations!
After upgrading from 21.1.9 to 21.7 (and now 21.7.1) I noticed my NAT throughput dropped to ~330 Mbit/s. Before the upgrade I had ~980 Mbit/s.
When setting up OPNsense (21.1.3 back then) I had followed this guide and set the following tunables:
Code: [Select]
net.inet.tcp.tso=1
net.inet.udp.checksum=1
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
legal.intel_igb.license_ack="1"Could it be the case that these tunables have no effect with 21.7 anymore?
I also noticed that all hardware offloading options in the webinterface (Interfaces -> Settings) are disabled (all checkboxes are checked). I didn't know they were there, so maybe I should uncheck them instead of using the tunables?
An iperf3 run from a LAN client to the LAN interface of the OPNsense box shows:
Code: [Select]
$ iperf3 -c 10.0.20.1 -p 14218
Connecting to host 10.0.20.1, port 14218
[ 5] local 10.0.20.10 port 35220 connected to 10.0.20.1 port 14218
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 26.8 MBytes 225 Mbits/sec 0 67.9 KBytes
[ 5] 1.00-2.00 sec 26.6 MBytes 223 Mbits/sec 0 67.9 KBytes
[ 5] 2.00-3.00 sec 26.7 MBytes 224 Mbits/sec 0 67.9 KBytes
[ 5] 3.00-4.00 sec 26.6 MBytes 223 Mbits/sec 0 67.9 KBytes
[ 5] 4.00-5.00 sec 26.4 MBytes 222 Mbits/sec 0 67.9 KBytes
[ 5] 5.00-6.00 sec 26.7 MBytes 224 Mbits/sec 0 67.9 KBytes
[ 5] 6.00-7.00 sec 26.5 MBytes 222 Mbits/sec 0 67.9 KBytes
[ 5] 7.00-8.00 sec 26.6 MBytes 223 Mbits/sec 0 67.9 KBytes
[ 5] 8.00-9.00 sec 26.0 MBytes 218 Mbits/sec 0 67.9 KBytes
[ 5] 9.00-10.00 sec 26.3 MBytes 221 Mbits/sec 0 67.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 265 MBytes 222 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 265 MBytes 222 Mbits/sec receiverThanks in advance for any recommendations!
9
21.7 Legacy Series / Re: Corrupt rootfs after upgrade to 21.7
« on: August 01, 2021, 12:43:49 pm »
Okay so I booted to single-user mode, re-ran fsck -f 20 times without an error being detected.
Then I rebooted to the default runlevel and dmesg.boot is clean.
So... it seems this all came down to a couple of unclean shutdowns then? Anyway, I'll consider this fixed, sorry for the noise!
Then I rebooted to the default runlevel and dmesg.boot is clean.
So... it seems this all came down to a couple of unclean shutdowns then? Anyway, I'll consider this fixed, sorry for the noise!
10
21.7 Legacy Series / Re: Corrupt rootfs after upgrade to 21.7
« on: July 31, 2021, 05:25:31 pm »@lynix
NEVER run fsck on a live read-write mounted filesystem. NEVER!
You must connect a console, boot into single user mode an run fsck with the filesystem mounted read-only.
My naive impression was that, when using UFS, the rootfs is read-only during normal operation. Okay I'll reboot to single user mode and trigger an fsck via serial.
11
21.7 Legacy Series / [resolved] Corrupt rootfs after upgrade to 21.7
« on: July 31, 2021, 10:32:34 am »
After upgrading from 21.1.9 to 21.7 (on a PCEngines APU2) a few days ago I noticed that I have the following in dmesg.boot:
When running multiple fsck consecutively the results seem to vary randomly:
I don't know much about UFS but shouldn't the results of multiple fsck runs be identical?
Apart from those error messages everything seems to run fine. Any idea what could have gone wrong during the upgrade or what I should do to fix the issue?
Code: [Select]
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: ATA8-ACS SATA 3.x device
ada0: Serial Number 50026B7268061F0D
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 28626MB (58626288 512 byte sectors)
Trying to mount root from ufs:/dev/gpt/rootfs [rw,noatime]...
WARNING: /mnt was not properly dismounted
WARNING: /mnt: mount pending error: blocks 224 files 3
WARNING: /mnt: reload pending error: blocks 224 files 3
WARNING: /mnt: reload pending error: blocks 224 files 3
WARNING: /mnt: reload pending error: blocks 224 files 3When running multiple fsck consecutively the results seem to vary randomly:
Code: [Select]
root@opnsense:~ # fsck -f /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
71629 files, 512368 used, 6327308 free (14868 frags, 789055 blocks, 0.2% fragmentation)
root@opnsense:~ # fsck -f /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
71629 files, 512368 used, 6327308 free (14868 frags, 789055 blocks, 0.2% fragmentation)
root@opnsense:~ # fsck -f /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=1605346 (8 should be 0)
CORRECT? no
INCORRECT BLOCK COUNT I=1605347 (120 should be 0)
CORRECT? no
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no
SUMMARY INFORMATION BAD
SALVAGE? no
BLK(S) MISSING IN BIT MAPS
SALVAGE? no
71629 files, 512351 used, 6327292 free (14860 frags, 789054 blocks, 0.2% fragmentation)I don't know much about UFS but shouldn't the results of multiple fsck runs be identical?
Apart from those error messages everything seems to run fine. Any idea what could have gone wrong during the upgrade or what I should do to fix the issue?
Pages: [1]