Messages

You mean 'VLAN Hardware Filtering'? That's currently disabled so it should be fine.

Most of the tunables have changed with 21.7. Only one of the three you have mentioned is still there but under a different name:

Code Select Expand

hw.igb.rx_process_limit  --> hw.em.rx_process_limit

Credit goes to Gary7 in this thread for clarifying this. For me that tunable doesn't seem to have an effect at all anymore.

With my APU2 I'm reaching ~300 Mbit/s NAT throughput when using a single connection and ~938 Mbit/s with four connections. An important thing to keep in mind is that if you run 'iperf3' on the box itself it will be CPU-bottlenecked, so always measure throughput between two machines with the APU in between.

My ISP gives me ~600 Mbit/s downstream so I really would like to reach that with a single connection. But I have to admit I'm not absolutely sure I had that with 21.1.9. Maybe I just never noticed.

Are you using a bridge interface to turn your apu into a small switch? Or just one LAN, one WAN?

No bridges and no fancy things with WAN, but for the LAN side I'm using 802.1Q VLANs to split the physical interface into 4 different 'trust zones'. Could that be the issue?

I've read that upstream BSD has merged two drivers for the Intel NICs, so maybe VLAN filter offloading doesn't work anymore?

Okay I've gone through the list. It seems the only applicable tunable is the one you already mentioned:

Code Select Expand

hw.em.rx_process_limit="-1"

Unfortunately it didn't change anything, still only 300-400 Mbit/s NAT throughput to WAN.

My ISP contract gives me ~600 Mbit/s so I really would like to see that with a single connection... So if anyone else has performance metrics for an APU2 with 21.7, I'm all ears.

In the meantime I will probably scratch my head trying to remember if I really had more throughput on 21.1.9 on one connection. And I will jealously look at OpenWRT (or Linux in general) which can pump through 1 Gbit/s on the APU2 without breaking sweat  ;D

With 21.7, some of the tunables have changed.

For instance, "hw.igb" doesn't exist anymore. Look at "hw.em"

Ah thanks, that's exactly what I have been asking myself.

Unfortunately, you will need to check all your configured tunables to verify that the names haven't changed.
At the console, enter "sysctl tunable-name" to verify the name and value.

Will I only need to check the ones I have added manually after the 21.1.x installation or do I need to check all of them, even those that I have never touched and left at default value?

@cookiemonster: Any changes regarding throughput? And did you keep those tunables?

My APU2 delivers ~400 Mbit/s iperf3 NAT throughput with a single connection and ~938 Mbit/s with 4 concurrent streams. I could swear I had that almost-gigabit throughput on 21.1.9 with one connection but I've got no data to prove it.

Does throughput look any better if you run the iPerf traffic through the firewall instead of making the APU2 initiate/terminate the traffic?

That's what I just finished testing :)  I've added another Linux box on the WAN side of the OPNsense box and used that as an iperf3 target from LAN.

So I have the following with one connection:

Code Select Expand

$ iperf3 -p 51337 -c 192.168.1.66
Connecting to host 192.168.1.66, port 51337
[  5] local 10.0.20.50 port 43764 connected to 192.168.1.66 port 51337
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  51.1 MBytes   429 Mbits/sec   64   1.12 MBytes
[  5]   1.00-2.00   sec  50.0 MBytes   420 Mbits/sec    0   1.24 MBytes
[  5]   2.00-3.00   sec  50.0 MBytes   419 Mbits/sec    0   1.34 MBytes
[  5]   3.00-4.00   sec  48.8 MBytes   409 Mbits/sec    0   1.41 MBytes
[  5]   4.00-5.00   sec  50.0 MBytes   419 Mbits/sec    2   1.03 MBytes
[  5]   5.00-6.00   sec  48.8 MBytes   409 Mbits/sec    0   1.10 MBytes
[  5]   6.00-7.00   sec  48.8 MBytes   409 Mbits/sec    0   1.16 MBytes
[  5]   7.00-8.00   sec  48.8 MBytes   409 Mbits/sec    0   1.20 MBytes
[  5]   8.00-9.00   sec  48.8 MBytes   409 Mbits/sec    0   1.22 MBytes
[  5]   9.00-10.00  sec  47.5 MBytes   398 Mbits/sec    0   1.23 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   492 MBytes   413 Mbits/sec   66             sender
[  5]   0.00-10.00  sec   490 MBytes   411 Mbits/sec                  receiver

~400 Mbit/s throughput including NAT, actually not bad. Still no gigabit but I maybe my memories are wrong and it has never been full speed with just one stream.

But interesting: does this mean that when running iperf3 on the APU2 its measurement is bottle-necked by CPU?

Using 4 connections I finally have my gigabit throughput with NAT:

Code Select Expand

$ iperf3 -p 51337 -c 192.168.1.66 -P 4
Connecting to host 192.168.1.66, port 51337
[  5] local 10.0.20.50 port 43776 connected to 192.168.1.66 port 51337
[  7] local 10.0.20.50 port 43778 connected to 192.168.1.66 port 51337
[  9] local 10.0.20.50 port 43780 connected to 192.168.1.66 port 51337
[ 11] local 10.0.20.50 port 43782 connected to 192.168.1.66 port 51337
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  25.9 MBytes   218 Mbits/sec   22   77.8 KBytes
[  7]   0.00-1.00   sec  26.8 MBytes   225 Mbits/sec    5    228 KBytes
[  9]   0.00-1.00   sec  36.7 MBytes   308 Mbits/sec   11    160 KBytes
[ 11]   0.00-1.00   sec  28.0 MBytes   235 Mbits/sec   13    122 KBytes
[SUM]   0.00-1.00   sec   117 MBytes   984 Mbits/sec   51
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  27.4 MBytes   230 Mbits/sec   17   80.6 KBytes
[  7]   1.00-2.00   sec  25.0 MBytes   210 Mbits/sec    3    148 KBytes
[  9]   1.00-2.00   sec  28.5 MBytes   239 Mbits/sec   19    133 KBytes
[ 11]   1.00-2.00   sec  30.9 MBytes   259 Mbits/sec    9    141 KBytes
[SUM]   1.00-2.00   sec   112 MBytes   938 Mbits/sec   48
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  27.8 MBytes   234 Mbits/sec   18   70.7 KBytes
[  7]   2.00-3.00   sec  24.9 MBytes   209 Mbits/sec    4    146 KBytes
[  9]   2.00-3.00   sec  28.5 MBytes   239 Mbits/sec   21   93.3 KBytes
[ 11]   2.00-3.00   sec  30.9 MBytes   259 Mbits/sec   14   89.1 KBytes
[SUM]   2.00-3.00   sec   112 MBytes   941 Mbits/sec   57
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  24.9 MBytes   209 Mbits/sec   21   91.9 KBytes
[  7]   3.00-4.00   sec  25.7 MBytes   215 Mbits/sec    1    173 KBytes
[  9]   3.00-4.00   sec  37.1 MBytes   311 Mbits/sec   12    144 KBytes
[ 11]   3.00-4.00   sec  24.4 MBytes   204 Mbits/sec   17    102 KBytes
[SUM]   3.00-4.00   sec   112 MBytes   940 Mbits/sec   51
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  28.3 MBytes   237 Mbits/sec   12    170 KBytes
[  7]   4.00-5.00   sec  24.8 MBytes   208 Mbits/sec   12    120 KBytes
[  9]   4.00-5.00   sec  33.9 MBytes   284 Mbits/sec   14    112 KBytes
[ 11]   4.00-5.00   sec  24.4 MBytes   204 Mbits/sec   16    100 KBytes
[SUM]   4.00-5.00   sec   111 MBytes   934 Mbits/sec   54
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  36.7 MBytes   308 Mbits/sec   15   86.3 KBytes
[  7]   5.00-6.00   sec  23.4 MBytes   196 Mbits/sec   12    116 KBytes
[  9]   5.00-6.00   sec  26.4 MBytes   222 Mbits/sec   15    167 KBytes
[ 11]   5.00-6.00   sec  25.4 MBytes   213 Mbits/sec   15   80.6 KBytes
[SUM]   5.00-6.00   sec   112 MBytes   938 Mbits/sec   57
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  27.0 MBytes   226 Mbits/sec   15    123 KBytes
[  7]   6.00-7.00   sec  25.1 MBytes   211 Mbits/sec    3    154 KBytes
[  9]   6.00-7.00   sec  35.0 MBytes   294 Mbits/sec   21   74.9 KBytes
[ 11]   6.00-7.00   sec  24.9 MBytes   209 Mbits/sec   15    112 KBytes
[SUM]   6.00-7.00   sec   112 MBytes   939 Mbits/sec   54
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  34.0 MBytes   285 Mbits/sec   11    115 KBytes
[  7]   7.00-8.00   sec  24.9 MBytes   209 Mbits/sec    6   94.7 KBytes
[  9]   7.00-8.00   sec  28.5 MBytes   239 Mbits/sec   18   96.2 KBytes
[ 11]   7.00-8.00   sec  25.2 MBytes   212 Mbits/sec   19   67.9 KBytes
[SUM]   7.00-8.00   sec   113 MBytes   945 Mbits/sec   54
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  32.3 MBytes   271 Mbits/sec   17   74.9 KBytes
[  7]   8.00-9.00   sec  22.9 MBytes   192 Mbits/sec   12   86.3 KBytes
[  9]   8.00-9.00   sec  29.6 MBytes   248 Mbits/sec   18    127 KBytes
[ 11]   8.00-9.00   sec  26.5 MBytes   223 Mbits/sec   10    126 KBytes
[SUM]   8.00-9.00   sec   111 MBytes   933 Mbits/sec   57
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  28.7 MBytes   241 Mbits/sec   15   76.4 KBytes
[  7]   9.00-10.00  sec  24.3 MBytes   204 Mbits/sec    9    127 KBytes
[  9]   9.00-10.00  sec  35.0 MBytes   293 Mbits/sec   15    100 KBytes
[ 11]   9.00-10.00  sec  24.4 MBytes   204 Mbits/sec   15    140 KBytes
[SUM]   9.00-10.00  sec   112 MBytes   942 Mbits/sec   54
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   293 MBytes   246 Mbits/sec  163             sender
[  5]   0.00-10.00  sec   292 MBytes   245 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   248 MBytes   208 Mbits/sec   67             sender
[  7]   0.00-10.00  sec   246 MBytes   207 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   319 MBytes   268 Mbits/sec  164             sender
[  9]   0.00-10.00  sec   317 MBytes   266 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   265 MBytes   222 Mbits/sec  143             sender
[ 11]   0.00-10.00  sec   264 MBytes   221 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec  537             sender
[SUM]   0.00-10.00  sec  1.09 GBytes   938 Mbits/sec                  receiver

Interesting side note: I unchecked all "disable offloading" boxes and re-ran all tests with hardware offloading enabled. Didn't make any difference, neither in terms of throughput nor in load.

Does anybody run 21.7 on an APU2 with proper throughput?

After upgrading from 21.1.9 to 21.7 (and now 21.7.1) I noticed my NAT throughput dropped to ~330 Mbit/s. Before the upgrade I had ~980 Mbit/s.

When setting up OPNsense (21.1.3 back then) I had followed this guide and set the following tunables:

Code Select Expand

net.inet.tcp.tso=1
net.inet.udp.checksum=1
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
legal.intel_igb.license_ack="1"

Could it be the case that these tunables have no effect with 21.7 anymore?

I also noticed that all hardware offloading options in the webinterface (Interfaces -> Settings) are disabled (all checkboxes are checked). I didn't know they were there, so maybe I should uncheck them instead of using the tunables?

An iperf3 run from a LAN client to the LAN interface of the OPNsense box shows:

Code Select Expand

$ iperf3 -c 10.0.20.1 -p 14218
Connecting to host 10.0.20.1, port 14218
[  5] local 10.0.20.10 port 35220 connected to 10.0.20.1 port 14218
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  26.8 MBytes   225 Mbits/sec    0   67.9 KBytes
[  5]   1.00-2.00   sec  26.6 MBytes   223 Mbits/sec    0   67.9 KBytes
[  5]   2.00-3.00   sec  26.7 MBytes   224 Mbits/sec    0   67.9 KBytes
[  5]   3.00-4.00   sec  26.6 MBytes   223 Mbits/sec    0   67.9 KBytes
[  5]   4.00-5.00   sec  26.4 MBytes   222 Mbits/sec    0   67.9 KBytes
[  5]   5.00-6.00   sec  26.7 MBytes   224 Mbits/sec    0   67.9 KBytes
[  5]   6.00-7.00   sec  26.5 MBytes   222 Mbits/sec    0   67.9 KBytes
[  5]   7.00-8.00   sec  26.6 MBytes   223 Mbits/sec    0   67.9 KBytes
[  5]   8.00-9.00   sec  26.0 MBytes   218 Mbits/sec    0   67.9 KBytes
[  5]   9.00-10.00  sec  26.3 MBytes   221 Mbits/sec    0   67.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   265 MBytes   222 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   265 MBytes   222 Mbits/sec                  receiver

Thanks in advance for any recommendations!

Okay so I booted to single-user mode, re-ran fsck -f 20 times without an error being detected.

Then I rebooted to the default runlevel and dmesg.boot is clean.

So... it seems this all came down to a couple of unclean shutdowns then? Anyway, I'll consider this fixed, sorry for the noise!

[NEVER]

@lynix

NEVER run fsck on a live read-write mounted filesystem. NEVER!

You must connect a console, boot into single user mode an run fsck with the filesystem mounted read-only.

My naive impression was that, when using UFS, the rootfs is read-only during normal operation. Okay I'll reboot to single user mode and trigger an fsck via serial.

After upgrading from 21.1.9 to 21.7 (on a PCEngines APU2) a few days ago I noticed that I have the following in dmesg.boot:

Code Select Expand

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0:  ATA8-ACS SATA 3.x device
ada0: Serial Number 50026B7268061F0D
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 28626MB (58626288 512 byte sectors)
Trying to mount root from ufs:/dev/gpt/rootfs [rw,noatime]...
WARNING: /mnt was not properly dismounted
WARNING: /mnt: mount pending error: blocks 224 files 3
WARNING: /mnt: reload pending error: blocks 224 files 3
WARNING: /mnt: reload pending error: blocks 224 files 3
WARNING: /mnt: reload pending error: blocks 224 files 3

When running multiple fsck consecutively the results seem to vary randomly:

Code Select Expand

root@opnsense:~ # fsck -f /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
71629 files, 512368 used, 6327308 free (14868 frags, 789055 blocks, 0.2% fragmentation)
root@opnsense:~ # fsck -f /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
71629 files, 512368 used, 6327308 free (14868 frags, 789055 blocks, 0.2% fragmentation)
root@opnsense:~ # fsck -f /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=1605346 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=1605347 (120 should be 0)
CORRECT? no

** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no

SUMMARY INFORMATION BAD
SALVAGE? no

BLK(S) MISSING IN BIT MAPS
SALVAGE? no

71629 files, 512351 used, 6327292 free (14860 frags, 789054 blocks, 0.2% fragmentation)

I don't know much about UFS but shouldn't the results of multiple fsck runs be identical?

Apart from those error messages everything seems to run fine. Any idea what could have gone wrong during the upgrade or what I should do to fix the issue?