Poor LAN Ethernet performance on APU2 E2

[Mewcenary]

Hi there,

I'm experiencing poor Ethernet LAN performance on an APU2 E2 board.

I am running gigabit Ethernet devices and switches.  I can get gigabit speeds between these devices when connected into the same switch, measured with iperf3.

However, running an iperf3 test between any of these devices and the OPNsense box only hits about 25% of the same speeds:

:~ % iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 1944 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.02   sec  26.2 MBytes   217 Mbits/sec    0    260 KBytes
[  5]   1.02-2.02   sec  26.2 MBytes   219 Mbits/sec    0    263 KBytes
[  5]   2.02-3.01   sec  25.6 MBytes   218 Mbits/sec    0   1.50 MBytes
[  5]   3.01-4.00   sec  25.0 MBytes   211 Mbits/sec    0   1.50 MBytes
[  5]   4.00-5.00   sec  26.3 MBytes   221 Mbits/sec    0   1.50 MBytes
[  5]   5.00-6.00   sec  27.2 MBytes   228 Mbits/sec    0   1.50 MBytes
[  5]   6.00-7.00   sec  27.1 MBytes   228 Mbits/sec    0   1.50 MBytes
[  5]   7.00-8.00   sec  24.4 MBytes   204 Mbits/sec    0   1.50 MBytes
[  5]   8.00-9.00   sec  27.1 MBytes   226 Mbits/sec    0   1.50 MBytes
[  5]   9.00-10.00  sec  27.2 MBytes   229 Mbits/sec    0   1.50 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   262 MBytes   220 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   262 MBytes   220 Mbits/sec                  receiver

I am running OPNsense 22.1.10.  I have flashed the APU2 BIOS to v4.17.0.1.

I've seen various guides on tunables but these seem very much out of date with conflicting information.

What's the next thing to look at?