Slow routing/firewall performance (VP6630)

[synfinatic]

Recently upgraded my hardware from a Intel i3 7100U running pfSense to protectli VP6630 (i3-1215U) running latest OPNsense.  Internet provided by AT&T fiber (2Gbps service). 

TL;DR: my old firewall has no problems forwarding at line rate 1Gbps (it only has 1GbE NICs).  But the new firewall with 2.5GbE NIC (Intel i226-V) uplink to AT&T and 10Gbps SFP+ connection to my switch is forwarding at ~400Mbps.  I've manually installed speedtest client on the OPNsense box and it's showing >2Gbps so I know it's not some negotiation issue.   I've also tried doing a 3x1Gbps LACP/layer4 on the LAN side (how my old pfSense box was configured) and that had the same problem.  I know there's been some talk of issues with the i225-V/i226-V that sound similar, but the fact that it can send/receive at basically line rate seems to indicate it's a forwarding issue?

Firewall rules are fairly simple.  No rate limiting policies or anything like that.  I've tried enabling/disabling all the hardware offloading features under Interfaces -> Settings.  Not sure what to try next?

[synfinatic]

installed iperf3 on the firewall and my NAS (both are 10Gbps):

iperf3 -p 50419 -c thewall.xxxxxxx
Connecting to host thewall.xxxxxxx, port 50419
[  5] local 172.16.1.90 port 51190 connected to 172.16.1.1 port 50419
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   414 MBytes  3.48 Gbits/sec    0    128 KBytes
[  5]   1.00-2.00   sec   428 MBytes  3.59 Gbits/sec    0    128 KBytes
[  5]   2.00-3.00   sec   369 MBytes  3.10 Gbits/sec    0    128 KBytes
[  5]   3.00-4.00   sec   446 MBytes  3.75 Gbits/sec    0    128 KBytes
[  5]   4.00-5.00   sec   402 MBytes  3.37 Gbits/sec    0    128 KBytes
[  5]   5.00-6.00   sec   387 MBytes  3.24 Gbits/sec    0    128 KBytes
[  5]   6.00-7.00   sec   441 MBytes  3.70 Gbits/sec    0    128 KBytes
[  5]   7.00-8.00   sec   417 MBytes  3.49 Gbits/sec    0    128 KBytes
[  5]   8.00-9.00   sec   413 MBytes  3.47 Gbits/sec    0    128 KBytes
[  5]   9.00-10.00  sec   408 MBytes  3.42 Gbits/sec    0    128 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.03 GBytes  3.46 Gbits/sec    0             sender
[  5]   0.00-10.53  sec  4.03 GBytes  3.29 Gbits/sec                  receiver

Both hosts are 1500 byte MTU.  So it's not a NIC/driver issue right?  I don't have IDS or any heavy CPU load running on the firewall and when I run speedtest from the NAS the CPU load on the firewall is ~0.4.

speedtest-go running on the firewall:

./speedtest-go

    speedtest-go v1.7.10 (git-1395781) @showwin

✓ ISP: xx.x.x.x (AT&T Internet) [xxxxx, xxxx]
✓ Found 21 Public Servers

✓ Test Server: [17846] 6.30km San Jose, CA (United States) by Sonic.net, Inc.
✓ Latency: 6.949538ms Jitter: 742.116µs Min: 5.804711ms Max: 8.060521ms
✓ Packet Loss Analyzer: Running in background (<= 30 Secs)
✓ Download: 2047.58 Mbps (Used: 2465.56MB) (Latency: 12ms Jitter: 6ms Min: 6ms Max: 24ms)
✓ Upload: 2008.28 Mbps (Used: 2457.25MB) (Latency: 13ms Jitter: 4ms Min: 5ms Max: 22ms)
✓ Packet Loss: 0.00% (Sent: 283/Dup: 0/Max: 282)

speedtest-go running on the NAS (note the performance is higher than 400Mbps... seems like the go implimentation is faster than Python? Not seeing this good normally, but it's still < 1Gbps):

./speedtest-go

    speedtest-go v1.7.10 (git-1395781) @showwin

✓ ISP: xx.x.x.x (AT&T Internet) [xxxx, xxxx]
✓ Found 21 Public Servers

✓ Test Server: [56175] 6.30km San Jose, CA (United States) by Acreto
✓ Latency: 5.164619ms Jitter: 773.843µs Min: 4.187189ms Max: 6.889106ms
✓ Packet Loss Analyzer: Running in background (<= 30 Secs)
✓ Download: 742.47 Mbps (Used: 929.82MB) (Latency: 8ms Jitter: 3ms Min: 4ms Max: 15ms)
✓ Upload: 833.61 Mbps (Used: 1072.77MB) (Latency: 10ms Jitter: 4ms Min: 4ms Max: 18ms)
✓ Packet Loss: 0.00% (Sent: 268/Dup: 0/Max: 267)

[synfinatic]

welp was definitely software related.  ended up re-installing from scratch and problem resolved itself.  dunno how I could of broken things.