Recently upgraded my hardware from a Intel i3 7100U running pfSense to protectli VP6630 (i3-1215U) running latest OPNsense. Internet provided by AT&T fiber (2Gbps service).
TL;DR: my old firewall has no problems forwarding at line rate 1Gbps (it only has 1GbE NICs). But the new firewall with 2.5GbE NIC (Intel i226-V) uplink to AT&T and 10Gbps SFP+ connection to my switch is forwarding at ~400Mbps. I've manually installed speedtest client on the OPNsense box and it's showing >2Gbps so I know it's not some negotiation issue. I've also tried doing a 3x1Gbps LACP/layer4 on the LAN side (how my old pfSense box was configured) and that had the same problem. I know there's been some talk of issues with the i225-V/i226-V that sound similar, but the fact that it can send/receive at basically line rate seems to indicate it's a forwarding issue?
Firewall rules are fairly simple. No rate limiting policies or anything like that. I've tried enabling/disabling all the hardware offloading features under Interfaces -> Settings. Not sure what to try next?
installed iperf3 on the firewall and my NAS (both are 10Gbps):
iperf3 -p 50419 -c thewall.xxxxxxx
Connecting to host thewall.xxxxxxx, port 50419
[ 5] local 172.16.1.90 port 51190 connected to 172.16.1.1 port 50419
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 414 MBytes 3.48 Gbits/sec 0 128 KBytes
[ 5] 1.00-2.00 sec 428 MBytes 3.59 Gbits/sec 0 128 KBytes
[ 5] 2.00-3.00 sec 369 MBytes 3.10 Gbits/sec 0 128 KBytes
[ 5] 3.00-4.00 sec 446 MBytes 3.75 Gbits/sec 0 128 KBytes
[ 5] 4.00-5.00 sec 402 MBytes 3.37 Gbits/sec 0 128 KBytes
[ 5] 5.00-6.00 sec 387 MBytes 3.24 Gbits/sec 0 128 KBytes
[ 5] 6.00-7.00 sec 441 MBytes 3.70 Gbits/sec 0 128 KBytes
[ 5] 7.00-8.00 sec 417 MBytes 3.49 Gbits/sec 0 128 KBytes
[ 5] 8.00-9.00 sec 413 MBytes 3.47 Gbits/sec 0 128 KBytes
[ 5] 9.00-10.00 sec 408 MBytes 3.42 Gbits/sec 0 128 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.03 GBytes 3.46 Gbits/sec 0 sender
[ 5] 0.00-10.53 sec 4.03 GBytes 3.29 Gbits/sec receiver
Both hosts are 1500 byte MTU. So it's not a NIC/driver issue right? I don't have IDS or any heavy CPU load running on the firewall and when I run speedtest from the NAS the CPU load on the firewall is ~0.4.
speedtest-go running on the firewall:
./speedtest-go
speedtest-go v1.7.10 (git-1395781) @showwin
✓ ISP: xx.x.x.x (AT&T Internet) [xxxxx, xxxx]
✓ Found 21 Public Servers
✓ Test Server: [17846] 6.30km San Jose, CA (United States) by Sonic.net, Inc.
✓ Latency: 6.949538ms Jitter: 742.116µs Min: 5.804711ms Max: 8.060521ms
✓ Packet Loss Analyzer: Running in background (<= 30 Secs)
✓ Download: 2047.58 Mbps (Used: 2465.56MB) (Latency: 12ms Jitter: 6ms Min: 6ms Max: 24ms)
✓ Upload: 2008.28 Mbps (Used: 2457.25MB) (Latency: 13ms Jitter: 4ms Min: 5ms Max: 22ms)
✓ Packet Loss: 0.00% (Sent: 283/Dup: 0/Max: 282)
speedtest-go running on the NAS (note the performance is higher than 400Mbps... seems like the go implimentation is faster than Python? Not seeing this good normally, but it's still < 1Gbps):
./speedtest-go
speedtest-go v1.7.10 (git-1395781) @showwin
✓ ISP: xx.x.x.x (AT&T Internet) [xxxx, xxxx]
✓ Found 21 Public Servers
✓ Test Server: [56175] 6.30km San Jose, CA (United States) by Acreto
✓ Latency: 5.164619ms Jitter: 773.843µs Min: 4.187189ms Max: 6.889106ms
✓ Packet Loss Analyzer: Running in background (<= 30 Secs)
✓ Download: 742.47 Mbps (Used: 929.82MB) (Latency: 8ms Jitter: 3ms Min: 4ms Max: 15ms)
✓ Upload: 833.61 Mbps (Used: 1072.77MB) (Latency: 10ms Jitter: 4ms Min: 4ms Max: 18ms)
✓ Packet Loss: 0.00% (Sent: 268/Dup: 0/Max: 267)
welp was definitely software related. ended up re-installing from scratch and problem resolved itself. dunno how I could of broken things.