Unexplained drop in LAN to WAN speed

[jaykumar2005]

OPNsense 24.7.11_2-amd64
Intel Core i5-8500 CPU @ 3.00GHz Tiny PC
Generic NIC with Intel T4xI350
WAN 1 gbps

iperf3 LAN host to Opnsense Firewall ~ 900 mbps
Opnsense Firewall to WAN speedtest ~ 900 mbps

But LAN host to Speedtest (same server id) ~ 300 mbps
Tested across multiple LAN hosts

Disabled Zenarmor, no change
Disabled traffic shaping, no change
Not running any IDS/IPS, Crowdsec etc.

Any troubleshooting tips?

[pfry]

OPNsense 24.7.11_2-amd64
Intel Core i5-8500 CPU @ 3.00GHz Tiny PC [...]

Not all that tiny (in performance). RSS, perhaps?

https://docs.opnsense.org/troubleshooting/performance.html

Not sure if/how this would interact with Zenarmor.

[jaykumar2005]

I have RSS enabled,

Code Select Expand

net.inet.rss.bucket_mapping: 0:0 1:1 2:2 3:3
net.inet.rss.enabled: 1
net.inet.rss.debug: 0
net.inet.rss.basecpu: 0
net.inet.rss.buckets: 4
net.inet.rss.maxcpus: 64
net.inet.rss.ncpus: 6
net.inet.rss.maxbits: 7
net.inet.rss.mask: 3
net.inet.rss.bits: 2
net.inet.rss.hashalgo: 2
hw.bxe.udp_rss: 0
hw.ix.enable_rss: 1

Tunables are also as per recommendations

Code Select Expand

net.isr.bindthreads = 1
net.isr.maxthreads = -1

net.inet.rss.enabled = 1

net.inet.rss.bits = 2


[pfry]

I have RSS enabled,
[...]

It was a thought. Not a great one, but worth the question. I imagine we can discard control domain issues, test differences, etc. Filter differences should not have a measurable effect, either. I imagine you've also looked at CPU and resource utilization, and looked for any anomalous differences in the test stats (namely latency).

I'm curious about this in part because my second firewall should be an i3-9300t, a 61% version of your machine, but with an x710-DA2, an i210, and an i219.

[jaykumar2005]

I am running homelab with Elastiflow, Grafana (prometheus exporter) and Librenms (snmpv3), none of the observability platforms are showing any anomaly.

The last change I made was to add bunch of VLAN interfaces, planning to use trunked L3 switch for VLAN seggregation. I will remove these interfaces to check if it makes any difference.

[jaykumar2005]

Solved.

Looks like mismatched jumbo frames enabled on the couple of switches might have been causing this issue. Disabled Jumbo frames on all devices across the network, which resolved the issue.