NoMoreRxBDs causing slow speed?

[Kaeltis]

Since deploying my OPNsense a couple of months ago, I've noticed my WAN download speed being much worse than before.

In theory I have a 1000/50 Mbits (down/up) connection, in practice I mostly got 800/50 Mbits before.
Since deploying OPNsense it's more around 400/50 Mbits.

When I start the download of a large file or run a parallel iperf against a VPS I've noticed OPNsense showing Packetloss for my Gateway and Errors In under Interface Statistics for my WAN interface.

To verify this I checked sysctl and the Errors seem to come from dev.bge.0.stats.NoMoreRxBDs (bge0 is my WAN interface), as that number is always the same as Errors In.

Any idea what could be causing this? The interface is

Code Select Expand

Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe [14e4:165f]

Code Select Expand

sysctl -a | grep bge.0

dev.bge.0.stats.tx.BroadcastPkts: 6
dev.bge.0.stats.tx.MulticastPkts: 172
dev.bge.0.stats.tx.UnicastPkts: 4880801
dev.bge.0.stats.tx.LateCollisions: 0
dev.bge.0.stats.tx.ExcessiveCollisions: 0
dev.bge.0.stats.tx.DeferredTransmissions: 0
dev.bge.0.stats.tx.MultipleCollisionFrames: 0
dev.bge.0.stats.tx.SingleCollisionFrames: 0
dev.bge.0.stats.tx.InternalMacTransmitErrors: 0
dev.bge.0.stats.tx.XoffSent: 0
dev.bge.0.stats.tx.XonSent: 0
dev.bge.0.stats.tx.Collisions: 0
dev.bge.0.stats.tx.ifHCOutOctets: 1432966086
dev.bge.0.stats.rx.UndersizePkts: 0
dev.bge.0.stats.rx.Jabbers: 0
dev.bge.0.stats.rx.FramesTooLong: 0
dev.bge.0.stats.rx.xoffStateEntered: 0
dev.bge.0.stats.rx.ControlFramesReceived: 0
dev.bge.0.stats.rx.xoffPauseFramesReceived: 0
dev.bge.0.stats.rx.xonPauseFramesReceived: 0
dev.bge.0.stats.rx.AlignmentErrors: 0
dev.bge.0.stats.rx.FCSErrors: 0
dev.bge.0.stats.rx.BroadcastPkts: 6
dev.bge.0.stats.rx.MulticastPkts: 25684
dev.bge.0.stats.rx.UnicastPkts: 11822115
dev.bge.0.stats.rx.Fragments: 0
dev.bge.0.stats.rx.ifHCInOctets: 16998479716
dev.bge.0.stats.RecvThresholdHit: 0
dev.bge.0.stats.InputErrors: 0
dev.bge.0.stats.InputDiscards: 0
dev.bge.0.stats.NoMoreRxBDs: 1252
dev.bge.0.stats.DmaWriteHighPriQueueFull: 0
dev.bge.0.stats.DmaWriteQueueFull: 0
dev.bge.0.stats.FramesDroppedDueToFilters: 0
dev.bge.0.forced_udpcsum: 0
dev.bge.0.msi: 1
dev.bge.0.forced_collapse: 0
dev.bge.0.%parent: pci2
dev.bge.0.%pnpinfo: vendor=0x14e4 device=0x165f subvendor=0x1028 subdevice=0x1f5b class=0x020000
dev.bge.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PCI0.SE1_.S00_
dev.bge.0.%driver: bge
dev.bge.0.%desc: Broadcom NetXtreme Gigabit Ethernet, ASIC rev. 0x5720000

[Kaeltis]

I've changed some tunables according to the following pages:
https://docs.opnsense.org/troubleshooting/performance.html
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html

Code Select Expand

net.inet.tcp.tso=0
kern.ipc.nmbclusters=1000000
kern.ipc.nmbjumbop=524288
hw.intr_storm_threshold=10000

net.isr.dispatch=deferred
net.isr.bindthreads=1
net.isr.maxthreads=-1

net.inet.rss.enabled=1
net.inet.rss.bits=4

The performance is a lot better now (even though I thought with IPS enabled RSS isn't supposed to work?), I'm getting between 500 - 750 Mbits download, the mentioned errors are still increasing though. I've ordered an Intel i350 card to see if this will improve things.

[Kaeltis]

After more additional tuning according to:
https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/
https://calomel.org/freebsd_network_tuning.html

I'm now at 800-950 down, much better and the rest is probably related to my provider just not providing the full speed.
The mentioned errors are still increasing, I'll post a last update when the new Intel card arrives.

[Kaeltis]

Replacing the Broadcom card with an intel card got in fact rid of the NoMoreRxBDs errors.
So to summarize, with only intel cards and the mentioned tunables, everything is going much better now :)