A couple of months ago, I migrated OPNsense from bare metal to a VM running on Truenas SCALE, which is similar to Proxmox since it's KVM on top of Debian.
My CPU is a Ryzen 7 Pro 5750G, and when I originally set it up, I allocated OPNsense four threads. It ran fine until yesterday, when I needed to transfer a large file (approximately 100 GB) to a remote destination. During the transfer, it started to get high packet loss on WAN, generally between 5% and 15%, and the transfer slowed to a crawl. What's strange is that my CPU usage shown in OPNsense was only around 10-20%.
I stopped the transfer, increased the number of threads allocated to OPNsense from 4 to 6, then restarted the transfer. This time, I've had no packet loss, and the transfer is working properly. So, it appears that OPNsense was bottle-necking and losing packets due to not having enough threads.
Does anyone know if this is normal behavior? In particular, why would CPU usage show at only 10-20% while packets are being lost due to insufficient threads? Also, do I need to increase it even more, from 6 to 8 threads, to ensure this doesn't happen again?