Poor speeds and high CPU usage when going through OPNsense?

[frozen]

Hello all, I've been posting on reddit and Proxmox forums seeking help with a problem and I still have no solution.  For quick background reading way better than I can possibly repeat in full:
https://www.reddit.com/r/Proxmox/comments/17chu7r/proxmox_opnsense_10_performance_vs_bare_metal/
https://www.reddit.com/r/Proxmox/comments/17d59ew/proxmox_opnsense_cpu_usage_maxed_out_on_vm_but/

Both of these topics I wish I could edit as they don't reflect the current state of my problems, which are: any time I am using OPNsense and downloading at high speeds, my CPU usage goes through the roof in OPNsense and also bottlenecks my downloads.

I have a 2 port Intel i225-V 2.5GbE NIC system which is running Proxmox Linux, and has OPNsense installed in a VM with 8GB RAM and 32GB disk space allocated.  Doesn't matter if I allocate 1 CPU core or 4 cores.

I deliberately am not even introducing other devices to keep things dead simple..  1 modem, 1 OPNsense router with 2.5gb dual ports, and 1 desktop PC to use the internet on the other end.

My OPNsense configuration is posted with full screenshots in the above posts, but it's super straight forward: I am on Bell Canada 3GBPS Fiber, with PPPoE credentials from them, which work perfectly fine..  I left all of the default values in the PPPoE section and simply input my username and password and it grabs the external IP perfectly, internet connection is all working.  I connect the 10Gbps port out of my ISP modem and goes into WAN port in my OPNsense dual port NIC.

When I don't go through OPNsense I download at 2350/mbps both up and down with no problems at all, maxing it out, even if I run it 30 times in a row.   However as soon as I decide to try going through OPNsense, I lose nearly 1Gbps of download throughput (oddly enough, upload speed does not seem affected as much) and my CPU usage goes through the roof.

We have tried so many things now, if you read those threads, ranging from adjusting CPU cores, to playing with multiqueue options on the NIC settings, VirtIO options, MTU options, you name it. 

I am at a complete loss as to why OPNsense is causing the system to basically melt down with 100% usage on simple downloads.  I downloaded an Ubuntu ISO torrent just for fun, and OPNsense was using 70% CPU for a tiny 4 gb torrent sadly  

I post here in hopes of finding any kind of help or guidance or even new ideas to try, as I desperately am trying to get my full download speed back that works no problem as soon as I stop using OPNsense..

A quick example picture of my predicament:  https://imgur.com/a/YXDQW70 and now https://imgur.com/a/0rCEtw3 but there are tons more in the reddit threads at the top of this post.  As soon as I unplug from OPNsense and even stay in Proxmox but connect to the ISP router instead of OPNsense, I get a full 2400 of my 2500...  EVERY time.  It's not a temporary issue or something unrelated to OPNsense -- it's definitely a problem here between the two, but I don't know which!  Or even if I did, how to work on correcting it..

Huge thanks in advance for any help.  I would actually donate to the project as I've mentioned before, but not until this issue is solved.  It's a big problem that needs fixing and I am not smart enough to do it on my own!  I have not even tried pf as I already know from what I've read about OPNsense that it's a project I'd much rather support as well

[dMopp]

First Question: Which CPU is used inside the VM? KVM64? If yes enable AES Flag! (Or change it to host, if it’s a single Proxmox server or the cluster has the same cpus everywhere)

[frozen]

Hi there, and thank you for the response. It is indeed set to host already and also already have AES enabled sadly

[Mars79]

AFAIK You don't need to have the AES flag enabled when using the host option as CPU within Proxmox. The VM will have exactly the same CPU flags as your host system.

Furthermore, set the CPU to 1 core and 4 sockets. Make sure you use VirtIO nics and set Multiqueue to 4 or 8. There is some debate going on if it should be 4 or 8. By my understanding, setting it to 4 will force the amount of queues to 4, which in this case matches your amount of CPU cores. Setting it to 8 will make OPNsense/FreeBSD select the correct amount.

Also, make sure (as you're using PPPoE) to set the correct MTU setting on the OPNsense WAN interface. PPPoE needs additional 8 bytes and truncates the Ethernet MTU to 1492.

Did you also take a look at some performance improvements for OPNsense? Especially the Spectre and Meltdown mitigations? I've seen some major improvement on some systems when disabling these mitigations.

https://docs.opnsense.org/troubleshooting/hardening.html

Also, don't forget to install the os-qemu-guest-agent plugin.

[tverweij]

Did you also take a look at some performance improvements for OPNsense? Especially the Spectre and Meltdown mitigations? I've seen some major improvement on some systems when disabling these mitigations.

https://docs.opnsense.org/troubleshooting/hardening.html

You're a life saver!
My intercontinental IPSec lines just went to a usable state 

[frozen]

AFAIK You don't need to have the AES flag enabled when using the host option as CPU within Proxmox. The VM will have exactly the same CPU flags as your host system.

Furthermore, set the CPU to 1 core and 4 sockets. Make sure you use VirtIO nics and set Multiqueue to 4 or 8. There is some debate going on if it should be 4 or 8. By my understanding, setting it to 4 will force the amount of queues to 4, which in this case matches your amount of CPU cores. Setting it to 8 will make OPNsense/FreeBSD select the correct amount.

Also, make sure (as you're using PPPoE) to set the correct MTU setting on the OPNsense WAN interface. PPPoE needs additional 8 bytes and truncates the Ethernet MTU to 1492.

Did you also take a look at some performance improvements for OPNsense? Especially the Spectre and Meltdown mitigations? I've seen some major improvement on some systems when disabling these mitigations.

https://docs.opnsense.org/troubleshooting/hardening.html

Also, don't forget to install the os-qemu-guest-agent plugin.

Hi Mars79!  Thank you very much for getting back to me - and with some unique insight that I don't think I've heard yet.

I have never tried adjusting sockets, for starters.  I was under the impression this was a single socket board with 4 cores (Intel N100) so I stuck to always staying on 1 socket, and just adjusted cores 1 to 4 with little difference in performance, if any.

I have just changed to 4 sockets and 1 core and setting multiqueue on the VirtIO NIC's back to 4, and then will try 8

QEMU agent has been installed and activated on both Proxmox and the OPNsense VM installation and is fully enabled & running & visible green status in the OPNsense dashboard

I have not adjusted any PPPoE settigs as per advice from others on the board.  Originally I had an MTU of 9000 set universally across my network, I reverted everything back to simply (default) now (so 1500 MTU) and yes indeed, the PPPoE screen says it defaults to 1492.  Am I supposed to do anything?  A bit confused by this point - what do I adjust?  My overall MTU, or my PPPoE MTU?

I have not heard of Spectre and Meltdown mitigations and most definitely did not even glance at any such thing, I'll read that now and then also report back if the sockets/multiqueue make any difference

Thank you again

[Mars79]

If the PPPoE screen says the MTU is 1942 I wouldn't touch it anymore. This is the correct setting.

As for MTU 9000, you really only want to use that on your internal network (OPNsense LAN interface). But, MTU 9000, also called Jumbo Frames, is mostly beneficial if your network is 10 Gbps. Yes, you can enable it on 1 Gbps networks, but your equipment needs to support it. For the time being I would leave it alone, unless you really have a reason to use it.

If you want to know a bit more about Jumbo Frames:
https://en.wikipedia.org/wiki/Jumbo_frame

[frozen]

If the PPPoE screen says the MTU is 1942 I wouldn't touch it anymore. This is the correct setting.

As for MTU 9000, you really only want to use that on your internal network (OPNsense LAN interface). But, MTU 9000, also called Jumbo Frames, is mostly beneficial if your network is 10 Gbps. Yes, you can enable it on 1 Gbps networks, but your equipment needs to support it. For the time being I would leave it alone, unless you really have a reason to use it.

If you want to know a bit more about Jumbo Frames:
https://en.wikipedia.org/wiki/Jumbo_frame

I have 3GBPS fiber to the home and 2.5gbE nics indeed!  I have left it at all default settings for now though unless you think now with that additional knowledge I should switch to 9000 ?  THanks again!

[Mars79]

An MTU of 9000 does reduce overhead, which can increase network efficiency. The packets which are being sent are larger. However, like I wrote earlier, your would need to configure your entire network to support Jumbo Frames. Not only OPNsense, but also your switches, AP's etc.

You would also need to configure all of your endpoint devices to make use of Jumbo Frames, if they even support it... Otherwise it will lead to fragmentation and can cause issues.

In your case I would leave it to the default, unless you are certain all devices on your network support it and you need the extra overhead. The real benefit of Jumbo Frames would be visible in high speed networks where large amounts of data are being transferred. For example data centers or storage networks.