[Solved] High memory usage in Proxmox VM

[REH]

Hello everyone!

I've recently got myself a Proxmox server and thought I would give OPNsense a go, but I'm wondering if there is a setting I'm missing somewhere in my install, because no matter how much RAM i throw at the OPNsense VM it always seems to use 85+% of RAM given. Is this normal on a fresh install of OPNsense?

[Waschbuesch]

A clean OPNsense install has really low memory usage in my experience.

Where did you take the information from? About the RAM usage, I mean? From the VM status in Proxmox?

Because: Outside of the VM, Proxmox (KVM) does not reliably / accurately display memory usage inside FreeBSD hosts.

FreeBSD's memory management knows several memory classes: Active, Inactive, Laundry, Free, Wired
(See https://wiki.freebsd.org/Memory)

KVM only displays the amount of 'used' memory. I do not know how they calculate that.
In short: I would disregard the value KVM displays and go with OPNsense's dashboard info.

[REH]

Where did you take the information from? About the RAM usage, I mean? From the VM status in Proxmox?

No, unfortunately it is the reading from the OPNsense dashboard.

[Waschbuesch]

Decidedly odd.
I just setup a new VM on my Proxmox cluster and with 8 GB RAM, OPNsense (20.1) showed only a usage of a bit under 300M.

Can you share more information on the setup of the VM?

[REH]

Decidedly odd.
I just setup a new VM on my Proxmox cluster and with 8 GB RAM, OPNsense (20.1) showed only a usage of a bit under 300M.

Great! That probably means it was just me who did something silly in the creation of my VM.   

Can you share more information on the setup of the VM?

Hope the attached screenshots can answer your question, otherwise ask away! 

[banym]

Can you share what services and plugins you have configured on the firewall itself.

Proxy, IDS etc. can lead to high memory consumption. Maybe this is a starting point to find what service is consuming all the memory.

You can use default tools like free and top to see what service is using all the memory.

[REH]

Can you share what services and plugins you have configured on the firewall itself.

Proxy, IDS etc. can lead to high memory consumption. Maybe this is a starting point to find what service is consuming all the memory.

You can use default tools like free and top to see what service is using all the memory.

As i stated in my first post it is a fresh install.
So I haven't touched any services or plugins yet.

[allebone]

Im no expert but from the screenshot memory ballooning might be being used? If that is the case then this behaviour is expected. I use OpnSense on unraid and the same thing happens for me. Also as an aside just as another tip, for me to get best performance I had to use machine type i440fx (im using version 4.2) and Seabios so that I could pass the virtio network drivers. These nics provide the best performance in a VM, unless of course you are using pci passthrough on the nics which would be fine also.

Pete

[REH]

Im no expert but from the screenshot memory ballooning might be being used? If that is the case then this behaviour is expected. I use OpnSense on unraid and the same thing happens for me.

That was it! With it turned off, the usage is down to 250mb.

Also as an aside just as another tip, for me to get best performance I had to use machine type i440fx (im using version 4.2) and Seabios so that I could pass the virtio network drivers. These nics provide the best performance in a VM, unless of course you are using pci passthrough on the nics which would be fine also.

Pete

Thanks for the tips, but I'm using an Intel NIC PCI card where i pass the ports through.

[allebone]

Thanks for the tips, but I'm using an Intel NIC PCI card where i pass the ports through.

Yup thats the gold standard. For those of us on a tighter budget, bridging is ok for a home setup

Good luck with your build - OpnSense is a very nice product

[Dunuin]

I had the same problem and asked the Proxmox developers and they explained it to me...

The problem is that KVM is reserving every RAM it can get if running OPNsense. So if you give the OPNsense-VM 12GB of RAM it will always use the full 12GB on the host, even if the OPNsense guest is only using 400MB of that and 11.6GB are free.

I created a OPNsense VM with 12GB RAM and enabled ballooning. Proxmox GUI is always showing "Memory usage 96,54% (11.58 GiB of 12.00 GiB)", even if OPNsense only uses 400MB of RAM (inlcuding buffer), because the FreeBSD implementation of the virtio ballooning driver is only reporting "actual=12288 max_mem=12288" and not addition data like "total_mem" and "free_mem" what a Win10 ballon driver will return.
Because of that Proxmox can't use the balloon driver to read the free RAM of OPNsense and is falling back to the RAM usage the kvm process is reserving.

If I run top on the host to monitor the OPNsenseVMs process I get this:

Code: [Select]

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
26838 root      20   0   12.5g  12.1g  10212 S  10.0  19.2   2:44.71 kvm
Running top inside the OPNsense VM I get this:

Code: [Select]

CPU:  0.0% user,  0.0% nice,  0.2% system,  0.2% interrupt, 99.7% idle
Mem: 103M Active, 15M Inact, 229M Wired, 97M Buf, 11G Free
Swap: 4096M Total, 4096M Free
So the question is, why is OPNsense reserving 12GB of RAM if only 400MB is needed?

Can I do something so a kvm virtualized OPNsense is only reserving the RAM it really needs?

[allebone]

Turn off memory ballooning. Assign it 2GB ram and there will no longer be a problem.

[Dunuin]

Its the same problem with and without ballooning. Sure, I could lower the RAM as much as possible so OPNsense and kvm aren't wasting that much RAM but even if I limit the RAM to 2GB, 1.6GB will be wasted and not usable by the host or other VMs. If I lower it too much the VM could run out of memory.

The question is why is OPNsense reserving all RAM available if it doesn't use it? Thats really not efficient if you want to virtualize it.

[allebone]

Hi Dunuin,

I dont use proxmox, but my VM of Opnsense does not do what you are saying. I use Unraid and Kvm.
It must be a proxmox issue. My VM of Opnsense only uses the memory it needs and does not use all the RAM I assign it.

I also dont use memory ballooning.

I suggest you ask on the proxmox forum why your choice of hypervisor does this. Opnsense is built with a physical machine in mind. If you have any issues with visualizing Opnsense the fix will have to come from the developer of whoever supports visualizing machines in the method you choose. The Opnsense developers dont work in the virtualization area, they maintain a firewall.

Good luck

Pete