OPNSense not able to re-claim memory space after killing multiple process

[Afif]

Hi,
I have a strange issue where memory fill up very quickly and then swap get running out of space error causing the system/router crashed. This has been happening since last year and I have to reboot every 2 days before my whole network down. I tried to reduce the RAM from 16G to 4G (I thought it was required more RAM but the issue still happening but it delays the system crashed up to 1 weeks) and add 2G to swap (enabled in settings). I also reduce the plugins and the issue is still there. I also tried to deploy from scratch and import the config but the issue still persist. I attached my system log and I don't have any clue what was causing this, So I need your advice/help on this. Thanks in advance.

Regards,
Afif

[pfry]

Sounds like you're already aware of "System: Settings: Miscellaneous" -> "Disk / Memory Settings (reboot to apply changes)". Under normal circumstances I would recommend unchecking all three checkboxes, unless you have very weak non-volatile storage and can mitigate the issues. This is a guess, but worth a try.

[sopex]

Yes, let the defaults be. You don't need swap.

[cookiemonster]

Seems like a virtual machine but that's not in itself a problem. Many run it on VM without problem.
If external monitoring is not setup or available, you need to regularly check for running processes to see which ones are using up the memory.
Telling what the setup and running services and plugins would help too.

[cookiemonster]

Yes, let the defaults be. You don't need swap.

IMHO running any system without swap is not a very good idea. If the system needs it for any reason, it is best for it to have it than not.

[sopex]

IMHO running any system without swap is not a very good idea. If the system needs it for any reason, it is best for it to have it than not.

It's not a perfect science. In my limited experimentation, the chances of swap getting used and the whole system becoming sluggish are much higher than a 16GB+ system being starved of memory.

[cookiemonster]

If creating swap is at all possible, it is hardly a bad idea to provide it to a system. Even when there is what appears to be plenty of memory for it.
It's quite easy to see most unix-like/*bsd systems with suitable amount of memory, they swap from time to time, even if in small amounts. For the sake of a few gigabytes of storage, it's an easy and cheap help to give it.

[Afif]

I already unchecked all the settings under disk/memory and restarted the router. FYI, this OPNsense is running inside the VM for 2+ years, so this issue is happening out of nowhere, and I can't identify which processes causing it to crash. I will let it run for 1-2 days and see if the router crash or not. I will also provide the info you are asking below.

VM Specs
CPUs: 4 cores
RAM: 4GB
Storage: 32GB

Running Services:

• caddy
• configd
• cron
• ddclient
• dnsmasq
• hostwatch
• login
• ntpd
• openssh
• pf
• qemu-ga
• routing
• sysctl
• syslog-ng
• webgui
• wireguard

Plugins:

• os-caddy
• os-ddclient
• os-q-feeds-connector
• os-qemu-guest-agent

[nero355]

This has been happening since last year

Did you start using

Running Services:

• hostwatch

At the same moment perhaps ?

There have been a lot of strange issues with it since it was introduced !!

[Afif]

This has been happening since last year

Did you start using

Running Services:

• hostwatch

At the same moment perhaps ?

There have been a lot of strange issues with it since it was introduced !!

Tbh, I don't know about it. I thought that if every service running, it should be fine. Since you mentioned hostwatch service have a lot of strange issue, should I disable it?

Also, a bit update from my end. My RAM usage looks stable (within 25-35%), but it starts eating disk space little by little. Previously, it was 3.6GB and now it's already 4.6GB from the total of 32GB (12+ hours since I disabled the Temp RAM and SWAP settings). Should I be concerned about this?