Trick: Sensei on low end hardware, MongoDB tuning for low memory footprint

[fabianodelg]

Hi everyone,

I'd like to share a trick to solve one of the issue I had using sensei on my APU2.

My APU2 has the following configuration:

- AMD GX-412TC SOC (4 cores) (firmware updated to gain 1.4GHz)
- 4 GB RAM
- 60 GB SSD

Sensei marked my hardware as low end, proposing the installation of a local MongoDB or a remote ElasticSearch. To be honest, I have no will to install an ElasticSearch on a separated server (providing the necessary resilency and security) so MongoDB was the perfect answer.

Everything worked (is working) fine, but I did notice that since the last start up, the memory allocation was growing to a point that the system started to swap (with all the negative consequence of a system that is swapping out memory pages)

I did few research regarding the mongoDB tuning and I find out a parameter that needs to be set in the mongodb.conf config file to limit the amount of caching MongoDB would use;

Reading the MongoDB documentation:

"Memory Use
With WiredTiger, MongoDB utilizes both the WiredTiger internal cache and the filesystem cache.

Starting in MongoDB 3.4, the default WiredTiger internal cache size is the larger of either:

50% of (RAM - 1 GB), or
256 MB.
For example, on a system with a total of 4GB of RAM the WiredTiger cache will use 1.5GB of RAM (0.5 * (4 GB - 1 GB) = 1.5 GB). Conversely, a system with a total of 1.25 GB of RAM will allocate 256 MB to the WiredTiger cache because that is more than half of the total RAM minus one gigabyte (0.5 * (1.25 GB - 1 GB) = 128 MB < 256 MB)."

In a system with 4 GB (and few other things running), 1.5 GB can be too much. Changing this value to as low as 0.5 (512MB) would not make any significant impact on performance (MongoDB will use the OS caching mechanism, regardless) but, it would keep the memory allocation well under control.

To change the setting, you should enable the SSH access to your OPNsense firewall and as root user, you should edit the /usr/local/etc/mongodb.conf as follow:


# Where and how to store data.
storage:
  dbPath: /usr/local/datastore/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
  wiredTiger:
    engineConfig:
        cacheSizeGB: 0.5

Feel free to experiment; in my case, as I don't run anything but sensei, I set that to 1 (1GB). Since then, my system is not swapping at all and everything works with no issue at all.

PS: Sensei team: what a great product.. I purchased an Home license to cover my 60 devices and I'm delighted about it!!! If only the number of policies could be raised to 5... (I did the survey )




 

[mb]

@fabianodelg,

Super happy to hear that you're enjoying the product.

And thanks for the research!. This looks like a very important tweak. I think we should go ahead and make this configuration default in the upcoming release (1.10).

[fabianodelg]

Great to hear @mb !!

I'm now 5 days uptime and no problem at all, memory allocation is stable at 45% which is what I'm expecting with the cache setup to 1GB. I'm managing circa 60 devices.

I can report more before you issue the next release, in particular the mongodb logs and memory allocation pattern.

Please do let me know if I can help with that.

Thanks
Fabiano

[hushcoden]

Thanks Fabiano, very helpful tweak.

I also run an APU2 with 4GB, but just 20 devices, and the memory allocation is between 75% and 80% without your tweak, why would that be a problem?

Tia.

[fabianodelg]

Hi @hushcoden

It is not necessary an issue... it will all depends by the number of devices, if you run any other service on your firewall (ie suricata on the WAN side) and if you do lots of reporting with Sensei.

Check your swap allocation, normally you don't want that important memory pages are stored into the swap file (and then restored when needed) as this will introduce latency (with the consequence of your firewall losing packets); if you don't notice any decrease in performance, then there's no need to decrease the cache

I love my APU2... if only I could have 8 GB of ram instead of 4....

Best wishes!

[dinguz]

Please keep in mind that the mongodb configuration directives are case-sensitive, I just found this out the hard way...

[zacheryurueta]

I tried to repeat a similar trick. But this trick of mine was not successful. I just erased most of the files from the disk to nowhere and could not get everything back on my own. Only in https://www.salvagedata.com/memory-card-recovery/, they help me. If I could do it myself, I'd do it, but this is beyond my strength. All the files then seemed to have evaporated, and I couldn't find them either on the memory card, on the disk, or in the trash, or in downloads, where some file I transfer usually ends up. In general, you understand that I was in a panic then. I've deleted several important projects that I had to deliver on time. So I'll tell you that you should not go where you aren't an expert. I'm not a data setup expert. That's why I screwed up.

[sy]

Hi,

Zenarmor comes with having this setting anymore. You can try to reinstall DB. Please follow the instruction:

https://www.sunnyvalley.io/docs/troubleshooting/reporting#how-do-i-reinstall-the-reporting-database

[beneix]

I'd like to share a trick to solve one of the issue I had using sensei on my APU2.

Hi and thanks for the helpful post. I am a new OPNSense user and also using an APU2 (APU2E4 with 4 GB RAM). Before installing Zenarmor, I'd be really interested in what kind of reporting you have with the MongoDB database. Can you share any examples – either screenshots or just descriptions of the type of reporting that is possible? All the examples I have found on the internet are using Elastic, so I'm curious as to how the interface and data is different with MongoDB.

Thanks!