Messages

46

Zenarmor (Sensei) / Re: Sensei on OPNsense - Application based filtering

« on: April 08, 2019, 09:12:03 am »

Hi!

I think keep the ram usage below 1GB would be a bit hard.
This is my smallest scenario, very low activity, sensei active only in one IF, around 8-10 users.

https://imgur.com/a/t8Bk8qg

This is a VM actually, the ram usage is below 2GB, but higher than 1GB. I cant keep below that. Of course this is the OS+Sensei RAM usage together. OPNSense eat 300-800MB RAM depending on scenario, so the 2GB usage with sensei means sensei use 1-1.5GB RAM with a low end settings.
A 2GB board should handle this, even with a swap file.
I think you can try to reach the ~1GB ram usage for a small scenario, that should be satisfy the low end HW users

Hi Archanfel80,

Many thanks for sharing your experience. Indeed, we found this very helpful.

Now I'm thinking we might be over optimizing. We were trying to keep the memory usage for the Sensei and DB below 1GB for small deployments, like 25 users. And also we are trying to provide at least a month of history.

If the median minimal RAM size for OPNsense small deployments are 2GB, your suggestion looks very viable.

Let's do a quick twitter poll:

https://twitter.com/sunnyvalley/status/1115109250479476737

With regard to beta8, glad to hear that it looks better. We've received similar feedback from several other users. Hopefully, we will be solving the remaining issue with Cloud with beta9.

47

Zenarmor (Sensei) / Re: Sensei on OPNsense - Application based filtering

« on: April 07, 2019, 09:21:15 am »

Hi!

I mostly played with heap sizes and buffer sizes. Lower values results lower memory usage in the cost of performance (slower queries) because the increased disk IO.
TimescaleDB is a good choice too. Im not sure about the Influxdb, i had to use it in the past but cause too much headache. Its not easy to operate.
Elasticsearch memory consumption also can limited. If i use in a low users <100 scenario and does not store more than 3 days data, the whole system memory usage is below 2GB. I run sensei in a 2GB board for almost a week now, small office 8 user only stored 3 days. The boss just want to see what the workers do so he check sensei reports in the end of the day. The whole system memory consumption is below 2GB. I use the default 2GB swap in opnsense but not a single byte used on that. I had to disable the sensei health check because its stopped the engine from time to time, but no issues so far. Also i have a bigger system, college with students, much more user much more data, stored 3 days history, the memory is just a bit above 4GB. I think the 8GB minimum recommended ram is a bit high. I dont have any system what eat this much.

What if sensei will detect the available system memory with the optional swap file too and gray out the big scenarios like 500 user and limit the maximum data history time limit, etc. So the user cant use a big scenario what break down the system?
For example with 2GB system, 25 users max, 3 days history
4GB system 100 users max, 7 days history
etc. And you can limit elasticsearch memory usage too.

And a quick report, after the beta8 the cloud threat query time a bit better but still cause delay what the user noticed.

Keep up the good work

Hi Archanfel80,

Many thanks for the suggestion. Actually didn't consider this as an option - wasn't aware that lucene had a lightweight option.

Currently we're evaluating Timescaledb and Influxdb. We'll also have a look at lucene lightweight option. Any pointers on this for me?

48

Zenarmor (Sensei) / Re: Sensei on OPNsense - Application based filtering

« on: April 05, 2019, 06:25:57 pm »

Hi!

Just a curious question. Did you consider using Apache Lucene as the db backend instead of Elasticsearch?
I use lucene in several projects (mostly bitnami) and its a very scalable and fast backend. There is an option to use as a "lightweight" scenario and also like as an "enterprise". It may solve the low memory hw problem.
Im just thinkin loudly

49

Zenarmor (Sensei) / Re: Sensei on OPNsense - Application based filtering

« on: April 03, 2019, 09:08:10 pm »

I just started testing and noticed the slowdown. In my case disabling cloud threat intel solved this.
maybe this helps.

I can confirm that, cloud threat intel cause noticable delay in the dns query. Its seems the cloud servers not stable enough, since i see packet loss. In a workaround use the opnsense builtin intrusion detection with ET Pro telemetry (can be installed as a plugin). Its free if you let your firewall send anonymous statistics (why not?).
Other than that sensei is an amazing product!

50

Zenarmor (Sensei) / Re: Sensei on OPNsense - Application based filtering

« on: March 31, 2019, 11:10:35 pm »

Archanfel80 could you please make a step by step guide? I will try Sensei on my apu2c4...

thank you!

Hi!

On a 2c4 which have 4GB ram enough to use the default 2GB swap file. Just enable in the system-miscellaneous.
Make sure you have limited Sensei to 100 user maximum, and you have no problem.

51

Zenarmor (Sensei) / Re: Sensei on OPNsense - Application based filtering

« on: March 29, 2019, 05:35:52 pm »

Thank You!
Both of you
I probably wait for the light version but i give it a try for the ssd swap just for testing. Its a low bandwidth system, just a few users, it might will be no problem. If yes we know its no good
Regards, Peter

52

Zenarmor (Sensei) / Re: Sensei on OPNsense - Application based filtering

« on: March 27, 2019, 09:53:49 pm »

Hi!

I use Sensei in couple of opnsense system. Works well so far.
I was wondering is there any way to run in a low memory board?
I have a pcengine APU2 board with 2GB memory, but i have a fast V-NAND msata SSD.
I setup 8GB swap file on the opnsense so i have 2GB physical and 8GB swap. The access speed not much differ since the SSD is very fast.
Im removed the memory checking row from the installation script so sensei installed succesfully.
I can configure too, it warns me the physical ram is low but i can continue.
However when i try to start the engine it says: Sensei detected swap usage is too high
And its stopped. Yes i know the swap usage is high but i dont think it can cause any issue since i use the fast ssd. Is there any way to override this? Let sensei use the swap file, i take the risk.

Thanks!

53

17.7 Legacy Series / Re: 17.7.7 to 17.7.8 upgrade issue

« on: November 22, 2017, 02:55:05 pm »

Thank You Franco!

Its fixed

54

17.7 Legacy Series / [SOLVED] 17.7.7 to 17.7.8 upgrade issue

« on: November 22, 2017, 01:41:13 pm »

Hi!

My boxes all runing the recently lastest OPNsense 17.7.7_1-amd64 version.
Today i see there is a new update availabe 17.7.8.
In the firmware upgrade page i see the new version but the changelog popup not showing unless i check that by hand, and cannot offered the upgrade. Just say my version is the latest. It is the same on the console too.
Console upgrade said im in the latest version.
All my box do the same.
This is just me or somebody else experienced this issue too?