Sensei on OPNsense - Application based filtering

[faisalreza]

hi, new user here,
been searching this thread but cannot find it yet

how to change db engine from mongodb to elasticsearch?
now already installed elasticsearch5 via the shell
does we have to reset sensei to default config or uninstall then install it back?

thanks for the clue

Regards
Reza

[mb]

Hi faisal,

Many thanks for trying Sensei.

Backend Database selection is done automatically based on your hardware resources. If you have less than 8GB RAM, sensei will pick mongodb.

[faisalreza]

hi mb thanks for answering
i have xeon e3 4 core 8 thread with 16gb ram

but no options for using elasticsearch, any required steps or i missed something?

regards
Reza

[mb]

Hi faisal,

Than it must be the cpu score. There is a 300.000 minimum cpu score requirement for Elasticsearch.

Here's  a quick hack:

1. Remove /usr/local/sensei/etc/.configdone

Code Select Expand

rm /usr/local/sensei/etc/.configdone

3. Edit /usr/local/opnsense/scripts/OPNsense/Sensei/check_hardware.sh file and locate these lines:

Code Select Expand

if [ $CPU_SCORE -le 300000 ]; then
       CPU_PROPER="false"
else
       CPU_PROPER="true"
fi


Change 300000 to a lower value, like 200000. 

4. Do a browser refresh on the OPNsense UI, and click on any sensei menu. It'll re-run the config wizard. Now it should select Elasticsearch.

Now I'm thinking: for cpu scores between 200K and 300K and if there is enough memory (>=8GB) I think we should let the user decide on the database backend.

[faisalreza]

hi mb, done that and here's the result

/usr/local/opnsense/scripts/OPNsense/Sensei/check_hardware.sh

Code Select Expand

{
   "memory": {
       "size": 17179869184,
       "proper": true
   },
   "cpu": {
       "model": "Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz",
       "proper": true,
       "score": 224783
   },
   "opnsense_version": "1979_1"
}

is there any possibility to separate log file location for the reporting?, i have opnsense installed on a 128GB ssd and sensei looks like take amount of space for keeping log, does 500gb - 1tb disk good enough for log and analytics?

[mb]

Hi faisal,

Good, you can now do the initial configuration, it should install Elasctic now.

Currently database location is /var/db. Upcoming 1.4 or 1.5 will move it to /usr/local since /var can be a temp memory file system in OPNsense.

For disk sizing, you can use this guide:

https://help.sunnyvalley.io/hc/en-us/articles/360025047373-Hardware-Requirements

[colourcode]

Keep gettin this error even after a clean reinstall.

[29-Jan-2020 07:02:25 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'mongodb.so' (tried: /usr/local/lib/php/20170718/mongodb.so (Shared object "libcrypto.so.9" not found, required by "mongodb.so"), /usr/local/lib/php/20170718/mongodb.so.so (Cannot open "/usr/local/lib/php/20170718/mongodb.so.so")) in Unknown on line 0

In /user/local/lib there is a libcrypto.so.11 ..

[dragon2611]

I can't install in a VM on a Mac Mini in proxmox, It should (just about) meet the minimum requirements

CPU Model:Intel(R) Core(TM) i5-4260U CPU @ 1.40GHz
CPU Score:384496
Physical Memory Size:2.13 GB (Mini only has 4GB)

Code Select Expand

Please make sure you are running the latest OPNsense version

Code Select Expand

OPNsense 19.7.10-amd64

OPNsense isn't finding any newer updates than this  ???


I could try an install on more powerful hardware but then I'd have to tunnel the traffic I wanted to pass through Sensei to the datacentre first.

Edit:

Seemed to work following a reboot, guess there was an installed update that needed a reboot.

[mb]

Hi @lakej,

I guess you are on OPNsense 20.1rc1. Current Sensei repo is not yet ready for 20.1 since it's not yet released.

Two options:

1. Wait until 20.1 is officially released and re-install sensei, since we'll ship the required dependency packages when 20.1 is officially released.
2. Use 19.7.10 for now.

I would suggest waiting a bit more since we expect that OPNsense will release 20.1 tomorrow. (then we'll ship the 20.1 repo)

[hbc]

Question:

What system privileges are needed to display / restrict sensei pages? I have several groups with just access to certain pages (viewonly, voucher creation, basic operation, etc.)

There exist no predefined privileges for sensei. I want just to allow reports and status. Without possibility to edit settings.

How can I restrict that?

[faisalreza]

hi mb
after doing the cpu score hack, is still installing mongodb, and i cannot see either options to install elasticsearch

i continue installation and after finished, shown error like this

Warning: Sensei is stopped because of a problem
Sensei has detected a problem during operation and has shut down Sensei services in order to prevent a network outage.

Cannot find workers map file

If you think this is something we should have a look, just click here to let us know about the details and we will investigate this further.

You can re-enable the services from Status page.

any clues?

Hi faisal,

Good, you can now do the initial configuration, it should install Elasctic now.

Currently database location is /var/db. Upcoming 1.4 or 1.5 will move it to /usr/local since /var can be a temp memory file system in OPNsense.

For disk sizing, you can use this guide:

https://help.sunnyvalley.io/hc/en-us/articles/360025047373-Hardware-Requirements

[mb]

Dear Sensei users, (especiallly Mongodb users)

We advise that you postpone 20.1 upgrades for a day or two while we confirm everything works as expected.

20.1 is a major upgrade, we want to make sure upgrade path for Sensei users is clear.

We'll post an update here and from twitter once we have confirmed everythins is ok.

[colourcode]

I'm getting 500mpbs without sensei and 0,5mbps with sensei.

I have a dogshit CPU, E3950 @ 1.60GHz (4 cores).

However the CPU is barely breaking a sweat and memory utilization is ~20% (8gb).

is this what I can expect performance-wise out of this hardware?
I was thinking abut upgrading but I'm doubting the Hades Canyon or similar can pull it if this isn't working out right now?

Or could there be some configuration error at play here?

Edit: Hardware offloading was the problem. now it's around 300 :)

[mb]

We advise that you postpone 20.1 upgrades for a day or two while we confirm everything works as expected.

20.1 is a major upgrade, we want to make sure upgrade path for Sensei users is clear.

We'll post an update here and from twitter once we have confirmed everythins is ok.

Tests have been completed and looks good. We're all clear for 20.1.

[xsfpo]

I also get errors like this:

Code Select Expand

PHP Warning:  PHP Startup: Unable to load dynamic library 'mongodb.so' (tried: /usr/local/lib/php/20170718/mongodb.so (Shared object "libcrypto.so.11" not found, required by "mongodb.so"), /usr/local/lib/php/20170718/mongodb.so.so (Cannot open "/usr/local/lib/php/20170718/mongodb.so.so")) in Unknown on line 0
Even after complete uninstall of sensei plugin.
OPNsense 19.7.10_1-amd64