Sensei on OPNsense - Application based filtering

[mb]

Hi @krdhtet,

You got it in your inbox ;)

[sol]

A couple of questions:

Is this CPU usage (60-70%) for the configuration Sensei is not running? (e.g. IPS+Proxy+AV) ?

Yes

When you launch Sensei, how much did you see it changed? Does it top to 100%?
It goes up to 95% and drops to ~50%. It also drops and peaks way more often

Furthermore I couldnt disable Sensei and I was only able to uninstall it right after a reboot. 
After a new try to install it again over the current system opnsense crashed and it had to reinstall Opnsense.
I guess some old settings made a clean reinstallation of Sensei impossible.
Lets hope that a new Sensei version will fix the option to stop it.

Looking forward to an update and will give it a try another time.

[mb]

Hi @sol,

Many thanks for reporting this and for the answer. This is very much helpful to understand what's going on.

Looks like a quite loaded system. I would not recommend running with a 60-70% cpu utilization if you're doing some kind of packet processing. Because packet processing requires dedicated resources and if the cpu is highly utilized and also shared with other applications, it's highly possible that you'll start losing packets. This is so, because at some point OS will fail to schedule the packet processing application to a CPU (because the CPU is already busy) and packets will be dropped in this timeframe. As a consequence,  this will create congestion, and finally you'll get lower throughput. This was what happened, lowering your throughput from 150 - 85 Mbps.

To remedy this kind of heavy load scenarios, there is one thing you can do, and one thing we can:

For you, as you wrote before, it'd be better if you can run the configuration with a more resourceful HW.
For Sensei, we'll pin it to a dedicated CPU core. This will help if you have a multi-core system. 

For not being able to stop Sensei, I'd guess it's related to the above scenario. Though it should stop anyway whatever the load is.

We'll try to reproduce this with your conditions in our lab. I'll let you know about our results.

For the sake of clarity: were you trying to stop it by clicking on the  "Stop" action button or by disabling "Start on Boot" option. Latter one controls whether Sensei should be run during boot time. If you disable it, it does not stop the engine, you'll need again to click on Stop. Most probably you clicked on "Stop", but just wanted to be 100% sure.

[krdhtet]

Dear mb,

I'm well received your link, thanks.

Currently, Sensei won't find out wifi interface.

Best regards,

[sol]

@mb Thank you for your support.
The system only uses that much cpu power when I'm fully saturating my internet connection (150mbit).
Apart from using sensei I haven't experienced any issues. But this explains the drop in my throughput for sure.

I tried stopping it by using the stop button first. Which didnt work. I was able to stop the elastic search engine using the stop button though. Then I disabled start on boot and rebooted the machine. Unfortunately this didnt disable sensei after the reboot and somehow I was able to stop it and uninstall it after a few tries.
After that I tried the install sensei on the same machine again, which resulted in an crash after the final installation. The PC wasnt accessible via gui or shell anymore and I had to reinstall opnsense.

So it seams that a machine with underpowered resources might not be able to be stoped using sensei 0.6 right now.

Cheers

[nospam]

Vapourware? Blackbox man-in-the-middle SSL password harvester?

No download links, no source code, no forums

[mb]

Hi @krdhtet,

This is done on purpose. We have an unresolved issue with the wireless adapters, so we filter them out while scanning existing interfaces.

For now, the workaround would be utilizing an external AP which would be connected to one of your ethernet ports.

I'll post an update when we're done with it.

Thank you for pointing this out. Also added to the product FAQ.

[mb]

Hi @sol,

Thank you very much for further information. Yes, under heavy CPU utilization, it looks like we've been able to re-produce the issue. I'll update the thread about the resolution.

[mb]

Hi,

Thank you for the straightforward feedback.

Vaporware?

No. Sensei is developed by Sunny Valley Networks. I'm Murat, founder of the company. Sunny Valley is a venture-backed, Delaware/US registered company, located in Sunnyvale,  California. Company website is https://sunnyvalley.io. I live in Bay Area. If you are around or will be one day, I'd very much like to meet you in person, grab a coffee and have a chance to get to know each other closer.

No download links?

Currently, we provide the download link for people who register for the BETA early access program. When we are done with the early issues reported by BETA users,  we'll release the final community edition, which will be downloadable directly from the website.

No forum?

We're quite new. We've released the BETA version in late July. We thought that it would be most efficient if we used the existing OPNsense forum for that purpose. Because the plugin is available for OPNsense, and this forum is where all the people discuss things around OPNsense.

No source code?

Sensei is closed source. We announce it on the product webpage. On the other hand, apart from Sensei community edition being available for free for the community, we have a list of open source contribution items, which we think will be of value to the whole project and the community.


Password harvester?

No. Sensei follows best practices implemented by Bro/Suricata; explicitly strips out and throws away octets that could be sensitive. For instance, it does not touch HTTP bodies,  and spends extra cpu cycles to strip out any parameter passed to GET/POST requests and cookies.

It is about our effort to tackle the increasing utilization of encryption by the recent cyber attacks to avoid detection:

https://www.wired.com/story/phishing-schemes-use-encrypted-sites-to-seem-legit/
https://www.thesslstore.com/blog/lets-encrypt-phishing/

However we also share your concern. We also agree that TLS code should be distributed in a more controlled way. This is why TLS will be part of the Enterprise edition.

Thank you for taking the time and comment.

[mb]

Hi @sol,

It looks like we've fixed the problem which in some cases leads to Sensei not stopping appropriately.

Fix will appear in 0.6.0-release, which will be released today US Pacific time.

Would be more than happy if you can give it a try.

[Nekromantik]

im interested in trying this out
I only have a 80/20 connection and am using a Celeron dual core mini pc with 4GB RAM.
Will this be too much for my hardware?

[mb]

Hi @Nekromantik,

Thank you very much for your interest in Sensei.

Yes, unfortunately this hardware configuration will be insufficient for running the software. Sensei installer will refuse to start. You'll need at least 8GB RAM and a more modern CPU.

Please see this blog post to get more information:

https://www.sunnyvalley.io/blog/sensei-hw-requirements

[samsonmcnulty]

I just replied to your email with the download link to v .6 and didnt realize that the hardware requirements had changed.

Code Select Expand

This is Awesome! But I have one small request. I use a system with 12 GB ram now for my opnsense install. Previously, I was using 16 GB since sensei requires it but I never noticed my ram usage go over 8 GB. My environment is only about 4 users with maybe 20 total devices connected at once but rarely being used all at the same time (think SOHO network). Is there any way to add an option for a smaller network like mine or is there some way I can bypass the 16GB minimum requirement?

Am I totally tripping here? have they always been 8GB minimum? I could have sworn when I tried to install the last version it stopped me since I only had 12 GB... I'm probably crazy lol

[mb]

Hi @samsonmcnulty,

Great to hear that it worked at your second try :) Yes, the check in the installer was for 8GB minimum RAM. I guess it was something else which went wrong.

[Alphakilo]

Is it required to run the Elastic stack on the Firewall?
Why not split it into two packages: The "Firewall" part and then Elasticsearch, Logstash, etc...