Sensei on OPNsense - Application based filtering

[mb]

Hi @franco, thank you very much. I hope this will be of some help to the project.

We're still testing the code in HEAD. After we're confident, it's going to be MFC'd to 11-STABLE. I'll be pinging you once we're done with that. 

I've been informed that we actually have the unmodified file (.default) with the package. Engine reads a "processed" version of that file, which -indeed- do not need to be included with the package. We're removing it. I guess we're done then.

[franco]

Yeah, that's all sorted then, great! 

[therec]

Hello Murat,
I had a question around blocking. (i.e. adds, trackers, etc.). is there a way to allow a specific site? if i go to neweggs web site, the site is unreadable. if i disable the blocking, its ok again. i see the option to the right to unblock, but it wants to unblock the group (ad category) and not the site. forgive me if I've missed something simple. and thanks for the work, this is a wonderful product, I cant wait to see where you take it.

Thanks
Robert

If i posted this in the wrong place, let me know and ill move it

[mb]

Hi Robert, @therec

Thank you very much for your feedback. Awesome to see you've found the plugin useful.

When you browse Reports -> Security->Session Explorer, see if the site is being blocked via Application filtering or Web filtering. You can differentiate it by looking at the "Block category" information. If by Application filter, it says "Application category", if via Web filtering, it reads "Web category".

To allow a specific "Application", just go to Application Control, find and expand the related category, find your specific application, and unblock it.

If the filtering is done via Web filtering, browse to Web Controls->User defined categories. Create a new category i.e. Whitelist, and put your websites which you want whitelisted here.

Click "Save Changes" and that should be it.

Thanks,
Murat

[therec]

Thanks, that makes a lot of sense. however it doesn't seem to be working. I've added

- https://www.newegg.com/
- secure.newegg.com/
- www.newegg.com/
- www.neweggbusiness.com/
- https://newegg.com

Maybe ive misses something?

as an alternate test i confirmed http://static.hotjar.com/ was blocked (webtracking site).
I added this to the web controls as requested (user defined group) and is had the green check (allow),
This site also remains blocked after whitelisting via web address.

I suspect im missing something, I have amatuer firewall skills at best. but i love this product and hope its a long term solution for me. thank you for the help

P.S. i just noticed https://flash.newegg.com works just fine.

[mb]

Hi @therec,

Let's dig a little deeper together. I'll be writing to you privately. I might need some logs. Let's see if there's something wrong or there is a configuration problem.

 

[manjeet]

Hi, Using Sensei plugin and its great. Need help in few thing:
1. Is there any live view to know which IPs are using the most bandwidth and then drill down to specific IP to check what sites / services it is using and which site / service is consuming the most. (I use ntopng and it has very nice view to tell which devices are consuming most bandwidth only)
2. I do not know why but when i check the "Table of Local / Remote hosts, it shows bytes in / out which is very low as compared to ntopng. and i have avg 25-30 devices running all the time out of 50 devices but it only show few so how can i list all of them.
3. Is there any way to get all the web history of a user or users ?
4. Is there any way to bind names to IP (local / LAN IP) as squid does in web proxy ?
5. It filter web traffic and works as transparent web proxy so is there any way to use it as cache server as well or if you are planning for it in future.

[mb]

Hi @manjeet,

Thank you for sharing your experience with Sensei. We very much appreciate that. Find the answer below:

Is there any live view to know which IPs are using the most bandwidth and then drill down to specific IP to check what sites - services it is using and which site - service is consuming the most

Yep. Navigate to Sensei -> Reports -> Connections. Look for the Chart named Top Local / Remote Hosts. But make sure to select the reporting criteria as "Volume" from the upper right hand corner of the reports page. Default is by sessions. You can do "Session based", "Packet based" or "Volume based" reporting.

When you left click on any IP, a submenu appears. Click "Drill-down" and all reports will be automagically filtered according to this IP address you've selected.

I do not know why but when i check the "Table of Local / Remote hosts, it shows bytes in / out which is very low as compared to ntopng. and i have avg 25-30 devices running all the time out of 50 devices but it only show few so how can i list all of them.

My guess is that you might be viewing the "Session" reports. Make sure you've selected "Volume" as the reporting criteria.

All devices currently active should be listed though. My guess would be that you might be viewing reports for the last 15 minutes. Make sure you've selected a longer time frame from the right hand corner.

 

Is there any way to get all the web history of a user or users ?

Yes. You can do that from the Web / TLS reports. You have the drill-down capability for every report type.

Is there any way to bind names to IP (local / LAN IP) as squid does in web proxy ?

Actually, packet engine automatically maps DNS names to IP addresses if it can find a matching DNS transaction. Soon there will be Active Directory / LDAP integration which you'll be able to see the user / group names.

It filter web traffic and works as transparent web proxy so is there any way to use it as cache server as well or if you are planning for it in future.

In theory, packet engine is capable of doing that. But we chose to focus on complementing features that are currently not existing. Squid is a great caching proxy. Indeed caching is its original reason of existence. That being said, Sensei roadmap does not have "caching" as a feature.

Many thanks for reporting your experience with us.

[maekar]

Hi,

The maximum of 1000 concurrent users is an approximation for better hardware performance or an strict software limit?

thanks

[mb]

Hi @maekar ,

This is the current field-tested maximum. Software arranges several tunables (e.g. cache sizes, connection table sizes etc.) according to the user size.

Current focus is to make the software super stable for SME use cases (which generally means user populations below 1000)

Looking forward, engine is able to scale to hardware resources, which makes it possible to secure thousands of users.

Hope this answers your question.

[johjoh]

Good morning, will Sensei one day consume less resources in terms of RAM and CPU?
For example an Atom CPU or a Celeron with 4GB or 8GB of ram?

[mb]

Hi @johjoh,

Yes

A big portion of the resource requirement come from the Reporting engine (Elasticsearch). The core packet engine has been tested to run on low resource systems: e.g. Celeron  < 1GB RAM.

A roadmap feature - remote reporting - allows to run packet engine on the firewall itself, and reporting on another more powerful server.

[bobbythomas]

Hi Murat,

Couple of questions? Is there anyway to find the current installation or patch status? Where are the Sensei logs installation logs stored and how can we view that? I received an rc1 update and it's about 36MB, but it's been more than an hour since I started the installation, I would like to know the status. While installing Sensei some packages took a lot of time to get downloaded and I suspect something like that. I believe there is some latency reaching some of the repositories. Could you help me troubleshoot this issue?

Thank you,
Regards,
Bobby Thomas

[shrdlu]

Not sure if this is just my setup but after upgrading to OPNsense 18.7.8 I get stuck in a loop that won't complete.  Because it reset my configuration of Sensei* after the OPNsense 18.7.8 upgrade, I have to go through the config wizard again and when I click finish, it attempts to configure everything but kicks out the attached error.  Essentially it tells me, "error indices could not be created," and I am stuck in that loop as it returns me to the beginning of the config wizard.

So, #1, is it just me?
and #2, assuming it is not me and before I simply try to uninstall/reinstall, any ideas?

Thanks
 
*Is it normal for an OPNsense upgrade to reset my Sensei configuration?  If the answer is yes, that is fine but also if there is a way to backup a config and restore it that would help me retain settings.  Either way, love the solution and reconfiguration is actually a minor thing in the grand scheme of things so if the answer is no here then that is fine as I still find huge value in the software.

Thanks

[mb]

Hi @bobbythomas,

/tmp/sensei_update.progress should have more detail regarding the update process. 36MB download shouldn't take that long.

We rolled back rc1 update in case there is something we miss with the update process.