Sensei on OPNsense - Application based filtering

[lbakyl]

Hi Murat and the team!

First of all, I wanted to praise you for a great innovation technology that you have worked on hard! I am testing a free version for home use but will consider deploying a paid version in work environment once thoroughly tested.

While in the free version, I blocked some hostnames while viewing live traffic. It was just for a test and now I would like to remove that. Yet I do not see any blacklist on whitelist. I do not mind finding it manually on the FreeBSD system if you tell me where I can find it.

In addition, it would be great to see a comparison with older NIDS tools like Snort (that runs with PulledPork, Barnyard2 and BASE).

Yours,
Jan

[Darkopnsense]

hi @lbakyl,

The functionalities to manage your own blocklit and whitelist lists are in
sensei-> web controls => user defined categories
auto blocklist hosts (edit)
auto whitelist hosts (edit)

Regards,

[mucflyer]

Good evening
Does Sensei support IPv6 ?

[mb]

Hi @mucflyer,

Yes, Sensei supports IPv6.

Cloud servers serving over IPv4 are able serve both IPv4 and IPv6 queries.

[sol]

I really like sensei so far.
I'm using dnscrypt proxy and sensei cannot resolve local hostnames.
As mentioned a few weeks before, is their an option in the roadmap for sensei to have their own resolver?

[packetmangler]

So I've had Sensei working really well for the past couple of months but it seems that reports aren't working or they're not working the way I think they should.

So I've got a host on my network that I want to get detailed info about.  I go to Sensei -> Reports and then click add filter [host] and enter the IP address I want to include.  Lastly, I click Refresh.

When I do this I the graphs don't really change nor does the output for Activity Explorer.  It seems that the filter isn't working for me.  There's no change when I use that same IP address for Source IP as a filter.

Am I missing something obvious? Should the reports only include the any applied filters and exclude anything that doesn't match?

Thanks!

[meazz1]

I have been using sensei  home version for last few month and really loving it.
Along with Sensei I have configured Unbound DNS using CloudFlare, Quad9, Clean Browsing, Google, and Frenom public DNS resolvers (https://sahlitech.com/opnsense-setup-unbound-dns/).
Is this an over kill or a redundant or unnecessary setup?

[packetmangler]

I have been using sensei  home version for last few month and really loving it.
Along with Sensei I have configured Unbound DNS using CloudFlare, Quad9, Clean Browsing, Google, and Frenom public DNS resolvers (https://sahlitech.com/opnsense-setup-unbound-dns/).
Is this an over kill or a redundant or unnecessary setup?

I personally limit myself to a single forwarder only because having multiple might make troubleshooting issues harder. 

My flow is: pi-holes -> quad9 -> Sensei.  And even then it's a matter of tracking down whether or not it's the pi-holes, quad9 or Sensei doing the blocking. Then add in the pressure of family saying they're unable to get to a super important site right now or something on their precious iOS device isn't working quite right.  It can be a pain. :D

[meazz1]

I have been using sensei  home version for last few month and really loving it.
Along with Sensei I have configured Unbound DNS using CloudFlare, Quad9, Clean Browsing, Google, and Frenom public DNS resolvers (https://sahlitech.com/opnsense-setup-unbound-dns/).
Is this an over kill or a redundant or unnecessary setup?

I personally limit myself to a single forwarder only because having multiple might make troubleshooting issues harder. 

My flow is: pi-holes -> quad9 -> Sensei.  And even then it's a matter of tracking down whether or not it's the pi-holes, quad9 or Sensei doing the blocking. Then add in the pressure of family saying they're unable to get to a super important site right now or something on their precious iOS device isn't working quite right.  It can be a pain. :D

This is the exact issue I'm facing, "is internet down? I can't open this site" from a family member alsmost everyday.

Can you  explain your flow? The Pihole doing the DNS using 9.9.9.9 and sensei doing web and app filtering/

[packetmangler]

This is the exact issue I'm facing, "is internet down? I can't open this site" from a family member alsmost everyday.

Can you  explain your flow? The Pihole doing the DNS using 9.9.9.9 and sensei doing web and app filtering/

Sure. 

Since I run my own internal DNS server, I don't allow clients to connect to external DNS servers over port 53.  I have OPNsense redirect any queries to my pi-holes. This applies to ipv4 and ipv6.

I run two pi-holes and both point to my internal dns server.  the internal dns server then connects to quad9 via stubby (uses DNS over TLS) for any further queries. 

Assuming clients resolve their queries OK (and don't get denied by pi-holes), they then go through Sensei for further web / app filtering and then out to the Internet.

[mb]

I have been using sensei  home version for last few month and really loving it.
Along with Sensei I have configured Unbound DNS using CloudFlare, Quad9, Clean Browsing, Google, and Frenom public DNS resolvers (https://sahlitech.com/opnsense-setup-unbound-dns/).
Is this an over kill or a redundant or unnecessary setup?

Hi @sol, many thanks for your feedback.

This has become one of the most wanted feature request (what we call in-flight dns query). We've added this to the road-map and should appear sometime around Q2-Q3 this year.

Quick update:
For remote IP addresses, even if Sensei cannot see DNS transactions, it should still be able to map hostnames with IP addresses if the session is HTTP/TLS/QUIC (since there are other places where we can extract hostnames)

For local IP addresses <-> hostnames mapping, in-flight dns reverse query feature will do the trick.

[mb]

So I've had Sensei working really well for the past couple of months but it seems that reports aren't working or they're not working the way I think they should.

So I've got a host on my network that I want to get detailed info about.  I go to Sensei -> Reports and then click add filter [host] and enter the IP address I want to include.  Lastly, I click Refresh.

When I do this I the graphs don't really change nor does the output for Activity Explorer.  It seems that the filter isn't working for me.  There's no change when I use that same IP address for Source IP as a filter.

Am I missing something obvious? Should the reports only include the any applied filters and exclude anything that doesn't match?

Hi @packetmangler, thank you very much for the report. We couldn't reproduce this in our lab.

Any chances that you can create a bug report (Report Bug menu on the upper right hand corner).

Let's take a closer look.

[mb]

Hi @meazz1, hi @packetmangler, thanks for sharing your setup, very much helpful.

For troubleshooting, a quick note:

If Sensei is blocking a connection, it should be reporting that in Reports -> Blocks.
Reports -> Live Blocked Sessions Explorer displays this information on a per-connection basis.

[mb]

Dear Sensei users,

I hope everyone is at home and staying healthy. During the Corona days, Sensei team was mostly busy with 1.5 features.

1.5 is in pilot tests right now, and will be most likely released late this month.

Here is the Release Notes for this upcoming release:

What is new in Sensei for OPNsense Release 1.5

Application Control
Application Database will be a seperate package and will be updated independently and more frequently

• New feature: More frequent (e.g. weekly) application database updates
• New feature: User-defined application signatures
• New feature: QUIC v46 support
• Improved app detection logic
• 100+ new application signatures

Privacy and Compliance

• New feature: Ability to anonymize local / remote IP addresses
  
• New feature: Ability to disable Username / DNS enrichments
  
• New feature: Ability to selectively delete reports for specified IP addresses
  

Policies and Filtering

• New feature: Multiple time schedules for a single policy
• New feature: Tool tips for policy screens
• New feature: Policies can now match inbound/outbound flows selectively (You can specify flow direction for Policy Configuration)
• New feature: Ordering and prioritizing policies
• New feature: Sensei can now inspect and filter Proxy-ed connections (CONNECT method - Not transparent Proxy)
• Improved Ad Blocking (Especially for Android mobile devices / Google Chrome mobile browsers)
• Fix: Whitelisting for App Controls issue is fixed
• Fix: Over-night time schedules
• Fix: Engine reloading (during rule updates) issue is fixed
• Fix: Mongodb Backend: Enlarged Charts can now pull data for all "Top" queries

Reporting

• New feature: You can now specify an external Elasticsearch instance for the main reporting database
• New feature: You can now select the Backend Database Engine during initial configuration
• New feature: Scheduled Reports: PDF Reports
• New feature: Scheduled Reports: Weekly e-mail reports
• New feature: Ability to provide an "exclude filter" for "Add filter" functionality
• New feature: Ability to move Reporting Database to a different directory (To be able to move database from a tempfs e.g. /var partition)
• New feature: Read-only access to reports: you can now restrict a OPNsense UI user to only be able to view reports (Select Dashboard permission)
• New feature: Ability to re-order/re-organize report charts

Cloud
Improved feedback loop for Web Categorization:

When you submit an entry for re-classification we can now re-categorize it within as fast as 10 minutes. Re-categorized web sites may become available via Cloud as soon as 15 minutes. You can submit web sites for re-classification either through our Web site (https://www.sunnyvalley.io/site-classification/) or through the Sensei UI when you add a site to whitelist/blacklist or to a user defined category.

• Optimized Cloud Query Caching
• Fix: case sensitive queries

Integrations

• Improved MS Active Directory caching performance

Other

• New feature: Configuration Backup and Restore
• New feature: Health: You can now specify your own threshold for SWAP high utilization ratio
• New feature: Health: Check and warn if reporting database is located on a tempfs
• Improvement: Install/Configuration: You can now re-try hardware compatibility check in case first try fails
• Other performance and reliability improvements

Stay safe,
Your Sensei team
https://sunnyvalley.io/sensei
https://help.sunnyvalley.io

[packetmangler]

Hi @packetmangler, thank you very much for the report. We couldn't reproduce this in our lab.

Any chances that you can create a bug report (Report Bug menu on the upper right hand corner).

Let's take a closer look.

Report sent!  I'm truly expecting a reply along the lines of:  You're doing something really stupid.  So don't do that. :D

Looking forward to the release of 1.5!

Tbanks!