OPNsense Forum

English Forums => Development and Code Review => Topic started by: mb on August 25, 2018, 03:38:14 am

Title: Sensei on OPNsense - Application based filtering
Post by: mb on August 25, 2018, 03:38:14 am
Hello,

I'm Murat, founder of Sunny Valley Networks, the company behind Sensei.

Very much pleased to meet the OPNsense community.

I've seen a thread about Sensei in the forum, so I thought it might be a good idea to start a dedicated topic to help people with the software.

Sensei is a plugin for firewalls which complement them with features like Application Filtering, Advanced Network Visibility and Cloud Application Control. Currently, Sensei community edition is available for OPNsense platform.

I've seen that some members have already downloaded and trying Sensei. Many thanks for that. We're grateful.

I've created this topic about Sensei to help you to try it out, and try to solve any problems you guys might have encountered.

Although we reached our target number of beta testers, we always have room for forum members.
If you're interested in trying it, please do not hesitate to contact me privately. I can share the URL to the latest installer.

Very much looking forward to reading your feedback and helping you with the software.

More information about Sensei can be found on the product web page: https://sunnyvalley.io/sensei

All the best
Murat
Title: Re: Sensei on OPNsense - Application based filtering
Post by: marjohn56 on August 26, 2018, 12:05:21 pm
Thanks to @mb for sending me a link to test this. This is a quick summery of my first impressions, also to prevent any cross-contamination issues I did a clean install using zfs and then bootstrapped opnsense install. Firmware flavour is development and core upgrade carried out.


Installation was straight forward as was configuration. Initial configuration left me with zero information, this appears to be because I had selected the LAN as the interface to monitor, however, my LAN is a bridge, changing this to the OPT1,OPT2,OPT3 interfaces solved this and then it all started working well.


Note I am using this on a Qotom i5 with 8Gb RAM. It is recommended that this is the minimum requirement for a 100 user system. On my test system there is minimal extra load on the CPU, but my test system is limited to only two devices attached to the LAN.


My first impressions are that is a very impressive package, it will be interesting to see what the differences will be between the commercial and community editions are when that time arrives.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on August 27, 2018, 07:43:54 am
@marjohn56, many thanks for giving Sensei a try and providing feedback. This is very valuable for us.

Glad to hear that installation & configuration went smooth.

Sensei utilizes netmap behind the scenes, which does not play well with bridged interfaces. Netmap in FreeBSD 11.x, which OPNsense is based on is quite old.  I think we can also contribute to OPNsense team with an improved netmap support. I believe this will also help resolve some Suricata issues.

We'd love to hear about performance figures with a larger user base if you happen to have access to one. Currently the largest deployment we know of is 200 Mbps sustained WAN throughput with about 850 users. HW is an old HP DL360-g8 (xeon e5-2450L @1.8GHz) and 16GB RAM.

Delighted to see that product is up to the duty.

Enterprise <-> Community edition work is ongoing. For now I'm happy to tell that community edition for OPNsense will always be there and forever free.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: Mundan101 on August 29, 2018, 02:01:30 pm
I have sensei up at running and so far so good!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: marjohn56 on August 29, 2018, 03:10:48 pm
I have sensei up at running and so far so good!


Just in case @mb has not told you, IPv6 is still WIP, so v4 only for now, still cool though  :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on August 30, 2018, 01:18:22 am
@Mundan101, thank you for testing and giving feedback.

@marjohn56, thank you for pointing it out. It's been FAQ'd now :)

To better support the software and help people who are having issues, we've created a Gitlab project.

Please feel free to send any bug-reports & enhancement requests there:

https://gitlab.com/svn-community/opnsense-sensei-plugin

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mimugmail on August 30, 2018, 09:16:18 am
@mb https://www.sunnyvalley.io/eastpect
What about TLS 1.3?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: svn on August 31, 2018, 01:10:20 am
Hi @mimugmail,

I am Hayati from SVN team.

As you probably know that TLS 1.3 has been finalized in this month after 28 drafts. TLS 1.3 will obviously dominate over other versions and most of the Linux/Unix distros and libraries should be giving support for it, sooner or later. This is no different for us.

We've been closely watching its progress and discussions on the TLS working group during our whole product development. So we expected and prepared for it, and Sensei's TLS inspection has been designed by taking TLS 1.3 into account. We'll be able to provide TLS 1.3 inspection without downgrading TLS version.

We expect the transition to TLS 1.3 in the field will start with the popular tls libraries following with the applications that are dependent on them. This will take some time. We target to be among the first network security providers to support TLS 1.3 with its most potential.

I've uploaded a video to SVN youtube channel illustrating TLS Inspection in action: https://www.youtube.com/watch?v=krG_VKt2_qk
Title: Re: Sensei on OPNsense - Application based filtering
Post by: samsonmcnulty on September 01, 2018, 12:12:45 am
Thanks you guys! I don't have a large userbase but I'll definitely report anything I come across. So far I really like it. My main goal at the moment is to see how it plays with squid and caching. I'm also using suricata and clamAV. I noticed a mention of some issues with suricata but that you were aware and working on a fix.
Edit I've seen a few people on 200Mb connections but I haven't seen many at 1Gb. Are you planning to add traffic shaping abilities? based on category?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 01, 2018, 03:46:59 pm
Hi @samsonmcnulty,

Thank you for testing & feedback. I'd very much appreciate if you can report any problems and/or issues you encounter.

Just like filtering based on application, shaping will also be there ;) Tentative plans is that we expect it to arrive in 2019.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sagem2004 on September 01, 2018, 04:37:58 pm
hello

can we block websites can be an integration in opnsense native

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 01, 2018, 07:58:14 pm
Hi @sagem2004,

Was your question about Sensei filtering based on web sites?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sol on September 03, 2018, 10:12:19 am
Great plugin so far.

On my machine running with 8GB RAM and an Intel I5 5250U (2x 1,6GHZ) the WAN throughput is at approx. 85 Mbps using IPS, Proxy + AV and around 8 active users.
Without Sensei my box can use the full 150 Mbps line (Cpu load is around 60 - 70%).
It takes a while to load on the first time and for some reason I cannot disable Sensei.
Due to the reduced internet speed I had to uninstall it and will give it another try once I have a faster router.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 03, 2018, 12:54:09 pm
Hi @sol,

Thank you for trying out Sensei and for the feedback.

A couple of questions:

Is this CPU usage (60-70%) for the configuration Sensei is not running? (e.g. IPS+Proxy+AV) ?

When you launch Sensei, how much did you see it changed? Does it top to 100%?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: krdhtet on September 04, 2018, 09:42:14 am
Dear mb,

Could you kindly provide Sensei link for me?

Thanks you.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 04, 2018, 07:31:35 pm
Hi @krdhtet,

You got it in your inbox ;)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sol on September 05, 2018, 06:02:06 pm

A couple of questions:

Is this CPU usage (60-70%) for the configuration Sensei is not running? (e.g. IPS+Proxy+AV) ?

Yes

When you launch Sensei, how much did you see it changed? Does it top to 100%?
 It goes up to 95% and drops to ~50%. It also drops and peaks way more often


Furthermore I couldnt disable Sensei and I was only able to uninstall it right after a reboot. 
After a new try to install it again over the current system opnsense crashed and it had to reinstall Opnsense.
I guess some old settings made a clean reinstallation of Sensei impossible.
Lets hope that a new Sensei version will fix the option to stop it.

Looking forward to an update and will give it a try another time.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 05, 2018, 07:28:31 pm
Hi @sol,

Many thanks for reporting this and for the answer. This is very much helpful to understand what's going on.

Looks like a quite loaded system. I would not recommend running with a 60-70% cpu utilization if you're doing some kind of packet processing. Because packet processing requires dedicated resources and if the cpu is highly utilized and also shared with other applications, it's highly possible that you'll start losing packets. This is so, because at some point OS will fail to schedule the packet processing application to a CPU (because the CPU is already busy) and packets will be dropped in this timeframe. As a consequence,  this will create congestion, and finally you'll get lower throughput. This was what happened, lowering your throughput from 150 - 85 Mbps.

To remedy this kind of heavy load scenarios, there is one thing you can do, and one thing we can:

For you, as you wrote before, it'd be better if you can run the configuration with a more resourceful HW.
For Sensei, we'll pin it to a dedicated CPU core. This will help if you have a multi-core system. 

For not being able to stop Sensei, I'd guess it's related to the above scenario. Though it should stop anyway whatever the load is.

We'll try to reproduce this with your conditions in our lab. I'll let you know about our results.

For the sake of clarity: were you trying to stop it by clicking on the  "Stop" action button or by disabling "Start on Boot" option. Latter one controls whether Sensei should be run during boot time. If you disable it, it does not stop the engine, you'll need again to click on Stop. Most probably you clicked on "Stop", but just wanted to be 100% sure.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: krdhtet on September 07, 2018, 10:35:12 am
Dear mb,

I'm well received your link, thanks.

Currently, Sensei won't find out wifi interface.

Best regards,
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sol on September 07, 2018, 05:49:57 pm
@mb Thank you for your support.
The system only uses that much cpu power when I'm fully saturating my internet connection (150mbit).
Apart from using sensei I haven't experienced any issues. But this explains the drop in my throughput for sure.

I tried stopping it by using the stop button first. Which didnt work. I was able to stop the elastic search engine using the stop button though. Then I disabled start on boot and rebooted the machine. Unfortunately this didnt disable sensei after the reboot and somehow I was able to stop it and uninstall it after a few tries.
After that I tried the install sensei on the same machine again, which resulted in an crash after the final installation. The PC wasnt accessible via gui or shell anymore and I had to reinstall opnsense.

So it seams that a machine with underpowered resources might not be able to be stoped using sensei 0.6 right now.

Cheers
Title: Re: Sensei on OPNsense - Application based filtering
Post by: nospam on September 07, 2018, 10:47:10 pm
Vapourware? Blackbox man-in-the-middle SSL password harvester?

No download links, no source code, no forums
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 07, 2018, 10:57:45 pm
Hi @krdhtet,

This is done on purpose. We have an unresolved issue with the wireless adapters, so we filter them out while scanning existing interfaces.

For now, the workaround would be utilizing an external AP which would be connected to one of your ethernet ports.

I'll post an update when we're done with it.

Thank you for pointing this out. Also added to the product FAQ.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 07, 2018, 11:58:12 pm
Hi @sol,

Thank you very much for further information. Yes, under heavy CPU utilization, it looks like we've been able to re-produce the issue. I'll update the thread about the resolution.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 08, 2018, 07:14:16 am
Hi,

Thank you for the straightforward feedback.

Vaporware?

No. Sensei is developed by Sunny Valley Networks. I'm Murat, founder of the company. Sunny Valley is a venture-backed, Delaware/US registered company, located in Sunnyvale,  California. Company website is https://sunnyvalley.io. I live in Bay Area. If you are around or will be one day, I'd very much like to meet you in person, grab a coffee and have a chance to get to know each other closer.

No download links?

Currently, we provide the download link for people who register for the BETA early access program. When we are done with the early issues reported by BETA users,  we'll release the final community edition, which will be downloadable directly from the website.

No forum?

We're quite new. We've released the BETA version in late July. We thought that it would be most efficient if we used the existing OPNsense forum for that purpose. Because the plugin is available for OPNsense, and this forum is where all the people discuss things around OPNsense.

No source code?

Sensei is closed source. We announce it on the product webpage. On the other hand, apart from Sensei community edition being available for free for the community, we have a list of open source contribution items, which we think will be of value to the whole project and the community.


Password harvester?

No. Sensei follows best practices implemented by Bro/Suricata; explicitly strips out and throws away octets that could be sensitive. For instance, it does not touch HTTP bodies,  and spends extra cpu cycles to strip out any parameter passed to GET/POST requests and cookies.

It is about our effort to tackle the increasing utilization of encryption by the recent cyber attacks to avoid detection:

https://www.wired.com/story/phishing-schemes-use-encrypted-sites-to-seem-legit/
https://www.thesslstore.com/blog/lets-encrypt-phishing/

However we also share your concern. We also agree that TLS code should be distributed in a more controlled way. This is why TLS will be part of the Enterprise edition.

Thank you for taking the time and comment.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 12, 2018, 05:51:59 pm
Hi @sol,

It looks like we've fixed the problem which in some cases leads to Sensei not stopping appropriately.

Fix will appear in 0.6.0-release, which will be released today US Pacific time.

Would be more than happy if you can give it a try.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: Nekromantik on September 13, 2018, 12:17:52 am
im interested in trying this out
I only have a 80/20 connection and am using a Celeron dual core mini pc with 4GB RAM.
Will this be too much for my hardware?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 13, 2018, 02:12:38 am
Hi @Nekromantik,

Thank you very much for your interest in Sensei.

Yes, unfortunately this hardware configuration will be insufficient for running the software. Sensei installer will refuse to start. You'll need at least 8GB RAM and a more modern CPU.

Please see this blog post to get more information:

https://www.sunnyvalley.io/blog/sensei-hw-requirements
Title: Re: Sensei on OPNsense - Application based filtering
Post by: samsonmcnulty on September 13, 2018, 02:50:10 pm
I just replied to your email with the download link to v .6 and didnt realize that the hardware requirements had changed.
Code: [Select]
This is Awesome! But I have one small request. I use a system with 12 GB ram now for my opnsense install. Previously, I was using 16 GB since sensei requires it but I never noticed my ram usage go over 8 GB. My environment is only about 4 users with maybe 20 total devices connected at once but rarely being used all at the same time (think SOHO network). Is there any way to add an option for a smaller network like mine or is there some way I can bypass the 16GB minimum requirement?
Am I totally tripping here? have they always been 8GB minimum? I could have sworn when I tried to install the last version it stopped me since I only had 12 GB... I'm probably crazy lol
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 13, 2018, 06:21:44 pm
Hi @samsonmcnulty,

Great to hear that it worked at your second try :) Yes, the check in the installer was for 8GB minimum RAM. I guess it was something else which went wrong.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: Alphakilo on September 15, 2018, 04:47:59 pm
Is it required to run the Elastic stack on the Firewall?
Why not split it into two packages: The "Firewall" part and then Elasticsearch, Logstash, etc...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 15, 2018, 07:24:13 pm
Hi @Alphakilo,

Many thanks for the input.

Currently it runs on the firewall. This was an important decision to make when we first started working on the plugin. All of the first users' feedback was to have it coupled with the firewall. Because the deployments were typical of a SOHO, SME, and they were not able to operate a separate deployment just for reporting.

So instead of starting with a distributed design, we started with this one, suggesting early users to increase the amount of memory they had. They were already using modern CPUs, so CPU was not a problem.

For a reference, with the current architecture, the largest deployment that we are reported is  700+ concurrent users and 500 Mbps/50 Mbps max, 300 Mbps sustained WAN throughput. HW: Dual-Core i5-2400 @3.10 GHz (4 threads) with 10GB RAM - OPNsense + Sensei. No IPS, No AV, No Caching. Use case is firewalling + application control + web security.

Looking forward, it looks like we'll offer this option. Since we see that more and more people want to see Sensei deployed in more large scale environments, with thousands of users.

For the time being, our focus is to have the software make super stable & make it cover the essential network security requirements of SOHO / SME users.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 16, 2018, 04:13:32 pm
Hi there,

Sensei 0.6.1 is released. This is a minor reliability release fixing a few issued reported for 0.6 release.


More on how to update to 0.6.1: https://www.sunnyvalley.io/blog/sensei-0-6-1



Title: Re: Sensei on OPNsense - Application based filtering
Post by: bulmaro on September 26, 2018, 12:00:18 am
Hi friends, thanks for the very interesting project work,
I'm testing version 0.6.1, my interface is vlan but I do not see Packets IN and Packets OUT, any settings I missed?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 26, 2018, 07:16:52 am
Hi @bulmaro,

@svn is working on your bug report. Hope to update you about this soon.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: bulmaro on September 26, 2018, 03:45:52 pm
thanks for your attention
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hyralak on September 29, 2018, 07:25:46 pm
I tested Sensei for a couple weeks. In that time I observed some unexpected behavior. First i need to say that I have had zero issues with opnsense in the year that i have been running it, rock solid. I am running it at home, my internet speed is 300/80. The hardware is a Dell Optiplex 8gb ram Intel(R) Core(TM) i5-3475S CPU @ 2.90GHz. Memory usage never exceeded 35% with sensei running and cpu usage was minimal. 
Issues I encountered after installing Sensei included web interface locking up, and unable to access opnsesne via ssh. I could still interact with the console. After this occurred i had to uninstall the plugin.
 
Also, I run a pi-hole for DNS poisoning which logged Sensei as the top domain. I was seeing 25,000-35,000 connection attempts to updates.sunnyvalley.io. I turned off auto updates but it continued to hammer away at updates.sunnyvalley.io. The screenshot below is from the last 24 hours. I uninstalled Sensei about 13 hours ago.

(https://i.imgur.com/nYv8rJw.jpg)

I liked the visibility and functionality that Sensei offered, but the instability was not acceptable. Perhaps my hardware is not adequate for the plugin?
Hopefully the information that the plugin sent back to sunnyvalley will provide insight into my web/ssh issues.
Keep up the great work and thanks for letting me try out the plugin. Perhaps I will try again at a later date.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 01, 2018, 08:29:22 pm
Hi @hyralak,

Many thanks for taking the time and reporting your issue. If you find value in Sensei, than it's our job to make it super stable.

Your Hardware configuration is just fine. CPU/memory utilization seems to be low & as expected.

Do you remember which Sensei version you installed first? Because the symptoms you're seeing, we had an issue which might be causing them, and was fixed at 0.6.1 release. I'm suspecting an upgrade issue.

Updates.sunnyvalley.io is being used by two purposes:

1. If you enabled Automated health-checks, it collects these info and sends them to the updates server, which we run a monitoring service with alerting capability (It's actually nagios). This way we instantly know that some Sensei instance has a problem, and try to diagnose it. Information that's sent:
    a) Check whether the packet engine is currently running
    b) Check whether the packet engine crashed and created any core files
    c) Check whether the Sensei engine has any issues with packet forwarding
    d) Check whether Elastic Search is running & healthy
    e) Check whether Sensei is utilizing any SWAP memory
    f) Check disk free space has at least 20% free.
    g) Check if Sensei is using excessive cpu/memory
    h) Check if Elastic Search is using excessive cpu/memory
    i) Check if overall load average is within safe limits
    j) Check if overall cpu/memory consumption is within safe limits
    k) Check if Sensei is put onto bypass mode because of a problem.

System health checks are done once a minute. Instead of collecting the information and sending in batch mode, health script connects to the server for every one of the checks. So this makes 11 connections for a minute. This is why you see some many connections. Yep, this is inefficient & we have an open JIRA issue to address this.

2. Software update checks. If you enable update checks, they are done once an hour.

Though the number seems to be double the number we should be seeing. Our guess is that there is a runaway cron job from previous versions.

I'd love to explore more, I'll be writing to you via a private message. I'd like to find the root cause relating to this. Than fix is the easy part :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hyralak on October 01, 2018, 08:39:12 pm
It appears that I installed sensei_installer_opnsense_0.6.1-release.sh
Title: Re: Sensei on OPNsense - Application based filtering
Post by: rhyse on October 02, 2018, 10:55:41 am
Hi

I am seeing an issue where the "Sensei Packet Engine" keeps stopping, clicking start makes it come back to life.

Enviro: VMware 6.7, 10GB RAM, 2 x vCPU's (host CPU 2 x  E5-2670) , disk space 2.2 gb used out of 18Gb , Sensie deployment size Small (I have just enabled " Enable Generation of Support Data:" ), Sensei version 0.6.1-release (installed from this version)

This is a test infra, so doesn't have much traffic going through it

Any ideas ?

Thanks
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 02, 2018, 04:00:12 pm
Hi @rhyse,

We did not have much users on VMware. Let's debug it together & make Sensei run there. I'm contacting you.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 02, 2018, 08:58:01 pm
Dear Sensei users,

@rhyse helping us debug his issue, we've spotted a bug with Netflow output formatter. If you're using Sensei with Netflow, better to disable it for now.

For the resolution, we'll issue a fix. Hopefully as 0.6.2.

Many thanks @rhyse !.




Title: Re: Sensei on OPNsense - Application based filtering
Post by: Csykes27 on October 16, 2018, 12:29:16 am
I am having an issue of when I Enable Cloud Reputation & Web Categorization all web traffic stops. all services are running and stay running from what I can tell.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: svn on October 16, 2018, 12:49:50 am
@Csykes27 thanks for reporting. We've heard this issue for the first time actually. Let's debug what is causing this together.

I shall be contacting you soon to resolve the issue.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: jjanzz on October 18, 2018, 10:48:08 am
During the initial installation, a dependency throws a 404 error:

Code: [Select]
pkg: https://updates.sunnyvalley.io/repo/libXtst-1.2.3.txz: Not Found
FAILED : Unable to install required packages. Please see install.log
Title: Re: Sensei on OPNsense - Application based filtering
Post by: svn on October 18, 2018, 07:11:19 pm
Hi @jjanzz,

We're currently working with OPNsense for the integration. As a part of this process, OPNsense started providing some of our dependencies. We'll update our installer and package to be compatible with the latest changes on OPNsense package repository, soon.

It seems that some of the dependencies not satisfied (namely, some configuration files of elasticsearch, and some java dependencies). We'll fix this urgently.

Right now, you can register for download and we'll send you a download link as soon as we fix the dependency issue.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: jjanzz on October 22, 2018, 04:10:32 pm
We're currently working with OPNsense for the integration. As a part of this process, OPNsense started providing some of our dependencies. We'll update our installer and package to be compatible with the latest changes on OPNsense package repository, soon.

Thank you for the reply. No problem; I'll gladly help you test it out as soon as it's possible :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 25, 2018, 12:09:10 am
@jjanz and community,

Elasticsearch5 was added to OPNsense packages as part of the 18.7.5 update. There was a problem in the FreeBSD elasticsearch package builds which was inherited by the OPNsense build system.

Because elasticsearch was problematic, Sensei installations were failing.

Today we fixed the problem. In the meantime, OPNsense will be removing the package from its repository in the upcoming release.

Starting 18.7.6, elasticsearch will be provided by Sunny Valley Package repository.

Long story short: We're resuming Sensei downloads. You can now download and install new Sensei version, which is 0.7.0-beta1 as of now.



Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 25, 2018, 12:10:55 am
Hello all,

As part of 0.7 release effort, we've launched Sensei Users' Manual & Documentation.

Please find it here:

https://guide.sunnyvalley.io/sensei/
Title: Re: Sensei on OPNsense - Application based filtering
Post by: wordsmith on October 25, 2018, 07:45:42 am
This plugin looks pretty interesting and I’d like to give you some non-technical feedback to consider. But first a question: will Sensei ever be open source?
See, the reason I ask is because to me it seems there is some confusing communication going on. I’m sure, some of it is non-intentional like:

Quote
For now I'm happy to tell that community edition for OPNsense will always be there and forever free.

"For now" and "always" don’t work well together. Basically, now you’re saying that this will always be the case, but later you might change your mind to “it isn’t free anymore”. I suspect that this was unintentional, but I just wanted to get it out of the way.

What rubs me a bit the wrong way is that the community edition is free, but not open source. According to your FAQ:
Quote
The Packet Engine coded in C++, and its source code is not open.

I think the reason there are community editions in the software space is precisely to indicate that a company/developer wants to build a trust model with others and, as a result, gives them the recipe so that they can build a community around it together. In short, it isn’t about getting something for free i.e. without having to pay, but to build trust.

Now, where your approach to marketing proofs to be rather problematic is with statements like this:

Quote
Empower your open source firewall with Next Generation features.

If you plan to keep parts of Sensei closed source, I’d suggest you’d drop the “open source” in your marketing, because it’s confusing at best, misleading at worst. Next, as long Sensei isn’t open source, I’d also reconsider the use of “community edition”: this is a rather well known way to describe the non-commercial version of a product that isn’t just for the community, but also by the community. If the community doesn’t have access to the code, it’s not a community edition, it’s a free edition.

The FLOSS community already suffers from a huge labeling problem (ever tried to explain to a non-technical user the difference between Free Software and Freeware?) so let’s not muddy the waters even more.

I don’t know about your business model, but for people who really care about open source it’s not about getting stuff for free, it’s to be able to verify the claims of company such as yours and, of course, to build a community around a solution that can be build by like-minded people without restrictions regarding code access.

Of course, at the end of the day there’s always the pragmatic side to consider and there will probably be a lot of users who are perfectly fine to run proprietary software on their open source OS, but for people like me who decided to use an open source solution not because it is free of charge, but precisely because it’s source code is available, Sensei won’t be the solution we’re looking for.

Now, with all that being said, I still appreciate your efforts.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 26, 2018, 08:33:09 pm
Hi @wordsmith,

Many thanks for taking the time and provide this valuable feedback. Now we become aware of a communication problem.

To clarify things:


As you’ve correctly pointed out, if there is any misunderstanding, it’s unintentional. Your comments shed a lot of light as to what needs to be adjusted in the messaging. We’ll be working on that.

Taking this chance, I’d like to give a little bit of background why we started with “open source firewalls”.

As Sensei team, we believe that we’ve created a powerful packet processing technology. We believe that better packet visibility means better decision making. Better decision making means better success rates in detecting malign traffic.

Sensei is the first of two products that we’re going to create for a large market.

We hope to make Sensei available for any network security equipment / product which needs application classification & web security features. L3-L4 firewalls, UTMs all fall into this category.

The thing we started with open source firewall space is that, it was a request by an MSP who was deploying open source firewalls onto customers and providing support services. Very happy with their current firewalls, they needed several features that we could provide. We quickly did an integration and voila! The resulting solution (OPNsense + Sensei) was found to be better than many of the current players in the UTM market.

This sparked a light for us. Why not deliver the product as a plugin instead of yet another full-blown firewall appliance? It’d be cost effective for us and we would than be able to relay this cost advantage for the benefit of our prospective users.

In this regard, open source firewalls is a delivery channel for us, though it’s not the complete target market. Via this initial channel, we learn very much from our users and improve Sensei. You can’t believe how much Sensei improved from the day we announced first beta up until this day. Then of course, we are looking for market visibility. It’s great to see people loving the solution and spreading the word.

A free of charge Sensei edition (maybe we should call this freemium edition) is a way of our giving back to the OPNsense community.

Having founded a local open source community (enderunix.org) and published some open source tools, I truly understand, appreciate and respect your stance.

Though we cannot make Sensei fully open source, I think the best we can do right now is to communicate what Sensei is and what it is not in a straight and open way. This way people would know what they will have and what they won’t; and will make an informed decision about using / not using it.

It's somewhat hard to figure out a way to communicate people that the current product is for “open source firewalls” without using the words “open source”. Because marketing wise, we would like to be as precise as possible so that people would know what it is for.

However I also see that it’s creating confusion. We’ll spend more time on this. I’d also like to consult you if you wouldn’t mind.

Again, many thanks for bringing this up to our attention.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 08, 2018, 02:24:10 pm
Dear Sensei users,

0.7.0-beta1 update is out for those who are on 0.6.x releases:

https://www.sunnyvalley.io/blog/0-7-beta1-update-available-for-0-6-x-users

0.7 Beta1 comes with the following functionality:
 
1. New Report - Blocked Connections Sessions Explorer and drill-down reporting
2. Reports enhancement: Daily executive reports. Selected reports delivered via a daily
    e-mail.
3. Customizable Landing Page for Blocked connections
4. Reports data retiring: disk space consumed by Elastic Search (Reports) is now
    configurable
5. Release Changelog is now displayed during Sensei updates
6. Shortcut to add Block/Allow rules based on fields (IP Address, Application, App
    Category etc.) via Session Explorer Reports. 
7. 350+ new applications identified.
8. Documentation: Sensei Users' Manual
9. Sensei speaks your language now, we added i18n support to match your OPNsense
    UI language. English & German are the two for now, more coming soon.
10. More performance & stability improvements


If you've downloaded & installed Sensei later than October 15, you should already be using 0.7.0 beta1. This is an update package for older versions.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: shrdlu on November 13, 2018, 06:15:37 pm
Not sure if this is the right place to post this, so if I am wrong please redirect me.

I have noticed with Sensei (BTW, it is working fine) that when I run a health audit in OPNsense I get the following (see attached screenshot) checksum mismatch for the nodes.csv file and was curious if this is normal or something is wrong.  Things appear to work fine and no matter what Cloud Threat Intel selections I make (not sure that is related but it might be) I get the mismatch and the Cloud Threat Intel is working fine regardless, or at least shows up and running.

And, on another note, in terms of processing when do the Sensei components process information in terms of order?  For example, I use the web proxy (squid) in OPNsense and was curious if Sensei process the packets before the proxy or after or somehow during, or frankly something completely different if I am misunderstanding the order of operations.

Thanks in advance.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 13, 2018, 06:27:42 pm
Hi @shrdlu,

You're in the correct place :) We're receiving feedback & comments and help requests here. You can also shoot a ticket if you think you've found an issue with the software:

https://gitlab.com/svn-community/opnsense-sensei-plugin/issues

The thing with Node.csv is not an issue. Web UI updates the contents of this file with the best servers available. I guess this creates a mismatch with the OPNsense File Integrity Checker. We'll handle that.

With regard to processing order: Sensei receives packets while they traverse from Network Adapter to the FreeBSD networking stack; which means it receives them before Squid and even before L3/L4 Filtering.

You're all welcome, and thanks for sharing your experience.

 
Title: Re: Sensei on OPNsense - Application based filtering
Post by: nasq on November 15, 2018, 03:56:00 pm
I Installed sensei. When I was on the dashboard to configure the protected interfaces only my 2 vpn interfaces show up. Not WAN, not LAN, nor any other interface on my firewall.

current version as of writing (0.7.0-beta1)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mimugmail on November 15, 2018, 04:11:55 pm
Do you have IPS enabled on LAN or WAN?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: nasq on November 15, 2018, 06:23:50 pm
Nope. Neither
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 15, 2018, 06:33:28 pm
Hi @nasq,

Any chances your LAN interface is virtio?

https://guide.sunnyvalley.io/sensei/support/faq#no-ethernet-interface-is-being-shown-in-the-interface-configuration

As quick workaround, select Intel E1000 as the adapter type.

As the final solution we're sponsoring a development which will ship the latest upstream netmap code into FreeBSD.

This will also fix lots of issues that you might be encountering with Suricata as well.

https://svnweb.freebsd.org/base?view=revision&revision=340436

It's us. Commit is done to HEAD, will be MFC'd to 11-STABLE in the following timeframe.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mimugmail on November 15, 2018, 06:36:53 pm
https://svnweb.freebsd.org/base?view=revision&revision=340436

It's us. Commit is done to HEAD, will be MFC'd to 11-STABLE in the following timeframe.

Really nice contribution Murat, thanks! :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 15, 2018, 06:44:59 pm
Hi @mimugmail,

Our pleasure. All welcome :) Super excited to see the changes land in 19.1.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: franco on November 16, 2018, 08:40:19 am
r340436 is indeed very nice. mb, please push these into my mailbox or open a src.git ticket for swift inclusion. we need the MFC for stable/11 to be committed first though.

for the csv, it's considered bad style to manipulate files shipped with the package. for that reason FreeBSD has the "sample" trick which creates a copy of the file and only checks in the unmodified file (suffix ".sample"). We use it in core in some places, too. Plugins don't support it yet, but they should eventually.


Cheers,
Franco
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 16, 2018, 05:24:23 pm
Hi @franco, thank you very much. I hope this will be of some help to the project.

We're still testing the code in HEAD. After we're confident, it's going to be MFC'd to 11-STABLE. I'll be pinging you once we're done with that. 

I've been informed that we actually have the unmodified file (.default) with the package. Engine reads a "processed" version of that file, which -indeed- do not need to be included with the package. We're removing it. I guess we're done then.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: franco on November 16, 2018, 06:33:35 pm
Yeah, that's all sorted then, great!  8)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: therec on November 18, 2018, 05:13:56 pm
Hello Murat,
I had a question around blocking. (i.e. adds, trackers, etc.). is there a way to allow a specific site? if i go to neweggs web site, the site is unreadable. if i disable the blocking, its ok again. i see the option to the right to unblock, but it wants to unblock the group (ad category) and not the site. forgive me if I've missed something simple. and thanks for the work, this is a wonderful product, I cant wait to see where you take it.

Thanks
Robert

If i posted this in the wrong place, let me know and ill move it

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 19, 2018, 02:51:31 pm
Hi Robert, @therec

Thank you very much for your feedback. Awesome to see you've found the plugin useful.

When you browse Reports -> Security->Session Explorer, see if the site is being blocked via Application filtering or Web filtering. You can differentiate it by looking at the "Block category" information. If by Application filter, it says "Application category", if via Web filtering, it reads "Web category".

To allow a specific "Application", just go to Application Control, find and expand the related category, find your specific application, and unblock it.

If the filtering is done via Web filtering, browse to Web Controls->User defined categories. Create a new category i.e. Whitelist, and put your websites which you want whitelisted here.

Click "Save Changes" and that should be it.

Thanks,
Murat
Title: Re: Sensei on OPNsense - Application based filtering
Post by: therec on November 20, 2018, 01:45:51 pm
Thanks, that makes a lot of sense. however it doesn't seem to be working. I've added

- https://www.newegg.com/
- secure.newegg.com/
- www.newegg.com/
- www.neweggbusiness.com/
- https://newegg.com

Maybe ive misses something?

as an alternate test i confirmed http://static.hotjar.com/ was blocked (webtracking site).
I added this to the web controls as requested (user defined group) and is had the green check (allow),
This site also remains blocked after whitelisting via web address.

I suspect im missing something, I have amatuer firewall skills at best. but i love this product and hope its a long term solution for me. thank you for the help

P.S. i just noticed https://flash.newegg.com works just fine.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 20, 2018, 09:42:23 pm
Hi @therec,

Let's dig a little deeper together. I'll be writing to you privately. I might need some logs. Let's see if there's something wrong or there is a configuration problem.

 
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on November 21, 2018, 08:04:50 am
Hi, Using Sensei plugin and its great. Need help in few thing:
1. Is there any live view to know which IPs are using the most bandwidth and then drill down to specific IP to check what sites / services it is using and which site / service is consuming the most. (I use ntopng and it has very nice view to tell which devices are consuming most bandwidth only)
2. I do not know why but when i check the "Table of Local / Remote hosts, it shows bytes in / out which is very low as compared to ntopng. and i have avg 25-30 devices running all the time out of 50 devices but it only show few so how can i list all of them.
3. Is there any way to get all the web history of a user or users ?
4. Is there any way to bind names to IP (local / LAN IP) as squid does in web proxy ?
5. It filter web traffic and works as transparent web proxy so is there any way to use it as cache server as well or if you are planning for it in future.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 21, 2018, 11:24:45 pm
Hi @manjeet,

Thank you for sharing your experience with Sensei. We very much appreciate that. Find the answer below:

Quote
Is there any live view to know which IPs are using the most bandwidth and then drill down to specific IP to check what sites - services it is using and which site - service is consuming the most

Yep. Navigate to Sensei -> Reports -> Connections. Look for the Chart named Top Local / Remote Hosts. But make sure to select the reporting criteria as "Volume" from the upper right hand corner of the reports page. Default is by sessions. You can do "Session based", "Packet based" or "Volume based" reporting.

When you left click on any IP, a submenu appears. Click "Drill-down" and all reports will be automagically filtered according to this IP address you've selected.


Quote
I do not know why but when i check the "Table of Local / Remote hosts, it shows bytes in / out which is very low as compared to ntopng. and i have avg 25-30 devices running all the time out of 50 devices but it only show few so how can i list all of them.

My guess is that you might be viewing the "Session" reports. Make sure you've selected "Volume" as the reporting criteria.

All devices currently active should be listed though. My guess would be that you might be viewing reports for the last 15 minutes. Make sure you've selected a longer time frame from the right hand corner.

 
Quote
Is there any way to get all the web history of a user or users ?

Yes. You can do that from the Web / TLS reports. You have the drill-down capability for every report type.


Quote
Is there any way to bind names to IP (local / LAN IP) as squid does in web proxy ?

Actually, packet engine automatically maps DNS names to IP addresses if it can find a matching DNS transaction. Soon there will be Active Directory / LDAP integration which you'll be able to see the user / group names.

Quote
It filter web traffic and works as transparent web proxy so is there any way to use it as cache server as well or if you are planning for it in future.

In theory, packet engine is capable of doing that. But we chose to focus on complementing features that are currently not existing. Squid is a great caching proxy. Indeed caching is its original reason of existence. That being said, Sensei roadmap does not have "caching" as a feature.

Many thanks for reporting your experience with us.



Title: Re: Sensei on OPNsense - Application based filtering
Post by: maekar on November 22, 2018, 02:09:46 pm
Hi,

The maximum of 1000 concurrent users is an approximation for better hardware performance or an strict software limit?

thanks

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 23, 2018, 02:02:50 am
Hi @maekar ,

This is the current field-tested maximum. Software arranges several tunables (e.g. cache sizes, connection table sizes etc.) according to the user size.

Current focus is to make the software super stable for SME use cases (which generally means user populations below 1000)

Looking forward, engine is able to scale to hardware resources, which makes it possible to secure thousands of users.

Hope this answers your question.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: johjoh on November 23, 2018, 11:57:10 am
Good morning, will Sensei one day consume less resources in terms of RAM and CPU?
For example an Atom CPU or a Celeron with 4GB or 8GB of ram?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 23, 2018, 02:47:31 pm
Hi @johjoh,

Yes :)

A big portion of the resource requirement come from the Reporting engine (Elasticsearch). The core packet engine has been tested to run on low resource systems: e.g. Celeron  < 1GB RAM.

A roadmap feature - remote reporting - allows to run packet engine on the firewall itself, and reporting on another more powerful server.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: bobbythomas on November 24, 2018, 07:19:02 am
Hi Murat,

Couple of questions? Is there anyway to find the current installation or patch status? Where are the Sensei logs installation logs stored and how can we view that? I received an rc1 update and it's about 36MB, but it's been more than an hour since I started the installation, I would like to know the status. While installing Sensei some packages took a lot of time to get downloaded and I suspect something like that. I believe there is some latency reaching some of the repositories. Could you help me troubleshoot this issue?

Thank you,
Regards,
Bobby Thomas
Title: Re: Sensei on OPNsense - Application based filtering
Post by: shrdlu on November 24, 2018, 07:35:55 am
Not sure if this is just my setup but after upgrading to OPNsense 18.7.8 I get stuck in a loop that won't complete.  Because it reset my configuration of Sensei* after the OPNsense 18.7.8 upgrade, I have to go through the config wizard again and when I click finish, it attempts to configure everything but kicks out the attached error.  Essentially it tells me, "error indices could not be created," and I am stuck in that loop as it returns me to the beginning of the config wizard.

So, #1, is it just me?
and #2, assuming it is not me and before I simply try to uninstall/reinstall, any ideas?

Thanks
 
*Is it normal for an OPNsense upgrade to reset my Sensei configuration?  If the answer is yes, that is fine but also if there is a way to backup a config and restore it that would help me retain settings.  Either way, love the solution and reconfiguration is actually a minor thing in the grand scheme of things so if the answer is no here then that is fine as I still find huge value in the software.

Thanks
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 24, 2018, 09:02:07 am
Hi @bobbythomas,

/tmp/sensei_update.progress should have more detail regarding the update process. 36MB download shouldn't take that long.

We rolled back rc1 update in case there is something we miss with the update process.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 24, 2018, 09:12:40 am
Hi @shrdlu,

It was unfortunate that both OPNsense & Sensei got updated at the same time. Looks like while OPNsense was upgrading, we shipped 0.7.0-rc1. OPNsense update manager also updated Sensei, a case which we did not handle.

Sorry for the inconvenience. We rolled back 0.7.0-rc1.

A final fix will be out shortly.

For a workaround, I'll be contacting you. We'll try to recover the old configuration.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 24, 2018, 11:26:30 pm
Dear Sensei users,

0.7.0-rc1 upgrade is back.

A quick update on 0.7.0-rc1 upgrade:

If you encountered any Sensei issues while upgrading your OPNsense to 18.7.8, this was due an unhandled case in our package updater when the upgrade process is triggered from the OPNsense firmware updater, not from the Sensei Status Page. This is fixed now in the upcoming 0.7.0-rc1.

But the fix will be in effect starting from 0.7.0-rc1.

So, If you’re on 0.7.0-beta1, and do NOT want to upgrade to 0.7.0-rc1 immediately we strongly recommend running the following command to avoid any issues with the OPNsense system updater.

pkg upgrade os-sensei-updater && pkg lock os-sensei

The command will upgrade your Sensei updater to the latest version and also put a lock on os-sensei package so that OPNsense package update utility will not try to update Sensei.

If you also want to upgrade  to 0.7.0-rc1: Navigate to Sensei -> Status -> Check Updates, and you’ll be guided to upgrade to 0.7.0-rc1.

pS: 0.7.0-rc1 introduces several minor bug-fixes both on the updater and the UI. If we do not hear any issues, we’ll hopefully release 0.7.0 in the coming week.

ppS: Thanks to increasing number of Sensei beta users, it looks like we need to increase bandwidth for Sensei Updates server (updates.sunnyvalley.io). Cool indeed  ;) This will be done in the following weeks. In the meantime, if you encounter slowdowns while installing / upgrading, we’d very much appreciate your understanding.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: samsonmcnulty on November 25, 2018, 08:54:10 am
Sounds fantastic! Good to see the adoption rate increasing at a healthy rate. I did encounter this error but it seems you are already aware of the issue:


***ERROR: Indices could not be created! Reporting may not work***



Is there a temp workaround? I assume uninstalling the package and reinstalling would work?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 25, 2018, 05:59:44 pm
Hi @samsonmcnulty

Yep, that would work.

Can you run the following commands. Basically it'll uninstall & install sensei

service eastpect onestop
service elasticsearch onestop
pkg delete elasticsearch5
pkg delete os-sensei
rm -rf /var/db/elasticsearch/nodes/*


You can also do that by selecting "Uninstall elasticsearch & Remove elasticsearch data" options while uninstalling from Web UI.

then to re-install it:

pkg install os-sensei

Sorry for the inconvenience.

One question: did that happen after you've done an OPNsense 18.7.8 upgrade? We're aware of this problem & hopefully fixed.

I wonder if there are other cases.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: dragon2611 on November 25, 2018, 10:05:50 pm
I'd like to try sensei but I suspect i'd run into problems with lack of RAM and also I have an opensense HA pair with one physical and one virtual (KVM) so I think i'd run into the KVM/VIRTIO issue.

I'm wondering if I'd be better off starting another virtual firewall and stuffing it in the traffic path for the machines i'd want to put behind sensei.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 26, 2018, 02:38:53 pm
Hi @dragon2611,

Good idea :) Let us know if you encounter any issues. On the virtual FW, you can use E1000 as network the adapter type:

https://guide.sunnyvalley.io/sensei/support/faq#can-i-run-sensei-on-a-virtualized-environment-like-proxmox-virtualbox-kvm

Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on November 27, 2018, 07:42:10 pm
Hi, Sunnyvalley.

The first hit and miss: try to block youtube used via google chrome...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 28, 2018, 05:56:32 am
Hi @Antaris,

Thanks for reporting this.

It's because of QUIC: Google's new protocol suite, a replacement for TCP + TLS + HTTP/2. Chrome defaults to QUIC when you browse Google services. Other browsers use TCP so Sensei is be able to identify & block.

Sensei is able to identify QUIC, though its detailed protocol parser is under development. When we're done with it, it'll be able to identify protocols which are transported through QUIC. We hope to have it with 0.8.0 release.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 29, 2018, 04:03:49 pm
Dear Sensei users,

An update to: https://forum.opnsense.org/index.php?topic=9521.msg47488#msg47488 :

If you got stuck in Sensei Configuration Wizard,  here is a quick fix for you:

open /usr/local/sensei/scripts/installers/opnsense/18.1/sensei-init.sh file with an editor, and locate this part. It should be line 64.

if [ "$INDICES_COUNT" -lt 6 ]; then

Update this line to read like:

if [ "$INDICES_COUNT" -lt 5 ]; then

Save the file and re-run the configuration wizard.

0.7.0-rc2 will come with a more intelligent provisioning script which will try to diagnose any inconsistencies with the backend database and try to fix them automatically.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 30, 2018, 02:22:27 am
Dear Sensei users,

An update to: https://forum.opnsense.org/index.php?topic=9521.msg47653#msg47653

We've decided to merge some of the code from the QUIC branch to 0.7.0.

Looks like we have a working app identification & filtering for applications running through QUIC :) e.g. Google Services + Chrome browser.

Update will be introduced with 0.7.0-rc2.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 01, 2018, 11:16:31 am
Dear Sensei users,

An update to: https://forum.opnsense.org/index.php?topic=9521.msg47653#msg47653

We've decided to merge some of the code from the QUIC branch to 0.7.0.

Looks like we have a working app identification & filtering for applications running through QUIC :) e.g. Google Services + Chrome browser.

Update will be introduced with 0.7.0-rc2.

Thanks guys, looking forward to it. Can we hope for an optimisation to reduce hardware requirements, especially about RAM?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 01, 2018, 10:36:03 pm
Hi @Antaris,

Many thanks for bringing this into our attention. Looks like with 0.7.0-rc2, Sensei is one of the first in the industry to offer granular control for QUIC based applications.

Currently, big vendors are advising to completely block QUIC protocol, thus forcing browsers to fall back to TCP+TLS. This is slower.

As for memory requirements, actually yes. We're planning a limited reporting option, which will require way less memory than we require today. This will still provide reporting but most probably will lack some advanced features like Drill-down and per-connection details. Other than reporting, all features will be there.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: opnsenseuser on December 02, 2018, 08:13:37 am
when will sensei appear in opnsense as a plugin?
which hardware requirement is necessary?
Is the latest generation Atom processor (c3558) with 8 gb of ram sufficient for sensei?

thx
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 02, 2018, 03:15:16 pm
There is an update Engine: 0.7.0-rc2, but when trying to update it, the system returns:  "No update is available
There are no updates available for you. You are using the latest version. " and stays on 0.7.0-rc1
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 03, 2018, 05:04:26 am
Hi @noname12123,

when will sensei appear in opnsense as a plugin?
which hardware requirement is necessary?
Is the latest generation Atom processor (c3558) with 8 gb of ram sufficient for sensei?

thx

We have a few small items left for the final OPNsense integration.  Then Sensei will be an OPNsense plugin which can installed from the OPNsense Plugins menu. If anything big does not come up, I guess we'll all be finished with them by the end of this month.

I'd expect that latest generation Atom would be ok. Might be a little bit slow to start Elasticsearch but when it warms up, it should be all fine. Crucial thing is RAM and 8GB is perfectly fine.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 03, 2018, 05:08:21 am
Hi @antaris,

There is a small blog post coming related to that. We'll need to use the command-line updater for the rc2 update. GUI code is missing a "pkg update -f".

Can you try to update via command line?

As the root user, just run:

sensei-updater

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 03, 2018, 07:32:42 pm
Dear Sensei users,

After testing 0.7.0-rc2 update with a few of Sensei users, it looks like 0.7.0-rc2 is ready to go.

We'll need to use the command-line updater for this update. GUI code is missing a "pkg update -f".

Login to the firewall console as the root user; and run:

sensei-updater

It'll take care of the rest, and you'll be updated to 0.7.0-rc2. You'll need to manually start the Sensei engine from Sensei->Status.

0.7.0-rc2 introduces fine grained application identification & filtering for Google Services through Chrome browser (QUIC protocol update); as well as several other reliability fixes for the sensei-updater.

If we do not see any issues reported; 0.7.0 will be finally released Thursday this week :)


Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 03, 2018, 07:33:50 pm
Thanks a lot:

"Sensei has been updated successfully."

Just have to start Sensei Packet Engine manually...

It's runnig as guest on Proxmox btw...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 03, 2018, 07:38:10 pm
Hi @antaris,

Glad that it went well. Thanks for the notice about starting Sensei. I've updated the message accordingly.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 04, 2018, 06:25:33 pm
Do i miss Web 2.0 controls and TLS Visibility menus as seen on advertisement video?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 05, 2018, 03:12:55 am
Hi @Antaris,

Web 2.0 Controls / Cloud Application Controls depend on port agnostic TLS Inspection functionality. TLS Inspection will be made available with Sensei Premium Edition.

Should you like to give an early try, I'll be happy to provide a trial license for you.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 05, 2018, 05:19:01 pm
It's too early i guess, and my Sensei is not ot production enviroment. When it's ready and the prices are known, will give it a try in one of the schools that i support. I can test it in network with up to 1500 devices and 1gbps symmetrical internet.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 06, 2018, 03:31:44 pm
Hi @Antaris,

Sounds great. Will get back to you when we have more progress with that.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 07, 2018, 01:21:42 pm
Hi, I just reinstalled the OPNsense and trying to install the Sensei plugin but script is timing out.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 07, 2018, 03:22:29 pm
Hi @manjeet,

Update server is operational again.

Make sure you're following the latest install instructions:

https://guide.sunnyvalley.io/sensei/getting-started/setup

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 10, 2018, 08:10:17 am
Thanks. It is installed and working.

I still have one same issue as mentioned before.. In "Table of Local Assets" and "Table of Remote Hosts" i only see 10 devices / hosts. I have changed criteria to volume, time interval 24 hours. I have also checked it by increasing settings. I am currently running more then 30 devices. So is there a limit on no of showing hosts or anything ?? I need to provide bandwidth usage (upload / download / total) reporting of all the devices in network daily basis. Any other way to do so ??
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sagem2004 on December 10, 2018, 04:02:06 pm
Good evening,
we can filter the site in safesearch " picture "
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 10, 2018, 06:15:16 pm
Hi @manjeet,

Glad that installation went smooth.

Thanks. It is installed and working.

I still have one same issue as mentioned before.. In "Table of Local Assets" and "Table of Remote Hosts" i only see 10 devices / hosts. I have changed criteria to volume, time interval 24 hours. I have also checked it by increasing settings. I am currently running more then 30 devices. So is there a limit on no of showing hosts or anything ?? I need to provide bandwidth usage (upload / download / total) reporting of all the devices in network daily basis. Any other way to do so ??

Ah, now I see what you're trying to accomplish. All charts on the report tabs will show top 10 records. For table reports, I guess we can put an option to provide the full list.

Stay tuned. We'll pass an update.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 10, 2018, 06:19:32 pm
Good evening,
we can filter the site in safesearch " picture "

Hi @sagem2004,

I don't think I was able to fully understand the question. Can I request that you rephrase it?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sagem2004 on December 10, 2018, 06:41:06 pm
can have blocked pornographic images via safesearch

exemple : https://support.opendns.com/hc/en-us/articles/227986807-How-to-Enforcing-Google-SafeSearch-YouTube-and-Bing

Merci.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Misant on December 10, 2018, 07:27:49 pm
Installed Sensei today on a Qotom. seems to be working fine. Setup is just for a small household with me and my girlfriend, but we are going to expand to a dog and 2 kids. So torture tests will have to wait for some time.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 10, 2018, 09:15:03 pm
Hi @sagem2004,

Thank you for the clarification.

Google Safesearch enforcement is in Sensei's short-term roadmap. Should arrive with Sensei 1.0, which is the first production release for Sensei (ETA Q1 2019).
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 10, 2018, 09:16:33 pm
@Misant, Good to hear that :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sagem2004 on December 10, 2018, 09:34:59 pm
Hi @sagem2004,

Thank you for the clarification.

Google Safesearch enforcement is in Sensei's short-term roadmap. Should arrive with Sensei 1.0, which is the first production release for Sensei (ETA Q1 2019).

very Good news thank you :) :) :) :) :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 11, 2018, 11:12:54 am
Thanks for it..

Hi @manjeet,

Glad that installation went smooth.

Thanks. It is installed and working.

I still have one same issue as mentioned before.. In "Table of Local Assets" and "Table of Remote Hosts" i only see 10 devices / hosts. I have changed criteria to volume, time interval 24 hours. I have also checked it by increasing settings. I am currently running more then 30 devices. So is there a limit on no of showing hosts or anything ?? I need to provide bandwidth usage (upload / download / total) reporting of all the devices in network daily basis. Any other way to do so ??

Ah, now I see what you're trying to accomplish. All charts on the report tabs will show top 10 records. For table reports, I guess we can put an option to provide the full list.

Stay tuned. We'll pass an update.

I do not how it calculate the top 10 but i think you have an issue here.. I was looking at "Insight" for current network usage and find out that one of the system has consumed 4GB of data since morning. I checked it in Sensei and it showed the same 4GB data usage for that IP.

But when i checked the top 10 list in "dashboard" and in "reports" (No filters, cross-checked) (it showed me that same report), this IP with 4GB usage was not there. Even some other IPs which Insight showed were not also there.

It showed me list of top 10 which i think is better match with the last night usage but not since this morning. Its been 6 hours and i do not see those IP in this list.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 11, 2018, 03:58:09 pm
Hi @manjeet,

I see. Let's dig deeper. Can you reach us through sensei -at- sunnyvalley.io?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on December 11, 2018, 04:17:14 pm
Hello, mb

Is there a way to clear all the logs in Sensei?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 11, 2018, 08:52:23 pm
Hi @cgwork,

Use the following two scripts to delete and re-create all reporting indices:

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py


This will be also a Web UI menu item with the upcoming Sensei update (0.7.0-rc3).
Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on December 12, 2018, 04:35:05 pm
Hi @cgwork,

Use the following two scripts to delete and re-create all reporting indices:

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py


This will be also a Web UI menu item with the upcoming Sensei update (0.7.0-rc3).

Awesome Thank you ... also have you thought of getting the reports to be printed or converted to .pdf format? i also noticed when i get the emails and "click to download and view the detailed reports" are blank see attachment. Did i miss an check in the box so i get them? I'm currently selected only Sessions but it would be nice if i could get all of them or select the once i would like to have.

Thank you again for the hard work.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 13, 2018, 02:37:55 am
Hi @cgwork,

You're all welcome. We had introduced PDF export previously.  It's being re-worked and will be available shortly.

You shouldn't receive an empty html file. Looks like a problem. Can you share which e-mail provider you are using? It's been tested with major ones like Gmail & Outlook. Let's try with yours.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on December 13, 2018, 01:53:12 pm
sure i'm using gmail for this setup
Title: Re: Sensei on OPNsense - Application based filtering
Post by: kagou on December 13, 2018, 02:06:17 pm
Hi. I'v some problems with sensei (look at the picture).
I'v tried first with my system but after some problems i'v rebuilt my interface assignments, removing bridge system.
Now i'v a WAN/DMZ/WIFI/LAN on my 4 ethernet ports.
I'v stoped and used the "You can restore all Sensei packet engine configuration to their original defaults by clicking 'Reset' button."
Set just ma LAN to be supervised, but look at the picture
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 14, 2018, 05:51:11 pm
Hi @kagou,

Looks like a problem with the backend indexes.

Can you try these if they fix the problem?

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py


It it does not, can you share your /var/log/elasticsearch/elasticsearch-2018-12-13.log log file to sensei - at - sunnyvalley.io ?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 14, 2018, 06:45:57 pm
Hi @cgwork,

sure i'm using gmail for this setup

Gmail should be fine. Can you forward the email to sensei - at - sunnyvalley.io ? If you can forward as an attachment, that'd be perfect.

Are you using Gmail through a browser, or through an email client?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 14, 2018, 06:53:21 pm
Update to @manjeet's post: https://forum.opnsense.org/index.php?topic=9521.msg48451#msg48451

Spotted the problem. A typo avoided reporting criteria to be reflected for some reports.

Fix should arrive with 0.7.0 release.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 14, 2018, 07:15:33 pm
Dear Sensei users,

We know you’re looking forward to seeing 0.7.0 release. We also do indeed.

Yet, we decided to ship another release candidate before the actual release because some updates to the code base might have more impact than we originally planned. These code updates are preliminary work related to an effort to minimize external library dependencies and compiling Sensei engine as a Position Independent Executable (PIE).

Minimizing external library dependencies will allow Sensei to be able to run on embedded platforms which run on very low resources.

PIE is a nice feature which will be default for OPNsense@HardenedBSD and will provide mitigation capabilities against exploit attempts to the packet engine. (Note: PIE is not enabled yet)

So there we have 0.7.0-rc3 publicly available for you to test. This is the Changelog from rc2 to rc3:

New features (from 0.7.0-rc2 to 0.7.0-rc3).
* More lightweight core packet engine
* Option to delete all reporting data
* Mobile web browsers compatibility. You’ll be able to view Sensei reports through a mobile device.
* Prevented scheduled jobs from submitting unnecessary emails.
* HW requirements check has been made available for the UI initial configuration wizard.
* Some stability improvements. 

0.7.0-rc3 has been under testing for about a week now, but if you’re running Sensei on a more production like environment, you might want to wait till we ship 0.7.0 final release, which should arrive in a week if we do not see any issues with 0.7.0-rc3.

To update to 0.7.0-rc3, login to OPNsense UI, navigate to Sensei -> Status and click Check for Updates. You should see an update reported. Click Update to proceed with the update. Sensei updater should take care of the rest.

Best
Sensei team
Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on December 18, 2018, 02:12:03 pm
Great News mb,

In my personal opinion RC (Release Candidate) are like the actual gold image, as it progress and other clients testing it will become better with the final release.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 19, 2018, 08:08:22 am
Hello MB, I can see the option in "Table of local / remote assets" to select different top users. Can you also add another option to sort it ascending or descending so that we can check the top user in top list rather then going to the entire list to find one.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 19, 2018, 02:55:22 pm
Another idea about "Session details": give the user ability to restrict begin and end date and time fields to reduce search results to concrete time period.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 19, 2018, 07:17:09 pm
@cgwork, @manjeet, @Antaris,

Many thanks for the suggestions. Feature requests have been added to 0.8 workload. We'll do a more general re-visit to table reports. Please feel free to reach out for more ideas.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: the-mk on December 19, 2018, 08:01:51 pm
a question from a maybe future sensei user:
since this elastic search module needs a lot of diskspace and sure does a lot of writing - is there a possibility to divide the installation into an "OS"-disk (binaries; usually on a SSD) and a "data"-disk (storage intensive data, lots of writes; usually on a HDD)?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: nikkon on December 20, 2018, 12:13:10 am
Hi all,

After upgrading to version 0.7.0-rc3 none of my dashboards or reports are loading anymore

That's an error example:
{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "resource.type": "index_or_alias",
        "resource.id": "conn_all",
        "index_uuid": "_na_",
        "index": "conn_all"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index",
    "resource.type": "index_or_alias",
    "resource.id": "conn_all",
    "index_uuid": "_na_",
    "index": "conn_all"
  },
  "status": 404
}

Any clue?

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 20, 2018, 06:23:27 am
Thanks @MB for considering this.

I have an another thing to ask. I am not if that is 100% possible or is it already implemented because i did not find it in any details.

In report we can see the source address, destination address or host, app category and protocol it is used. It gives us huge information about who has download / uploaded to where and how much data, also time stamp of session etc. But i do not see any ways to check what exactly the user has downloaded. For e.g one of my user used 5GB data in one day which is used by google services and it gives us the list of when and where, but no info about what exactly which for now we have to ask the user. This could be useful because if user is downloading / uploading something not allowed to server / account which they are allowed to access then they probably will deny it.

Also can you add option to export reports (excel or pdf) including custom / filtered reports so that we can provide report to management whenever needed rather then filling mail box with auto reporting.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: maekar on December 20, 2018, 02:25:15 pm
Hi,

Is there anything special to do with VLAN?

We have interfaces tagged and untagged. When I activated Sensei and configured just a few web categories to test, everything worked well with the untagged interface but all VLAN networks lost connectivity, devices in all VLAN not even get IP address by DHCP. And the problem persisted even when I deselected those interfaces to get managed by Sensei, I had to stop it and uninstall it to get VLAN networks working again.

Thanks!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 20, 2018, 05:38:55 pm
Hi @the-mk,

Thank you very much for the suggestion: We get this request quite many times. People who’d like to see this functionality seem to be either running on the low end - the device is very weak and lack the resources to run reporting on the device itself, or they run on the high end - throughput & number of users are quite high (>1K users) and it makes  sense to put reporting on a separate device.

In addressing this requirement, we’ll offer an option - in the initial configuration wizard - asking the user whether s/he wants the reporting on the device itself, or on a remote server.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 20, 2018, 05:44:05 pm
Hi @nikkon,

Looks like alias indexes are messed up. By any chance, did you do any "reset to factory defaults" ?

We'd like to dig deeper. Can you share your /var/log/elasticsearch/elasticsearch-2018-12-19.log through sensei - at - sunnyvalley.io ?

For a workaround, you can run these two commands to reset the indexes: (beware: this will erase your reporting history)

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py


Let us know if this does not fix the problem.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 20, 2018, 05:53:57 pm
@manjeet, you’re all welcome.

If the connection is clear-text (e.g. HTTP), you can see the individual downloaded files from Web Reports: Web - Table of URIs. For the TLS encrypted sessions (e.g. HTTPS), this will be possible with the all ports TLS Inspection feature - though it’s going to be available for Premium Subscriptions.

For the Table reports, development & tests have been completed, and it’s ready to ship with 0.7.0 release.
I’ve sent you a link today to try it and see if there are any more issues.

Reports - PDF export - its’ on the short term roadmap. Probably it will ship with 0.8.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 20, 2018, 06:07:27 pm
Hi @maekar,

Thanks for reporting this. Yes, we’re aware of this problem. Unfortunately part of the solution required some development on the Operating System itself (FreeBSD netmap implementation).

Good news is that hopefully it’ll be fixed with OPNsense 19.1. On the FreeBSD side, we’ve sponsored a development which fixes this and some other issues with the netmap implementation on FreeBSD (https://svnweb.freebsd.org/base?view=revision&revision=340436)

We’ve been testing the 11.2-STABLE MFC code for some time and it looks good to be finally integrated with OPNsense.

We’re working very closely with the OPNsense team on this. I’ll be posting an ETA after we sync with @franco.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: nikkon on December 21, 2018, 02:57:34 pm
@mb thanks for replying
I did execute the 2 scripts.

please check the log below:

cat /var/log/elasticsearch/elasticsearch-2018-12-
elasticsearch-2018-12-16.log  elasticsearch-2018-12-20.log
root@Skynet:~ # cat /var/log/elasticsearch/elasticsearch-2018-12-20.log
[2018-12-20T01:05:36,849][INFO ][o.e.n.Node               ] [yCObJMR] stopping ...
[2018-12-20T01:05:36,889][INFO ][o.e.n.Node               ] [yCObJMR] stopped
[2018-12-20T01:05:36,889][INFO ][o.e.n.Node               ] [yCObJMR] closing ...
[2018-12-20T01:05:36,911][INFO ][o.e.n.Node               ] [yCObJMR] closed
[2018-12-20T01:07:19,550][INFO ][o.e.n.Node               ] [] initializing ...
[2018-12-20T01:07:19,707][INFO ][o.e.e.NodeEnvironment    ] [yCObJMR] using [1] data paths, mounts [[/var (tmpfs)]], net usable_space [1.9gb], net total_space [2.4gb], spins? [unknown], types [tmpfs]
[2018-12-20T01:07:19,707][INFO ][o.e.e.NodeEnvironment    ] [yCObJMR] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-12-20T01:07:19,711][INFO ][o.e.n.Node               ] node name [yCObJMR] derived from node ID [yCObJMRsQcSMKeQy7KNhyA]; set [node.name] to override
[2018-12-20T01:07:19,711][INFO ][o.e.n.Node               ] version[5.6.8], pid[32322], build[688ecce/2018-02-16T16:46:30.010Z], OS[FreeBSD/11.1-RELEASE-p17/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_172/25.172-b11]
[2018-12-20T01:07:19,711][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/local/lib/elasticsearch]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [aggs-matrix-stats]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [ingest-common]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-expression]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-groovy]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-mustache]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-painless]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [parent-join]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [percolator]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [reindex]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [transport-netty3]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [transport-netty4]
[2018-12-20T01:07:21,819][INFO ][o.e.p.PluginsService     ] [yCObJMR] no plugins loaded
[2018-12-20T01:07:25,240][INFO ][o.e.d.DiscoveryModule    ] [yCObJMR] using discovery type [zen]
[2018-12-20T01:07:26,419][INFO ][o.e.n.Node               ] initialized
[2018-12-20T01:07:26,420][INFO ][o.e.n.Node               ] [yCObJMR] starting ...
[2018-12-20T01:07:26,927][INFO ][o.e.t.TransportService   ] [yCObJMR] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-12-20T01:07:30,078][INFO ][o.e.c.s.ClusterService   ] [yCObJMR] new_master {yCObJMR}{yCObJMRsQcSMKeQy7KNhyA}{QHCtod64RcOkM74GkkvW-g}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-12-20T01:07:30,120][INFO ][o.e.h.n.Netty4HttpServerTransport] [yCObJMR] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-12-20T01:07:30,121][INFO ][o.e.n.Node               ] [yCObJMR] started
[2018-12-20T01:07:30,140][INFO ][o.e.g.GatewayService     ] [yCObJMR] recovered

in Gui i got this:
Error at /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php:74 - fsockopen(): unable to connect to 127.0.0.1:4343 (Operation timed out) (errno=2)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 21, 2018, 03:01:24 pm
Hi @Nikkon,

Is this the log after you executed the delete/create scripts, or the one with the errors?

Looks like the former? Did the scripts resolve the problem?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: nikkon on December 21, 2018, 03:16:43 pm
yes. this is before i executed both scripts
it's not solved.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 21, 2018, 05:06:43 pm
Hi @nikkon, understood. Let's do some more debugging together. I'll contact you.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 23, 2018, 06:02:32 pm
Very often i see remote hosts in local table and vice versa. Is something wrong with my setups?
And sometimes i see comunication between two local ip addresses and one of them is marked as remote...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 23, 2018, 08:03:06 pm
Hi @Antaris,

Do you have multiple interfaces configured for Sensei? Are these IP addresses multicast / broadcast addresses?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 23, 2018, 10:53:34 pm
I have only LAN selected in Sensei with only one IP and no VLANs on it. The adresses are known internal hosts. Not broadcast or net addresses.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 26, 2018, 09:56:25 pm
Dear Sensei & OPNsense users,

Happy new year to all. Here is a humble new year present from Sensei team.

We're happy to announce the availability of Sensei 0.7.0 release. It was ready since last Friday, but we wanted to make sure everyone had a calm Christmas holiday, spending time with friends and family instead of doing Sensei deployments :)
 
This is the full list of features that this release brings (from 0.6.x):

1. 350+ new applications identified.
2. Google applications browsed via Chrome are now being identified (QUIC over UDP protocol support).
3. Mobile browser compatibility: you can view reports from your mobile browser
4. Reports enhancement: Data retirement option introduced. With this option you can define how long to keep your reports (days)
5. Reports enhancement: Option to erase all reporting data
6. Reports enhancement: Drill-down in Security reports is now available
7. Reports enhancement: Daily executive reports. Selected reports delivered via a daily e-mail.
8. You can easily add block/allow rules within Session Explorer based on Application and Application Category or SNI / hostname
9. User's Manual in English.
10. More deployment options for Home and Large scale users
11. Changelog between updates
12. Fixed Rebellion Theme compatibility issues.
13. Better Cloud Nodes availability
14. Better & smoother updates
15. We speak your language now, we added i18n support to match your OPNsense UI language. English and German are there for now, more coming soon.
16. Removed some large dependencies in preparation for embedded devices & PIE (Position Independent Executable) support.More performance & stability improvements.

To update your installation, simply navigate to Sensei -> Status and you should see 0.7.0 update being reported and an option to install it. If you do not see the update notification, just click "Check for updates" and you'll be guided through the update process.
 
A quick note: Although this is marked "release", Sensei is still under BETA development. We strongly advise to test the software on one of your test-beds to see if it fits your requirements. When we finally release Sensei 1.0, the BETA program will cease and the software will be publicly available for all users. We expect to release Sensei 1.0 in Q1 2019.
 
If you find any issues or you want to reach out for comments and feedback, please do not hesitate to contact us through sensei -at- sunnyvalley.io or through this forum thread.
 
Happy new year to all

Sensei team
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 27, 2018, 07:18:12 am
Thanks for the update @MB.

For Table Reports update is working as expected.

As @Antaris mentioned, i also see remote host in local table but no local host in remote table except OPNsense LAN IP which i think, in one way, is not an issue because firewall itself generates traffic for interface access etc..
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 27, 2018, 09:04:14 pm
Also thanks from me for the update.

"12. Fixed Rebellion Theme compatibility issues."

In session details the headers of the columns are still with white text on white background:

https://www.dropbox.com/s/0v72em2bch0rk0q/Reb.jpg?dl=0 (https://www.dropbox.com/s/0v72em2bch0rk0q/Reb.jpg?dl=0)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on December 28, 2018, 01:02:32 am
Can't tell if this is a new issue or not as I only installed of of .7.0-rc3. When the packet engine is running unbound overrides are being ignored.

My nslookup results show "UnKnown" in the server spot and are forwarding my overrides to public servers.
As soon as I stop packet engine this works again.
I was able to add my root domain to the "local domain to override" section and it fixed that one issue there but I have overrides for other hosts. Am I missing a setting where Sensei is overriding DNS?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 28, 2018, 03:18:43 am
Thanks for the update @MB.

For Table Reports update is working as expected.

As @Antaris mentioned, i also see remote host in local table but no local host in remote table except OPNsense LAN IP which i think, in one way, is not an issue because firewall itself generates traffic for interface access etc..

Hi @manjeet, you're very welcome. Can you share with me a screenshot of  the remote hosts table (you know my email). Would like to see how they look like. Normally you should only see local hosts behind the firewall there.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 28, 2018, 03:20:06 am
Hi @Antaris,

You're all welcome & thx for the pointer. We'll fix it.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 28, 2018, 03:32:44 am
Hi @donatom3,

Actually this is an expected behavior. We're utilizing DNS override for Web Reputation & Threat Intel. Since DNS occurs before the actual connection attempt, we gather prior threat intelligence & reputation about the remote IP & host.

For a quick workaround you can disable Cloud Reputation & Web Categorization from Sensei -> Configuration. Then you'll still have reputation data for the top 1Million domains from the local database, but not for +140M :(.

We're exploring ideas to do this in parallel. This way you'll still be able to do your DNS through your DNS server and Sensei will do a parallel query for its intelligence.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on December 28, 2018, 05:04:55 am
@mb this is good to know.
So if I'm in an environment where I'm using Windows domain controllers for DNS to get the full effect of Sensei would I need to have the opnsense router be the DNS forwarder?

Also does this mean if I just hand out public DNS servers via DNS am I not getting the full advantage of Sensei?

P.S. I do want to add that I am liking Sensei so far.
I am still able to download at 1gbps on my i5-5250u but thinking of picking up a box that has an i5-6500.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 29, 2018, 07:29:00 am
Hi @donatom3,

For application control, dns does not play any role there, so you'll be utilizing Sensei at its full potential in any ways.

For security & web filtering, yes, you'll lose some data there, provided that you do not enable Web Reputation & Cloud Threat Intel, which requires DNS override.

If your DNS transactions are traversing through an interface which is protected by Sensei & you have Cloud enabled, dns override will be in place (like unbound). Because we are way earlier in the process, we'll also override unbound.

That being said, if you place the DC in the same broadcast domain with your clients, they'll be utilizing your DC (this way they do not traverse through Sensei to reach the DNS server). -- Local DNS server will still be subject to the same behavior if its traffic passes through Sensei. -- Here you have your local queries untouched.

We've been asked (by some more users) of a possibility of using both Sensei Cloud database & local dns servers. We're evaluating methods to do it. Most probably we'll be back into this in the second or third quarter of 2019.

Very happy to hear that you've attained gigabit speeds and happy with the software ;)

Title: Re: Sensei on OPNsense - Application based filtering
Post by: jinn on January 02, 2019, 09:17:50 am
I tested sensei last week. after I activated it, however, access to the internet was barely possible (eg google was not available at all). since it was a productive system, I deactivated sensei for now and did not continue to use it. Nevertheless, I would like to know why it was located, how should I proceed best for analysis here?

In addition, I wanted to ask whether it is even possible, what I want to achieve: I would like an evaluation for special services (social media, online gaming, ...). Is sensei able to give me an evaluation of how much time / data was used for special services?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on January 02, 2019, 12:03:23 pm
Hello @MB, I need another favor from you if possible.

Can you please work with OPNsense team to add an option for Sensei "Dashboard" and "Reports" in "Assigned Privileges" for users/groups. Well, I need to create few users/group so that they can only check the reports of team assigned to them. I do not want to provide root user access level to them to avoid them poking around and change my configuration or delete any logs or data..
Title: how to work with local hostnames?
Post by: the-mk on January 02, 2019, 07:45:19 pm
I finally decided to install Sensei on my box with several network interfaces.
I also have some servers running at those interfaces where I configured different hostname suffixes (configured with dhcp reservations and the checkbox to register the names in unbound dns). i.e. server1.lan, server2.home, server3.iot,...
before running sensei I was able to resolve all hostnames fine.
I guess the setting "local domain name to exclude" in the cloud threat intel tab has something todo with it? tried to enter here one servername for testing - did not work for me... is saving the setting enough or do I need to restart something?
how to tell sensei to honor local servernames when tried to resolve local hostnames?

EDIT: after reading the post of donatom3 and the suggestion of mb to turn off cloud threat intel I can resolve my local hostnames again!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on January 03, 2019, 06:54:05 am
the-mk,

In my case I left that feature turned on. All I did is put my domain in the local domain section of the cloud threat intel section.

Now my local domain is ad.xxxx.com, but I have entries for domain xxxx.com, so I put in xxxx.com into the local domain and all subdomains are passed through correctly to my custom names in unbound.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 03, 2019, 07:21:25 am
@manjeet,

This is a cool feature request. Thanks. Added to roadmap.

A quick note on remote IP addresses on "local assets table": We've had a look at the screenshots. 169.254.x.x is actually a local ip address. Your PC is automatically assigned an IP address, if it cannot get an IP address from a DHCP server. More on this: https://www.techrepublic.com/forums/discussions/where-did-ip-16925451183-come-from/

Screenshots show that some PCs (or a PC) wanted to communicate with the outside world, but it did not get any replies (Incoming packets all zero).

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on January 03, 2019, 11:00:29 am
Thanks @MB and Thanks for the update.

Can you also add one option in reports for looking a live reports without manually refresh time. When in Dashboard / Reports -> Filter (Reports Interval) -> When selecting Custom interval there is "Start time" and "End time".

It will be great if you can add another option or select box there to select "End time" as ongoing.

For e.g: If i want to see current reports from a specific time let says since morning and wants to check the reports after every 10 or 15 min gap then every time i have to select the option "Go to today" in End time. It would be better if there is an option as ongoing which will automatically change time in some specific interval of time or select "refresh interval" as time to refresh and update the time in 'Reports Interval"
Title: Re: Sensei on OPNsense - Application based filtering
Post by: dp on January 03, 2019, 08:02:06 pm
I see that shaping at layer 7 is on the roadmap for sensei. Is there any time table on that feature? Has it even started? I am looking to use it in a 1500-2000 user environment to replace some aging equipment if it is slated for the near future.

Also I have several ideas that I would like to see implemented as I have used application shapers for over 10 years in our environment.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 04, 2019, 06:09:15 am
@manjeet, you're right. They are already in the workload for 0.8 ;)

Hi @dp, correct. Shaping is on the roadmap. Our plan is to feed the currently existing shaping infrastructure on OPNsense. Sensei development is quite booked with IPv6 support nowadays. Though, you should see it implemented like Q2 or Q3 2019. We'd like to keep in touch about ideas on that ;)
Title: Sensei on OPNsense - Cloud Node Status
Post by: lmwalker71 on January 04, 2019, 07:44:40 pm
Under Cloud Node Status, The Nodes are always showing Down, with a count down runs with a 'Check Now" button. If the count down runs its cource the status changes to up for about 15 seconds or if I click 'Check Now' is this the normal??? :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 04, 2019, 08:01:18 pm
Hi @lmwalker71,

Not quite ;)

If you're based in USA, make sure you have the "US - Central" Cloud nodes checked & in green color (Sensei -> Configuration -> Cloud Threat Intelligence). (If in Europe, Europe nodes should be active)

If that's already the case, can you reach out to us through sensei - at - sunnyvalley.io so that we can dig deeper together?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on January 09, 2019, 09:26:35 am
Services are randomly (?) stopping.

I read somewhere that services will stop, when there is less performance, to save power for opnsense native tasks, but I run Sensei on a Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (8 cores) with 24 GB ram which should be quite enough power.

Since I have lacp interfaces for lan (lagg0) and wan (lagg1), each with 2x1g and vlans on lan interface and due to some remarks in this thread that vlans are not supported yet (due to FreeBSD netmap) and will be fixed with OPNsense 19.1, I added an additional, plain interface and just connected 1 pc.

Then I added this single interface with 1 pc as protected interface in Sensei. I even reduced the deployment size from x-large (what I would need if vlans would work) to small in hope that memory footprint will be reduced (actually just 1 user/pc is connected).

But the packet engine still stops randomly. I blocked shopping categories to verify whether engine is just shown as stopped or really stopped. And it is really stopped, since I can open shopping page when service is marked as down.

Same with the ElasticSearch service. Sometimes when I open reports or dashboard, I get a pop-up that ElasticSearch service has to be started first and whether I want this. When I deny and check status page, then both services are down - means whenever packet engine stops, ElasticSearch stops, too.

Any hints what could be the reason for the stopping services.

The service crashes pretty soon. 1-2 minutes after starting up.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: jinn on January 09, 2019, 09:52:04 am
Quote
I tested sensei last week. after I activated it, however, access to the internet was barely possible (eg google was not available at all). since it was a productive system, I deactivated sensei for now and did not continue to use it. Nevertheless, I would like to know why it was located, how should I proceed best for analysis here?

In addition, I wanted to ask whether it is even possible, what I want to achieve: I would like an evaluation for special services (social media, online gaming, ...). Is sensei able to give me an evaluation of how much time / data was used for special services?

currently sensei works with deactivated cloud threat intel.
Unfortunately, "Egress New Connections by APP Over Time" and "Egress New Connections by Source Over Time" show no data:"no egress new connection" what do I have to configure to make it work?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 09, 2019, 03:56:00 pm
Hi @jinn,

Thank you for giving Sensei a try. I see your quoted message did not get response. Sorry for that. I looks like we missed it.

I guess you've been able to figure out the first part yourself. But I wonder why Cloud Threat Intel did not work for you. I'll write to you about this.

For reporting about application categories, yes you can do it. I guess you've started using it.

As for the egress connections report does not show anything. Is it just a single report or all reports which shows egress connections (i.e. local assets, remote assets, eggress conns by source ) do not show anything at all.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 09, 2019, 04:34:43 pm
But the packet engine still stops randomly. I blocked shopping categories to verify whether engine is just shown as stopped or really stopped. And it is really stopped, since I can open shopping page when service is marked as down.

Same with the ElasticSearch service. Sometimes when I open reports or dashboard, I get a pop-up that ElasticSearch service has to be started first and whether I want this. When I deny and check status page, then both services are down - means whenever packet engine stops, ElasticSearch stops, too.

Any hints what could be the reason for the stopping services.

The service crashes pretty soon. 1-2 minutes after starting up.

Hi @hbc,

Thanks for reporting this. After services stop, and when you look at Status-> Services page, do you also see that both services are disabled at boot time?

If yes, most probably this is because Sensei's Health Check subsystem. Because Sensei is in BETA now, checks are more sensitive to problems. Even if it finds a small problem it disables both services in an effort to keep  network connectivity up & running.

Can you try disabling Health Check and see if services are running persistently?

If they do and it turns out because of Health checks, I'd still recommend investigating this. While running Sensei & ES, can you do 'top' on OPNsense console and see if any processes (not necessarily Sensei (eastpect) processes) are consuming much CPU/Memory?

Performance-wise, your system looks pretty decent. We've been reported a similar system handling 700 concurrent users.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on January 10, 2019, 07:52:01 am
Hi @mb,

you are right, I just set ElasticSearch to start on boot and left packet engine disabled for auto-start. I'll try to set both to start on boot.

But I already had try with health check disabled and after a while, no traffic passed at all. But I'll re-check it again.
First with both starting on boot and then with health check disabled.

Update:
The start on boot was not the reason. Whenever packet engine stopped for unknown reason, the option was automatically disabled. I tried it 3 times and reenabled start on boot. But within 5 minutes service stopped again.

As next option I disabled Health Check. Currently the service runs for 20 minutes which is 4 times longer then ever before. I'll keep an eye on it.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 11, 2019, 02:38:11 am
Hi @hbc,

Thank you for further information. Let us know if anything weird comes up.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on January 11, 2019, 07:50:52 am
Hi @MB, I had a similar issue for "Sensei Packet Engine" stops within 5min everytime I enable it. It didn't fix with the reboot as well. But since "health check" is disabled (its been more than 24 hours and reboot few times), service is running without an issue.

I only faced this issue after updated OPNsense to 18.7.10.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: jinn on January 11, 2019, 01:47:37 pm
hey mb, ty for reply!

Quote
For reporting about application categories, yes you can do it. I guess you've started using it.

Not yet. At least not as detailed as I would like (facebook, online shopping, ...)


Quote
As for the egress connections report does not show anything. Is it just a single report or all reports which shows egress connections (i.e. local assets, remote assets, eggress conns by source ) do not show anything at all.

in fact, several do not work: Egress New Connections by App Over Time, Egress New Connections by Source Over Time, Egress New Connections Heatmap, Top Destination Locations Heatmap, Table of Apps (maybe this one is what im really looking for?), Table of Local Assets, Table of Remote Hosts
Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on January 11, 2019, 02:59:16 pm
Good Morning, mb

is it possible to incorporate and additional "TAP" for  Hostname in your tab-bar see picture attachment
Title: Re: Sensei on OPNsense - Application based filtering
Post by: l0rdraiden on January 13, 2019, 10:25:38 pm
What are the plans between sensei and opensense? it will be embedded in opnsense or it will be available as a pluging at some point?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 14, 2019, 06:09:04 am
@hbc, @manjeet: thanks for your update. We're fine-tuning health check auto-bypass.

@cgwork, how about this: likewise destination hostnames, the default will be hostnames for source addresses, and when you move the mouse over the src hostname field, IP address is displayed as a tooltip. Adding another column would make the table more crowded. We'd like to use the space allowance for the new-coming "username" column.

@jinn, are you running Sensei on your LAN? Any chances that it might be on your WAN interface?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 14, 2019, 06:13:58 am
What are the plans between sensei and opensense? it will be embedded in opnsense or it will be available as a pluging at some point?

Hi @l0rdraiden,

It'll be a plugin.

Currently, we're working together to address some issues related to netmap (e.g. virtio). Once it's done, whole integration will be completed, and you'll be able to install it from OPNsense plugin manager.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on January 14, 2019, 07:17:53 pm
Quote
@cgwork, how about this: likewise destination hostnames, the default will be hostnames for source addresses, and when you move the mouse over the src hostname field, IP address is displayed as a tooltip. Adding another column would make the table more crowded. We'd like to use the space allowance for the new-coming "username" column.

That sound even better thank you
Title: Re: Sensei on OPNsense - Application based filtering
Post by: OPNsenseN00b on January 15, 2019, 09:36:30 am
Hello there,
Shortly I've registered on the beta program to obtain the required Downloadlink but ssh is rejecting the provided download link after I login into opnsense. The link is slithly different than in the tutorial.
Could you update the Installer-URL please. Many thanks.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: jinn on January 15, 2019, 09:39:33 am
@jinn, are you running Sensei on your LAN? Any chances that it might be on your WAN interface?

it is currently on LAN. The WAN interface is not displayed to me under available interfaces.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 15, 2019, 02:45:44 pm
Hi @OPNsenseN00b,

The command to install Sensei is:

curl https://updates.sunnyvalley.io/getsensei | sh

I checked again. It should be the same both Users' guide (https://guide.sunnyvalley.io/sensei/getting-started/setup) and Website.

Can you copy/paste the error message you get when you run the command on the OPNsense console?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 15, 2019, 02:46:48 pm
Hi @jinn,

Got it. Will send you a few commands to diagnose the issue.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: OPNsenseN00b on January 16, 2019, 02:04:57 pm
Hi mb,
Thanks for your respnose. This time the firewall comes with the message of not having enough memory (8GB required, I have only 4 GB). Yesterday it said something different. I'll come back to you when I can replicate the previous error.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on January 17, 2019, 06:45:10 am
Hi mb,
Thanks for your respnose. This time the firewall comes with the message of not having enough memory (8GB required, I have only 4 GB). Yesterday it said something different. I'll come back to you when I can replicate the previous error.

8GB is currently required to run Sensei. It checks for that when you first initialize it.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: xames on January 17, 2019, 02:18:41 pm
ssl_error_syscall

I attach
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 17, 2019, 06:33:08 pm
ssl_error_syscall

I attach

Hi @xames,

Looks like everything is ok on the server side. Can you try with fetch:

# fetch https://updates.sunnyvalley.io/getsensei
# sh getsensei


Title: Re: Sensei on OPNsense - Application based filtering
Post by: Space on January 27, 2019, 10:18:35 pm
Hi,

I have Sensei running on my OPNsense and I wondered why big part of the traffic did not show up and I see in the FAQ that IPv6 support is still work in progress.

Do you have an ETA for that feature already?

Thanks and looks great so far!

Space
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 28, 2019, 09:16:05 pm
Hi @Space,

Many thanks for trying Sensei. Yep, 0.7 is IPv4 only.

Good news is that IPv6 will be coming very shortly with 0.8. It's been under testing for the past months. Looks like it's good to go for a test ride by BETA users.

We'll ship 0.8-beta1 this week or early next week :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on February 02, 2019, 10:20:23 am
19.1 goes in production now, and we that have sensei have a problem:
https://forum.opnsense.org/index.php?topic=11400.msg51520#msg51520 (https://forum.opnsense.org/index.php?topic=11400.msg51520#msg51520)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 02, 2019, 10:27:54 am
Hi @Antaris,

Thanks for reporting this. Looking into it now.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 02, 2019, 11:24:53 am
19.1 goes in production now, and we that have sensei have a problem:
https://forum.opnsense.org/index.php?topic=11400.msg51520#msg51520 (https://forum.opnsense.org/index.php?topic=11400.msg51520#msg51520)

Looks like there was a typo in that command. Correct command should be: (from https://forum.opnsense.org/index.php?topic=11400.msg51521#msg51521)

For OpenSSL:

# opnsense-update -fp -n "19.1\/latest"

Or LibreSSL:

# opnsense-update -fp -n "19.1\/libressl"



Just did an OPNsense 19.1 upgrade on two of our firewalls. Looked good. 

Anyone who had any other issues upgrading to 19.1 ?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on February 04, 2019, 09:08:21 am
Quote
Anyone who had any other issues upgrading to 19.1 ?

Update did not work with sensei nor without. Update started and just installed two kernel/base files, then restarted with 18.7.10. Even when sensei was uninstalled, update did not work. I tried GUI and console.

So I saved config, installed 19.1 clean from image and restored backup and reinstalled sensei.

Now with 19.1, sensei finally works with tagged vlan interfaces  8)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 06, 2019, 02:55:31 am
Hi @hbc,

Thanks for sharing you experience. We're looking into the upgrade problem if it's something related to Sensei repository.

Glad to see that you're enjoying it now :)

Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on February 06, 2019, 02:23:14 pm
Yes, works pretty nice. Just the cloud nodes seem a bit flappy. Most time at least one is displayed down.

One hint:

Traffic to local squid proxy on port 3128 is categorized as "Generic TCPIP". I think it is intention that not labeled as 'Proxy' which would properly cause problems when blocking 'Proxy' category.

But maybe you can label it category 'Web Browsing', application 'Squid Proxy'
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 07, 2019, 02:43:32 am
Hi @hbc,

Thank you very much for the feedback. With regard to Cloud servers, we have a fix for that in 0.8.

Thanks for the suggestion. You're right, and suggestion sounds good ;)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 07, 2019, 05:48:33 pm
Dear Sensei users,

Regarding https://forum.opnsense.org/index.php?topic=11477.0;

To be able to utilize the new functionality that comes with the new netmap - enabled kernel, we'll need to ship Sensei 0.8-beta1 which will re-enable virtio interfaces.

Actual ETA was this week. Still working on a few issues reported. Stay tuned for updates. 
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on February 12, 2019, 10:28:26 am
Hi!

Quote
utilize the new functionality that comes with the new netmap - enabled kernel

One question. I had opnsense 19.1 (fresh install) active with shipped kernel and tagged vlans already worked in sensei (what they did not with 18.7). I assume the new c4ec367c3d9(master) kernel is just for virtio interfaces?
Well, I updated kernel and it still works.

Will there ever be the possibility to set different policies for different interfaces? I have interfaces where I would like to be more restrictive and just allow productive things and interfaces where social media, gaming, etc. would be ok.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 13, 2019, 02:38:07 am
Hi hbc,

Yep, correct. VLANs were also broken, but it looks like it was fixed with the FreeBSD 11.2 update. My note was about virtio support. Sensei 0.7 filters out any virtio interfaces. 0.8 will remove this filtering so that they will be presented in the Interface Selection.

Source Interface/Network Address/IP Address/VLAN/User/Group filtering is a feature of Policy based enforcement, which will be showing up with the Premium Subscription.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on February 13, 2019, 09:42:35 pm
Hi hbc,

Yep, correct. VLANs were also broken, but it looks like it was fixed with the FreeBSD 11.2 update. My note was about virtio support. Sensei 0.7 filters out any virtio interfaces. 0.8 will remove this filtering so that they will be presented in the Interface Selection.

Source Interface/Network Address/IP Address/VLAN/User/Group filtering is a feature of Policy based enforcement, which will be showing up with the Premium Subscription.

My advice is to consider exchange "Source Interface/Network Address/IP Address/VLAN/" for volume of users above 1000 or so... It's vital for usability and development at all IMHO.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 14, 2019, 03:22:24 am
@Antaris, Thanks for your input. We'll definitely make use of your feedback.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Kruemel on March 01, 2019, 11:39:29 am
Hi,

greetings from germany.  :)
Great so see such a powerful addon for OPNSense. It was the reason to migrate my APU2C4 to VMWare on HPE ProLiant Xeon CPU, to fulfill the Sensei requirements.

However, it's working great. But I miss a feature: If something is blocked, it's just not loading, right? But the user is not aware, if it's a not working webpage (or parts on it) or if it's blocked. It would be great, if Sensei delivers some kind of block page, something like "This page has been blocked - block category is xxx. Please contact abc@def.de for further information".

Did I miss something in the settings or this feature currently missing?

Keep on the good work!
Cheers
Marco
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 02, 2019, 02:38:46 pm
Hi Kruemel,

From Sunnyvale, California, greetings to you too :) Glad to hear that Sensei is of value to your OPNsense installation. Many thanks for sharing your experience.

We hope to bring some news with regard to less demanding hardware requirements. We're planning to employ an alternative less resource-intensive database engine for reporting.

Quote
But I miss a feature: If something is blocked, it's just not loading, right?

Yep. This is so because, your Sensei policy configuration hits a TLS SNI or application rule. TLS and some app detection jump into the scene way too early before the HTTP protocol starts being conversed back and forth between your browser and the server. 

So when we decide that we need to apply filtering, neither server nor client does not yet know how to talk HTTP. They just know how to talk TCP. This is why we just do a TCP RST, and you see a blank page in your browser.

We'll have a feature called "delayed action" (requires TLS inspection) where we'll flag a particular connection as being blocked and will let them talk a little bit more so that they can handle a HTTP response. As soon as we get a HTTP request from the client, we'll send the landing page and just close the connection at that particular time.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: astoklas on March 03, 2019, 10:27:51 am
Hi,

I just installed Sensei on my OPNsense and I think it's working great.
I found in the dashboard an interesting "HotSpot" I'd like to investigate further. However, the "Top Destinations Locations Heatmap" does not allow for a Drill Down, nor is there a geo location filter available.

Can you please advise on how to investigate on such hotspots?
Is it possible to retrieve DNS/IP for a certain geo location hotspot?

Regards
Alexander
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 04, 2019, 06:46:19 am
@astoklas,

Many thanks for the feedback. Currently, drill-down is not possible with the map. We'll take this as a feature request. Will get you updated.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 04, 2019, 06:56:07 am
Dear Sensei users,

After several months of field testing, we are super happy to announce the availability of Sensei 0.8.0 Beta.

Release 0.8 introduces long awaited support for IPv6 and virtual ethernet adapters. Below is the full list of features that are coming along with this release (from 0.7.0)


For more information: https://www.sunnyvalley.io/blog/sensei-0-8-beta1-is-released

Currently we're shipping 0.8.0 beta1 from a separate package repository. So, if you are on 0.7, you'll not be able to see the software update as of now. When 0.8.0 rc1 is released, we'll move the packages to the main repository and you'll then be able to update to 0.8.0.

The reason behind this is that we want to allow 0.8.0 a bit more field testing before we make it an update for 0.7 stable users.

ETA for 0.8.0.rc1 is March 18, 2019.

If you don't want wait and want to see 0.8 in effect now, just uninstall Sensei from the UI and use the following one-liner command to re-install:

# curl https://updates.sunnyvalley.io/getsensei8 | sh
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on March 04, 2019, 08:46:19 pm
Thanks, mb, and keep up with good work!

Is "VLAN child interfaces support *with OPNsense 19.1.x" means that filtering on VLANs work without netmap kernel?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 04, 2019, 09:15:24 pm
Hi @antaris,

Many thanks. You're correct. It looks like FreeBSD 11.2 default kernel had some fixes with regard to that.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on March 06, 2019, 06:45:49 am
I'm having a problem where elasticsearch won't start after a reboot. I have to clear the settings completely and re setup sensei to get elasticsearch to start.

Just seeing the below in the general log.

Code: [Select]
root: /usr/local/etc/rc.d/elasticsearch: WARNING: failed to start elasticsearch
This is in the backend log and it keeps adding to it.
Code: [Select]
Mar 5 21:44:55 configd.py: [7d62e2b1-bcce-48d3-a80b-4b665aed6cb4] read sensei stats
Mar 5 21:44:54 configd.py: [a4351d00-f929-466b-a18d-1752f72e0a8c] read sensei stats
Mar 5 21:44:53 configd.py: [40ea2e8d-6574-4662-a135-a4c817bf7f0c] read sensei stats
Mar 5 21:44:52 configd.py: [86399ab0-e991-4493-b62f-d6a2b29d88b3] read sensei stats
Mar 5 21:44:51 configd.py: [b8bfc148-83a2-407f-91d3-7091c77b7832] read sensei stats
Mar 5 21:44:50 configd.py: [baf1dddc-39c6-49e4-aad3-f6d87d29a0da] read sensei stats
Mar 5 21:44:49 configd.py: [f08d4d14-f236-4d25-8011-8b25a848eeec] read sensei stats
Mar 5 21:44:48 configd.py: [571d2e9b-d0cb-402c-b5ac-8bf7ff72d811] read sensei stats
Mar 5 21:44:47 configd.py: [e77883ce-8f8b-4a2b-aebb-7c4125ed7e17] read sensei stats
Mar 5 21:44:46 configd.py: [18dd5adf-9437-4e15-90ba-1ee6e08c4bff] read sensei stats
Mar 5 21:44:45 configd.py: [105c9ddc-960b-4bff-98fa-3e202c9ac49e] read sensei stats
Mar 5 21:44:44 configd.py: [87cb6f2f-e3ca-42b0-8040-4cfacd647de8] read sensei stats
Mar 5 21:44:43 configd.py: [4228579b-7e43-4138-8ea8-414fc9ec1c1a] read sensei stats
Mar 5 21:44:42 configd.py: [a755740c-45d8-438c-99e4-a232bd02c661] read sensei stats
Mar 5 21:44:41 configd.py: [024f64e4-2fa6-4558-8482-d8330cbc7742] read sensei stats
Mar 5 21:44:40 configd.py: [327c339b-b0b2-484c-92f9-3c9e9364820e] read sensei stats
Mar 5 21:44:39 configd.py: [396bb45c-c1f1-4728-91d0-e33bbcaea1f5] read sensei stats
Mar 5 21:44:38 configd.py: [d6b674d1-dd2f-494b-927d-ad55791063e4] read sensei stats
Mar 5 21:44:37 configd.py: [40338097-db55-4b60-b45f-877a1ae76b7c] read sensei stats
Mar 5 21:44:36 configd.py: [304857d4-7d26-45aa-ae75-6c520958fba9] read sensei stats
Mar 5 21:44:35 configd.py: [13675e7f-5dc6-4457-b5c9-c4b4c21e8a58] read sensei stats
Mar 5 21:44:34 configd.py: [4f0f6ae9-f39f-48ae-a799-876c86cb3164] read sensei stats
Mar 5 21:44:33 configd.py: [f4a1bb7f-8d12-47bd-b7d3-403d159450b4] read sensei stats
Mar 5 21:44:32 configd.py: [9c67445c-4ffe-444e-ba3c-a5f444ffbf21] read sensei stats
Mar 5 21:44:31 configd.py: [1cfc4b5a-c263-4240-b627-938197d72afe] read sensei stats
Mar 5 21:44:30 configd.py: [adbefd78-9c10-45e9-9cad-8d6495388773] read sensei stats
Mar 5 21:44:29 configd.py: [ad4176d3-1c8a-4890-a90c-c9b734979673] read sensei stats
Mar 5 21:44:28 configd.py: [22ff41e4-fc8f-4ba7-9f27-63d6c2b23b7e] read sensei stats
Mar 5 21:44:27 configd.py: [1fe553d1-06c5-4db6-b950-7a71e5af7bd4] read sensei stats
Mar 5 21:44:26 configd.py: [c3252f98-b238-448a-af02-d311a6f75e49] read sensei stats
Mar 5 21:44:25 configd.py: [09153632-0bff-46ad-ad98-c45319cd5ff8] read sensei stats
Mar 5 21:44:24 configd.py: [0bbec0b1-6e86-4930-a57c-f57be9e83008] read sensei stats
Mar 5 21:44:23 configd.py: [dcf30e51-763b-4df9-9f53-239615912384] read sensei stats
Mar 5 21:44:22 configd.py: [49c214e7-9b60-44c8-9ded-b22ac257f02c] read sensei stats
Mar 5 21:44:21 configd.py: [463b3e7f-c8d6-48ae-8064-08a414fa7e5d] read sensei stats
Mar 5 21:44:20 configd.py: [6ead17e8-53b9-48aa-a6b7-a644d5f170d2] read sensei stats
Mar 5 21:44:19 configd.py: [12378048-9b6d-4c5c-852d-6575fab78706] read sensei stats
Mar 5 21:44:18 configd.py: [bc415b0c-fe6c-404e-a5fb-a99e6b2646bc] read sensei stats
Mar 5 21:44:17 configd.py: [2b46da7d-1325-4e1c-aba0-20bc12e7e4b3] read sensei stats
Mar 5 21:44:16 configd.py: [720bebee-2387-4735-b794-085b94f5b505] read sensei stats
Mar 5 21:44:15 configd.py: [829b4c54-6629-4ae1-81fc-5a3255ba1c91] read sensei stats
Mar 5 21:44:14 configd.py: [80d84ec1-5cee-4f60-9290-bcaba50a351d] read sensei stats
Mar 5 21:44:13 configd.py: [6b233cd4-81d2-4569-99f6-2989332cb14b] read sensei stats
Mar 5 21:44:12 configd.py: [31706105-d805-41bf-b201-8f75e72fe5b3] read sensei stats
Mar 5 21:44:11 configd.py: [e0f1c395-db7e-4ee1-bdd7-e20ee8ff1dfa] read sensei stats
Mar 5 21:44:10 configd.py: [3f704530-859b-4e1f-95dd-136f85219d4b] read sensei stats
Mar 5 21:44:09 configd.py: [ab29e24e-2146-49e3-9bb6-fb6064233ff2] read sensei stats
Mar 5 21:44:08 configd.py: [645ca172-5629-4ea5-ad1f-8538c1b1ea06] read sensei stats
Mar 5 21:44:07 configd.py: [f8b70f86-0bee-4880-9306-bb4450d7db4d] read sensei stats
Mar 5 21:44:06 configd.py: [8bd95d71-bd13-4ec0-8f27-ed3932579bd3] read sensei stats
Mar 5 21:44:05 configd.py: [be4feb64-ef8e-4756-9e0c-0bbe00f5d4d0] read sensei stats
Mar 5 21:44:04 configd.py: [1aa6cf3a-da0e-473c-b710-553aa1287d69] read sensei stats
Mar 5 21:44:03 configd.py: [12d70d27-8724-477b-a274-99e795bcac42] read sensei stats
Mar 5 21:44:02 configd.py: [91adebc2-e1ee-4cf8-87c2-e1d8a5e8eee1] read sensei stats
Mar 5 21:44:01 configd.py: [ac505fe1-4ebb-4c68-99a7-a684c7f43a99] read sensei stats
Mar 5 21:44:00 configd.py: [7acfc145-9a17-40eb-be37-841d034621e7] read sensei stats
Mar 5 21:44:00 configd.py: [92b767af-81f1-4a5e-9e00-25219f89c715] check sensei engine health
Mar 5 21:43:59 configd.py: [d32f3278-e509-4969-b4a8-7ae7c79c700c] read sensei stats
Mar 5 21:43:58 configd.py: [ad2a102f-b1e0-4bb5-a593-09df77d04bac] read sensei stats
Mar 5 21:43:57 configd.py: [b92813e9-1cef-4b7f-8480-87b49d02d4f6] read sensei stats
Mar 5 21:43:56 configd.py: [d54e5bf2-f367-428a-a8d6-831488f4023e] read sensei stats
Mar 5 21:43:55 configd.py: [189af746-8852-4feb-bc24-2a13da1ff032] read sensei stats
Mar 5 21:43:54 configd.py: [dc2193ce-51c2-451e-917e-ebd56814ad1a] read sensei stats
Mar 5 21:43:53 configd.py: [08950c34-f59e-4fa5-95d5-0af61c02bdd1] read sensei stats
Mar 5 21:43:52 configd.py: [ea882489-9044-4768-b09c-ed6a0d5edd6d] read sensei stats
Mar 5 21:43:51 configd.py: [a4beae9e-0848-46df-bfd2-9e884d455d64] read sensei stats
Mar 5 21:43:50 configd.py: [66bc19f1-867a-4cff-bd31-e21221374c82] read sensei stats
Mar 5 21:43:49 configd.py: [1cff607f-dfba-4adb-8839-82dc49b1b83f] read sensei stats
Mar 5 21:43:48 configd.py: [7fee0851-b848-48d8-8d26-bc84b8bdce1b] read sensei stats
Mar 5 21:43:47 configd.py: [a5261abd-d409-4b27-921c-4f7f7ec41b90] read sensei stats
Mar 5 21:43:46 configd.py: [b8b7127a-5d56-408d-b7dd-902dd95e9ea2] read sensei stats
Mar 5 21:43:45 configd.py: [48a32138-cf91-4641-be4f-045f04ec7af6] read sensei stats
Mar 5 21:43:44 configd.py: [8c4ef497-2b33-4144-ba5b-4ef31a654070] read sensei stats
Mar 5 21:43:43 configd.py: [37cfb408-8ef5-408b-9348-53bcbb5bd089] read sensei stats
Mar 5 21:43:42 configd.py: [939282e0-234c-4b5f-ab00-9113bd803c96] read sensei stats
Mar 5 21:43:41 configd.py: [2989a365-034b-4aa6-b69f-a11ad3bd61c9] read sensei stats
Mar 5 21:43:40 configd.py: [5264a79b-1cf0-4d63-83a7-01129eead1ce] read sensei stats
Mar 5 21:43:39 configd.py: [3a8b90d3-46eb-494f-a19f-78817048cd12] read sensei stats
Mar 5 21:43:38 configd.py: [950f188d-26bd-4e9c-ac76-d65cdb48e212] read sensei stats
Mar 5 21:43:37 configd.py: [cea553fe-507d-492d-ab6d-f4318a600400] read sensei stats
Mar 5 21:43:36 configd.py: [f5b111b5-b585-4843-83bb-0a1bbfb2c1cd] read sensei stats
Mar 5 21:43:35 configd.py: [606ca68b-d3c0-4331-b410-afd4fef1a96c] read sensei stats
Mar 5 21:43:34 configd.py: [995954f6-fa00-4a3a-b32a-5638fa5eaffc] read sensei stats
Mar 5 21:43:33 configd.py: [3a856c39-6a60-4c23-83d7-15e7a00c2472] read sensei stats
Mar 5 21:43:32 configd.py: [3cfda134-4227-4c55-bcca-8ee10229e527] read sensei stats
Mar 5 21:43:31 configd.py: [9e43feed-c461-47fa-b692-8d445f317f4f] read sensei stats
Mar 5 21:43:30 configd.py: [02568a2b-6285-4431-bd2e-081b6bc3d77e] read sensei stats
Mar 5 21:43:29 configd.py: [72dbb649-88a3-4991-b51a-47c698256ce4] read sensei stats
Mar 5 21:43:28 configd.py: [1473e74d-fce9-4173-a6fa-bf54eb577778] read sensei stats
Mar 5 21:43:27 configd.py: [4a6222fc-465d-4528-9dcc-c906a5de1855] read sensei stats
Mar 5 21:43:26 configd.py: [b82dd2a5-8c9a-4a02-be10-6ad52bbaac5e] Show system activity
Mar 5 21:43:26 configd.py: [670749ac-91e3-4643-a9c4-5b9fd44f94da] read sensei stats
Mar 5 21:43:25 configd.py: [30d3970c-86fe-4d91-bca6-7353c654df63] read sensei stats
Mar 5 21:43:25 configd.py: [9a8daded-b8e5-4f51-bc56-d016e8ac7c02] read sensei stats
Mar 5 21:43:24 configd.py: [ebb18255-5159-4ab9-b641-b88821bf1e7d] read sensei stats
Mar 5 21:43:24 configd.py: [5120fa8d-e8ef-48a4-96e9-ffe553f81d30] read sensei stats
Mar 5 21:43:23 configd.py: [b727b40c-13ef-4d1e-b251-bf71c98a5b2f] read sensei stats
Mar 5 21:43:23 configd.py: [3634a274-5368-48a6-8867-b9932cd4809d] read sensei stats
Mar 5 21:43:22 configd.py: [0fb20dcf-c03b-4582-9c36-535207c9fa7f] read sensei stats
Mar 5 21:43:22 configd.py: [7d93ab3c-e1d8-452a-9863-c048ca11e7ff] view elasticsearch disk size
Mar 5 21:43:22 configd.py: [f09b62e6-cbf1-41be-97ae-56cce24ed05f] control services
Mar 5 21:43:22 configd.py: [e52be1cb-68be-4eea-b9e1-6c7b0f4e583c] check sensei ui version
Mar 5 21:43:22 configd.py: [02277005-468d-418c-aeea-5f26e03a016a] check sensei db last modified
Mar 5 21:43:22 configd.py: [5d851b8a-fda4-41cc-9967-7fe8ac178622] check sensei db version
Mar 5 21:43:22 configd.py: [99541288-f562-4f59-aa05-8a9b326cac81] check sensei db last modified
Mar 5 21:43:22 configd.py: [a29ac723-7f8f-41c0-8f73-26d60fc2493e] check sensei db version
Mar 5 21:43:22 configd.py: [37de4a96-014a-47fb-b12c-9c6c6aef5f37] check sensei last modified
Mar 5 21:43:22 configd.py: [7b58d2c8-5505-4df3-8a36-c4a6cf63c70b] check sensei version
Mar 5 21:43:22 configd.py: [9f2677fa-a66d-4e81-9d48-3191f60db682] control services
Mar 5 21:43:21 configd.py: [271b39f0-44fd-4ca1-9a0d-57e074e2ac8c] read sensei stats
Mar 5 21:43:20 configd.py: [8be4d78e-c447-4ff4-92b9-8d2de2a0b9a1] view license
Mar 5 21:43:20 configd.py: [ed3ffc6c-13a6-4468-b09d-2c2cba7469d6] read sensei stats
Mar 5 21:43:19 configd.py: [8483e0c4-6b9e-4cb6-a9ff-ac0cceed2488] read sensei stats
Mar 5 21:43:19 configd.py: [eb9e9a55-1aa1-4ece-a8cb-f71a0b1e3d0c] control services
Mar 5 21:43:18 configd.py: [caaf4bb7-d2af-4258-bba1-960e1b3b3bcb] read sensei stats
Mar 5 21:43:17 configd.py: [77b7f220-2a12-4238-a4f4-622639abb5a2] read sensei stats
Mar 5 21:43:16 configd.py: [fbb0669d-a17f-4918-b158-f28d2cc86aae] read sensei stats
Mar 5 21:43:15 configd.py: [f22ac12a-fdbe-45aa-9e2e-cd75abbc5c68] read sensei stats
Mar 5 21:43:14 configd.py: [04bf4e69-7021-48d4-a14c-429bad0bcd9e] read sensei stats
Mar 5 21:43:13 configd.py: [7f0bca65-1c34-45a5-9816-192eedcadc21] read sensei stats
Mar 5 21:43:13 configd.py: [cde48204-6443-48be-93b8-5c57c8d3cb4b] read sensei stats
Mar 5 21:43:12 configd.py: [d9669127-1ec6-482b-9800-34bf1090604d] read sensei stats
Mar 5 21:43:12 configd.py: [9fd1971a-e907-4704-b0b6-9ef8c193b4a0] read sensei stats
Mar 5 21:43:11 configd.py: [7e084ad4-bd04-40b7-a269-f86b030d470b] read sensei stats
Mar 5 21:43:11 configd.py: [e2f40c45-1449-4eaa-adad-392535ab65b9] read sensei stats
Mar 5 21:43:10 configd.py: [c06c00d0-29c3-424c-805a-624b8bb86c2c] read sensei stats
Mar 5 21:43:10 configd.py: [d44777a5-aede-4403-9963-65f5caf835f8] read sensei stats
Mar 5 21:43:09 configd.py: [5d031005-ce3b-4ddb-b119-c15818b64d7c] read sensei stats
Mar 5 21:43:09 configd.py: [4aaab29d-dd26-499b-8a94-114f728d447c] read sensei stats
Mar 5 21:43:08 configd.py: [32811901-60a5-41fb-8a70-23df003b409a] read sensei stats
Mar 5 21:43:08 configd.py: [e7f2cf0d-5ba4-4b5e-bb0f-6483884c55a7] read sensei stats
Mar 5 21:43:07 configd.py: [7e830b6f-f83d-417e-ad4c-a9ed577644dc] read sensei stats
Mar 5 21:43:07 configd.py: [997cb509-1145-43ea-a461-ed291432856c] read sensei stats
Mar 5 21:43:06 configd.py: [54e86060-313f-4c37-b7c8-ce55f24c5363] read sensei stats
Mar 5 21:43:06 configd.py: [b580155d-f96d-4c35-a94a-19b784208558] read sensei stats
Mar 5 21:43:05 configd.py: [eeddf8f5-89b1-491e-a627-aa879133e63a] read sensei stats
Mar 5 21:43:05 configd.py: [4beb04bf-4103-48ae-86ed-98c9ee7f96d0] read sensei stats
Mar 5 21:43:04 configd.py: [08eac025-5388-4807-9da7-f1d6004c4926] read sensei stats
Mar 5 21:43:04 configd.py: [106e18d5-ee88-4dba-b5e7-6d0d4921d065] read sensei stats
Mar 5 21:43:03 configd.py: [3532ac59-95e9-4439-9837-7a1ab5188a8a] read sensei stats
Mar 5 21:43:03 configd.py: [966fa7d7-c5f7-4809-b72f-fafd7e230bf0] read sensei stats
Mar 5 21:43:02 configd.py: [c87d2a2b-3b5c-44be-8e78-5fc89b1ee7b4] read sensei stats
Mar 5 21:43:02 configd.py: [fbc26fe4-dfc6-4991-bf26-6fa726d28c13] read sensei stats
Mar 5 21:43:01 configd.py: [2cfd5f28-21ce-4651-8a6f-68d7bc4ee5bf] read sensei stats
Mar 5 21:43:01 configd.py: [ad503b54-302c-4534-961b-7f4ffd830022] read sensei stats
Mar 5 21:43:00 configd.py: [edd42365-060e-4e8f-8bfb-9022ae8630e2] read sensei stats
Mar 5 21:43:00 configd.py: [9dc39d58-07bd-443d-bd2d-781a88573d10] read sensei stats
Mar 5 21:43:00 configd.py: [bf2bdcc2-2775-40c7-98c9-512ff7032409] check sensei engine health
Mar 5 21:42:59 configd.py: [ef64a92c-1456-4c26-92fd-72d259adfb70] read sensei stats
Mar 5 21:42:59 configd.py: [bd987828-89f8-46c4-8104-1f78e2c395da] read sensei stats

I attached the elasticsearch log. This only happens after a reboot with sense .8 beta 1 installed.

Here is the error I get when I start elasticsearch from the shell

Code: [Select]
root@OPNsense:~ # service elasticsearch start
Starting elasticsearch.
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME
/usr/local/etc/rc.d/elasticsearch: WARNING: failed to start elasticsearch


Looks like the java env variable isn't being saved in the elasticsearch file or getting overwritten on a startup.

I ran this part of the sensei-init.sh script manually and elasticsearch started with no error now.

Code: [Select]
echo -n "Setting up elasticsearch..."
mkdir -p /usr/local/lib/elasticsearch/plugins
chmod -R 755 /usr/local/lib/elasticsearch/plugins
sysrc elasticsearch_login_class="root" >/dev/null 2>&1
sed -i '' -E '/auto_create_index/d' /usr/local/etc/elasticsearch/elasticsearch.yml
echo "action.auto_create_index: false" >> /usr/local/etc/elasticsearch/elasticsearch.yml
/usr/bin/sed -i '' 's/opt\/eastpect\/run\/elasticsearch/var\/run\/elasticsearch/g' /usr/local/etc/rc.d/elasticsearch
/usr/bin/sed -i '' 's/Xms512m/Xms2g/g' /usr/local/etc/elasticsearch/jvm.options
/usr/bin/sed -i '' 's/Xmx512m/Xmx2g/g' /usr/local/etc/elasticsearch/jvm.options
echo 'elasticsearch_enable="YES"' > /etc/rc.conf.d/elasticsearch
echo 'elasticsearch_env="JAVA_HOME=/usr/local/openjdk8"' >> /etc/rc.conf.d/elasticsearch
echo "done"
I'm fairly certain it's the second to last line that's fixing elasticsearch. Just why that isn't surviving past a reboot is beyond my skill set with this.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 06, 2019, 02:09:11 pm
donatom, thanks for the detailed report.

You are right, it's:

echo 'elasticsearch_env="JAVA_HOME=/usr/local/openjdk8"' >> /etc/rc.conf.d/elasticsearch

that's fixing it. JAVA_HOME variable should be set to openjdk8 directory.

We're having a look at it why it is not persisting.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on March 06, 2019, 03:57:19 pm
Mb,

Beyond the elasticsearch issue everything else is working so far. IPv6 is definitely working and blocking categories.
With .7 my ram usage would hover around 4.8gb. With .8 it started around 4.8 but when I went in this morning dropped down to 2.7gb. The only time ram dropped on .7 was when elasticsearch had crashed.

I don’t know if it’s from enabling ipv6 again on my lan or something with .8 but web pages are loading quicker by a noticeable margin as well. I did also turn on cloud threat intel so it could be that too.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 07, 2019, 02:28:48 am
Hi donatom3,

Many thanks for the detailed feedback. Very good to see 0.8 with IPv6 is running good.

We've fixed a bug with regard to the Elasticsearch rc script. Our configuration manager was overriding it under a condition. Now elasticsearch starts on boot with no problem.

Wait for 0.8.0.beta2 update. It should be arriving momentarily.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: cfsl1994 on March 09, 2019, 02:27:36 am

Good day to all  :),

Recently I'm trying out the sensei package at OPNsense and I thought it was very good, it left me surprised. My questions are:

I would like to know if the primium subscription option is available?

How can I apply filtering for certain IPs?

Thanks
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 09, 2019, 02:46:22 am
Hi cfsl1994,

Many thanks for sharing your feedback. Great to see that Sensei is up to your expectations.

Yep, premium subscription will be available and will come with source IP/network based filtering. You'll be able to create custom policies and apply them to different user groups.

We expect to have Sensei 1.0 in early April and will start offering Premium subscription beginning early May.

Beginning with 1.0 version, Sensei will be directly instalable from OPNsense plugin manager.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: opnsenseuser on March 09, 2019, 10:10:53 am
Hi cfsl1994,

Many thanks for sharing your feedback. Great to see that Sensei is up to your expectations.

Yep, premium subscription will be available and will come with source IP/network based filtering. You'll be able to create custom policies and apply them to different user groups.

We expect to have Sensei 1.0 in early April and will start offering Premium subscription beginning early May.

Beginning with 1.0 version, Sensei will be directly instalable from OPNsense plugin manager.

I would wish to incorporate a function that may have fewer features, but also works on low end cpu's better or at all works.
Because in order to really use sensei you need a cpu that consumes a lot of electricity and therefore generates a lot of costs for the private user.
I would be very happy about such a feature and certainly others as well.

Thanks for the great product! Regards rené
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 09, 2019, 02:22:25 pm
Hi rené,

Many thanks for sharing your suggestion.

I'd like to happily tell that we have two ongoing projects which involve:

1. To make Sensei run on very low end devices, which have weak CPU and memory under 1GB. 
2. To make Sensei run on very large deployments e.g. sites with thousands of users.

For the former, the hurdle is the backend database. Although it's very efficient for medium to large settings, Elasticsearch is heavy for small deployments. It simply does not successfully run under 4GB memory. We're currently evaluating and testing several other databases which will do the job for small settings.

Expect to hear more on this late fall this year.

With regard to the latter, also this year, we'll announce a solution which will be able to handle many thousand concurrent users.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: opnsenseuser on March 09, 2019, 04:03:52 pm
Hi rené,

Many thanks for sharing your suggestion.

I'd like to happily tell that we have two ongoing projects which involve:

1. To make Sensei run on very low end devices, which have weak CPU and memory under 1GB. 
2. To make Sensei run on very large deployments e.g. sites with thousands of users.

For the former, the hurdle is the backend database. Although it's very efficient for medium to large settings, Elasticsearch is heavy for small deployments. It simply does not successfully run under 4GB memory. We're currently evaluating and testing several other databases which will do the job for small settings.

Expect to hear more on this late fall this year.

With regard to the latter, also this year, we'll announce a solution which will be able to handle many thousand concurrent users.

if you really want to do that and really do it, some users would be very grateful to you. Me too of course! ;-)

How can I or how can others keep an eye on the development of this feature?
Is there a kind of roadmap or something similar?

Greetings René
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 10, 2019, 12:27:05 am
Quote
if you really want to do that and really do it, some users would be very grateful to you. Me too of course! ;-)

How can I or how can others keep an eye on the development of this feature?
Is there a kind of roadmap or something similar?

Hi René,

We will do it :) You're all welcome.

To keep up with the development, roadmap etc, best is to keep following this forum thread and also following company web site and twitter account:

https://twitter.com/sunnyvalley

Beginning April, we'll share more information about the upcoming feature set and more about the technology.

For now, I can tell that the technology at the heart of Sensei is a powerful packet analysis engine which is aimed at providing contextual network visibility, protection at all ports for all devices and also protection against encrypted threats which are gaining momentum.

Utilizing this core tech, our mission is to provide enterprise grade cyber protection for everyone, let it be a household, a small business or an enterprise with thousands of users.

From this perspective, making Sensei run on any scale is our priority.

 
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on March 10, 2019, 05:20:55 am
And you start working on getting it to run on lower end machines after I order the new qotom case with 6 built in intel nics and a lga 1151 slot for 6th of 7th gen core desktop processors.

It's the Qotom Q600G6 for anyone interested.
https://www.aliexpress.com/item/Qotom-DIY-Powerful-Firewall-Router-Appliance-Q600G6-Barebone-System-Support-6th-7th-Gen-Processor-DDR4-RAM/32967092263.html?spm=a2g0s.9042311.0.0.154d4c4d2CNERH
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on March 16, 2019, 01:44:03 pm
HI, I Can not open report in either Dashboard or Reports giving me an error "An error occurred while report is being loaded!".

In view error message it says:
{
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": []
  },
  "status": 503
}

Both "Sensei Packet Engine" and "Elasticsearch" are running. I have restarted the system and error is still there.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 16, 2019, 04:41:36 pm
Hi manjeet,

Thanks for reporting this. Are you on 0.7?

We've got two more reports for the same problem and currently investigating it.

We'd like to dig deeper. Can you share your relevant elasticsearch.log ( located at /var/log/elasticsearch/ ) through sensei - at - sunnyvalley.io ?

For a workaround, you can run these two commands to reset the indexes: (beware: this will erase your reporting history)

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py

Let us know if this does not fix the problem.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: ltb76 on March 17, 2019, 04:41:31 pm
Hi,

I'm new to OPNsense and Sensei, testing it to replace my soon expering PaloAlto home firewall.

Just did a default install and it seems to be working well (I see several blocked add sites under "Blocked Sites Explorer").
I might be missing something though. I tried adding "Bing" under "App Controls" - however I can still access bing.com. (I then tried adding Facebook - and that blocks Facebook). might the "bing" app be broken or am I missing something?

Another question, I looked in the manual but did not find the answer. Initially I added all my interfaces (WAN, LAN, LAN2 and DMZ) under "Protected Interfaces". dooing that seems to block DNS.
With the WAN interface protected, DNS trafic seems to be blocked with "Network Management category is administratively restricted" - even if does not appear to be blocked under "App Controls". Should I only add "LAN" interfaces to "protected"?

Is there a way to "not protect" an IP on a protected interface? Lets asume I have a device / client on the LAN interface that I for some reasone want to bypass all checks - is that posible?

I'm running
Sensei: 0.8.0.beta4
OPNsense: 19.1.4
Running ontop of VMware, 4 vCPU (D1540), 12GB RAM, vmxnet3 NICs
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on March 17, 2019, 05:58:19 pm
Quote
Should I only add "LAN" interfaces to "protected"?
AFAIK Sunnyvalley recommends not to block WAN and use suricata for this instead.

Quote
Is there a way to "not protect" an IP on a protected interface?
Not in the free version. That is a feature of the premium edition.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on March 19, 2019, 09:08:30 am
Thanks @MB. This fixed the issue.

I am currently running 0.7 & I am sending you the email for logs and screen shot error.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on March 19, 2019, 09:35:04 pm
I have a question about the VLAN feature.
I use some VLAN on OPNSense and added all my interfaces to the "protected interfaces".
After that all connected VM´s inside the VLAN´s are offline and unable to access the opnsense (which means they are offline for all networks)

If i remove the "LAN" interface from the "protected interfaces" which is my physical interface,
the access from the VM´s inside the VLAN´s is ok again.
I have clients connected to "LAN" as well and would like to protect them, too.

Here is a overview:

LAN (em0) is my physical device and all VLAN are added to this interface:

Code: [Select]
10_DMZ (em0_vlan10) -> v4: 172.16.10.254/24
                    v6/t6: 2003:f2:63c9:63e1:4c1f:32ff:fe6d:4ae/64
 20_VPN (em0_vlan20) -> v4: 172.16.20.254/24
 30_Pentest (em0_vlan30) -> v4: 172.16.30.254/24
                    v6/t6: 2003:f2:63c9:63e3:4c1f:32ff:fe6d:4ae/64
 40_WifiGuest (em0_vlan40) -> v4: 172.16.40.254/24
                    v6/t6: 2003:f2:63c9:63e4:4c1f:32ff:fe6d:4ae/64
 50_IoT (em0_vlan50) -> v4: 172.16.50.254/24
                    v6/t6: 2003:f2:63c9:63e5:4c1f:32ff:fe6d:4ae/64
 60_Dev (em0_vlan60) -> v4: 172.16.60.254/24
                    v6/t6: 2003:f2:63c9:63e6:4c1f:32ff:fe6d:4ae/64
 70_WiFi (em0_vlan70) -> v4: 172.16.70.254/24
                    v6/t6: 2003:f2:63c9:63e7:4c1f:32ff:fe6d:4ae/64
 80_Server (em0_vlan80) -> v4: 172.16.80.254/24
                    v6/t6: 2003:f2:63c9:63e8:4c1f:32ff:fe6d:4ae/64
 90_Clients (em0_vlan90) -> v4: 172.16.90.254/24
                    v6/t6: 2003:f2:63c9:63e9:4c1f:32ff:fe6d:4ae/64
 LAN (em0)       -> v4: 172.16.17.254/24
                    v6/t6: 2003:f2:63c9:63e0:4c1f:32ff:fe6d:4ae/64
 PIA_VPN (ovpnc1) -> v4: 10.56.10.6/32
 WAN (igb0)      -> v4: 192.168.217.2/24
                    v6/DHCP6: 2003:f2:63c9:6300:6eb3:11ff:fe1b:aedf/64


I´m on Sensei 0.8.0.beta4 and OPNsense 19.4.1

Do you need some more informations ?
Thank you!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 19, 2019, 09:41:29 pm
Hi BeNe,

We're aware of this issue. There's another Sensei deployment exactly the same setting with yours and experiencing the same problem.

Looks like something weird with em-vlan-netmap trio. We're on this. Will update the thread when it's done.

One question: are you fine when you remove the trunk interface and just protect vlan child interfaces?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on March 19, 2019, 10:21:31 pm
Hi mb,

thanks for that fast information.

Yes, if i remove the trunk Interface (LAN em0 in my case) from the protected interfaces list, the machines inside the VLAN 's are reachable again.

Gesendet von meinem Pixel 2 mit Tapatalk

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 20, 2019, 06:12:34 pm
Hi Bene,

All welcome. Thanks for the information. Can I ask a favor? Can you try the new netmap kernel to see if your current setup works? (child interfaces protected, trunk not protected).

Here's how to do it:

https://forum.opnsense.org/index.php?topic=11477.msg55261#msg55261


Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on March 20, 2019, 08:09:48 pm
Hello Murat,

of course  ;) But the problem is still the same. I installed the new Kernel:
Code: [Select]
# uname -a
FreeBSD surtur.my-network.de 11.2-RELEASE-p9-HBSD FreeBSD 11.2-RELEASE-p9-HBSD  4ea457eb7b8(master)  amd64
If i add "LAN (em0)" to the protected interfaces, the VLAN´s are offline.
So revert back to the stock kernel. Added a screenshot from my OPNsense Console after adding the interface.