OPNsense Forum

English Forums => Development and Code Review => Topic started by: mb on August 25, 2018, 03:38:14 am

Title: Sensei on OPNsense - Application based filtering
Post by: mb on August 25, 2018, 03:38:14 am
Hello,

I'm Murat, founder of Sunny Valley Networks, the company behind Sensei.

Very much pleased to meet the OPNsense community.

I've seen a thread about Sensei in the forum, so I thought it might be a good idea to start a dedicated topic to help people with the software.

Sensei is a plugin for firewalls which complement them with features like Application Filtering, Advanced Network Visibility and Cloud Application Control. Currently, Sensei community edition is available for OPNsense platform.

I've seen that some members have already downloaded and trying Sensei. Many thanks for that. We're grateful.

I've created this topic about Sensei to help you to try it out, and try to solve any problems you guys might have encountered.

Although we reached our target number of beta testers, we always have room for forum members.
If you're interested in trying it, please do not hesitate to contact me privately. I can share the URL to the latest installer.

Very much looking forward to reading your feedback and helping you with the software.

More information about Sensei can be found on the product web page: https://sunnyvalley.io/sensei

All the best
Murat
Title: Re: Sensei on OPNsense - Application based filtering
Post by: marjohn56 on August 26, 2018, 12:05:21 pm
Thanks to @mb for sending me a link to test this. This is a quick summery of my first impressions, also to prevent any cross-contamination issues I did a clean install using zfs and then bootstrapped opnsense install. Firmware flavour is development and core upgrade carried out.


Installation was straight forward as was configuration. Initial configuration left me with zero information, this appears to be because I had selected the LAN as the interface to monitor, however, my LAN is a bridge, changing this to the OPT1,OPT2,OPT3 interfaces solved this and then it all started working well.


Note I am using this on a Qotom i5 with 8Gb RAM. It is recommended that this is the minimum requirement for a 100 user system. On my test system there is minimal extra load on the CPU, but my test system is limited to only two devices attached to the LAN.


My first impressions are that is a very impressive package, it will be interesting to see what the differences will be between the commercial and community editions are when that time arrives.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on August 27, 2018, 07:43:54 am
@marjohn56, many thanks for giving Sensei a try and providing feedback. This is very valuable for us.

Glad to hear that installation & configuration went smooth.

Sensei utilizes netmap behind the scenes, which does not play well with bridged interfaces. Netmap in FreeBSD 11.x, which OPNsense is based on is quite old.  I think we can also contribute to OPNsense team with an improved netmap support. I believe this will also help resolve some Suricata issues.

We'd love to hear about performance figures with a larger user base if you happen to have access to one. Currently the largest deployment we know of is 200 Mbps sustained WAN throughput with about 850 users. HW is an old HP DL360-g8 (xeon e5-2450L @1.8GHz) and 16GB RAM.

Delighted to see that product is up to the duty.

Enterprise <-> Community edition work is ongoing. For now I'm happy to tell that community edition for OPNsense will always be there and forever free.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: Mundan101 on August 29, 2018, 02:01:30 pm
I have sensei up at running and so far so good!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: marjohn56 on August 29, 2018, 03:10:48 pm
I have sensei up at running and so far so good!


Just in case @mb has not told you, IPv6 is still WIP, so v4 only for now, still cool though  :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on August 30, 2018, 01:18:22 am
@Mundan101, thank you for testing and giving feedback.

@marjohn56, thank you for pointing it out. It's been FAQ'd now :)

To better support the software and help people who are having issues, we've created a Gitlab project.

Please feel free to send any bug-reports & enhancement requests there:

https://gitlab.com/svn-community/opnsense-sensei-plugin

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mimugmail on August 30, 2018, 09:16:18 am
@mb https://www.sunnyvalley.io/eastpect
What about TLS 1.3?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: svn on August 31, 2018, 01:10:20 am
Hi @mimugmail,

I am Hayati from SVN team.

As you probably know that TLS 1.3 has been finalized in this month after 28 drafts. TLS 1.3 will obviously dominate over other versions and most of the Linux/Unix distros and libraries should be giving support for it, sooner or later. This is no different for us.

We've been closely watching its progress and discussions on the TLS working group during our whole product development. So we expected and prepared for it, and Sensei's TLS inspection has been designed by taking TLS 1.3 into account. We'll be able to provide TLS 1.3 inspection without downgrading TLS version.

We expect the transition to TLS 1.3 in the field will start with the popular tls libraries following with the applications that are dependent on them. This will take some time. We target to be among the first network security providers to support TLS 1.3 with its most potential.

I've uploaded a video to SVN youtube channel illustrating TLS Inspection in action: https://www.youtube.com/watch?v=krG_VKt2_qk
Title: Re: Sensei on OPNsense - Application based filtering
Post by: samsonmcnulty on September 01, 2018, 12:12:45 am
Thanks you guys! I don't have a large userbase but I'll definitely report anything I come across. So far I really like it. My main goal at the moment is to see how it plays with squid and caching. I'm also using suricata and clamAV. I noticed a mention of some issues with suricata but that you were aware and working on a fix.
Edit I've seen a few people on 200Mb connections but I haven't seen many at 1Gb. Are you planning to add traffic shaping abilities? based on category?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 01, 2018, 03:46:59 pm
Hi @samsonmcnulty,

Thank you for testing & feedback. I'd very much appreciate if you can report any problems and/or issues you encounter.

Just like filtering based on application, shaping will also be there ;) Tentative plans is that we expect it to arrive in 2019.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sagem2004 on September 01, 2018, 04:37:58 pm
hello

can we block websites can be an integration in opnsense native

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 01, 2018, 07:58:14 pm
Hi @sagem2004,

Was your question about Sensei filtering based on web sites?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sol on September 03, 2018, 10:12:19 am
Great plugin so far.

On my machine running with 8GB RAM and an Intel I5 5250U (2x 1,6GHZ) the WAN throughput is at approx. 85 Mbps using IPS, Proxy + AV and around 8 active users.
Without Sensei my box can use the full 150 Mbps line (Cpu load is around 60 - 70%).
It takes a while to load on the first time and for some reason I cannot disable Sensei.
Due to the reduced internet speed I had to uninstall it and will give it another try once I have a faster router.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 03, 2018, 12:54:09 pm
Hi @sol,

Thank you for trying out Sensei and for the feedback.

A couple of questions:

Is this CPU usage (60-70%) for the configuration Sensei is not running? (e.g. IPS+Proxy+AV) ?

When you launch Sensei, how much did you see it changed? Does it top to 100%?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: krdhtet on September 04, 2018, 09:42:14 am
Dear mb,

Could you kindly provide Sensei link for me?

Thanks you.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 04, 2018, 07:31:35 pm
Hi @krdhtet,

You got it in your inbox ;)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sol on September 05, 2018, 06:02:06 pm

A couple of questions:

Is this CPU usage (60-70%) for the configuration Sensei is not running? (e.g. IPS+Proxy+AV) ?

Yes

When you launch Sensei, how much did you see it changed? Does it top to 100%?
 It goes up to 95% and drops to ~50%. It also drops and peaks way more often


Furthermore I couldnt disable Sensei and I was only able to uninstall it right after a reboot. 
After a new try to install it again over the current system opnsense crashed and it had to reinstall Opnsense.
I guess some old settings made a clean reinstallation of Sensei impossible.
Lets hope that a new Sensei version will fix the option to stop it.

Looking forward to an update and will give it a try another time.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 05, 2018, 07:28:31 pm
Hi @sol,

Many thanks for reporting this and for the answer. This is very much helpful to understand what's going on.

Looks like a quite loaded system. I would not recommend running with a 60-70% cpu utilization if you're doing some kind of packet processing. Because packet processing requires dedicated resources and if the cpu is highly utilized and also shared with other applications, it's highly possible that you'll start losing packets. This is so, because at some point OS will fail to schedule the packet processing application to a CPU (because the CPU is already busy) and packets will be dropped in this timeframe. As a consequence,  this will create congestion, and finally you'll get lower throughput. This was what happened, lowering your throughput from 150 - 85 Mbps.

To remedy this kind of heavy load scenarios, there is one thing you can do, and one thing we can:

For you, as you wrote before, it'd be better if you can run the configuration with a more resourceful HW.
For Sensei, we'll pin it to a dedicated CPU core. This will help if you have a multi-core system. 

For not being able to stop Sensei, I'd guess it's related to the above scenario. Though it should stop anyway whatever the load is.

We'll try to reproduce this with your conditions in our lab. I'll let you know about our results.

For the sake of clarity: were you trying to stop it by clicking on the  "Stop" action button or by disabling "Start on Boot" option. Latter one controls whether Sensei should be run during boot time. If you disable it, it does not stop the engine, you'll need again to click on Stop. Most probably you clicked on "Stop", but just wanted to be 100% sure.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: krdhtet on September 07, 2018, 10:35:12 am
Dear mb,

I'm well received your link, thanks.

Currently, Sensei won't find out wifi interface.

Best regards,
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sol on September 07, 2018, 05:49:57 pm
@mb Thank you for your support.
The system only uses that much cpu power when I'm fully saturating my internet connection (150mbit).
Apart from using sensei I haven't experienced any issues. But this explains the drop in my throughput for sure.

I tried stopping it by using the stop button first. Which didnt work. I was able to stop the elastic search engine using the stop button though. Then I disabled start on boot and rebooted the machine. Unfortunately this didnt disable sensei after the reboot and somehow I was able to stop it and uninstall it after a few tries.
After that I tried the install sensei on the same machine again, which resulted in an crash after the final installation. The PC wasnt accessible via gui or shell anymore and I had to reinstall opnsense.

So it seams that a machine with underpowered resources might not be able to be stoped using sensei 0.6 right now.

Cheers
Title: Re: Sensei on OPNsense - Application based filtering
Post by: nospam on September 07, 2018, 10:47:10 pm
Vapourware? Blackbox man-in-the-middle SSL password harvester?

No download links, no source code, no forums
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 07, 2018, 10:57:45 pm
Hi @krdhtet,

This is done on purpose. We have an unresolved issue with the wireless adapters, so we filter them out while scanning existing interfaces.

For now, the workaround would be utilizing an external AP which would be connected to one of your ethernet ports.

I'll post an update when we're done with it.

Thank you for pointing this out. Also added to the product FAQ.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 07, 2018, 11:58:12 pm
Hi @sol,

Thank you very much for further information. Yes, under heavy CPU utilization, it looks like we've been able to re-produce the issue. I'll update the thread about the resolution.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 08, 2018, 07:14:16 am
Hi,

Thank you for the straightforward feedback.

Vaporware?

No. Sensei is developed by Sunny Valley Networks. I'm Murat, founder of the company. Sunny Valley is a venture-backed, Delaware/US registered company, located in Sunnyvale,  California. Company website is https://sunnyvalley.io. I live in Bay Area. If you are around or will be one day, I'd very much like to meet you in person, grab a coffee and have a chance to get to know each other closer.

No download links?

Currently, we provide the download link for people who register for the BETA early access program. When we are done with the early issues reported by BETA users,  we'll release the final community edition, which will be downloadable directly from the website.

No forum?

We're quite new. We've released the BETA version in late July. We thought that it would be most efficient if we used the existing OPNsense forum for that purpose. Because the plugin is available for OPNsense, and this forum is where all the people discuss things around OPNsense.

No source code?

Sensei is closed source. We announce it on the product webpage. On the other hand, apart from Sensei community edition being available for free for the community, we have a list of open source contribution items, which we think will be of value to the whole project and the community.


Password harvester?

No. Sensei follows best practices implemented by Bro/Suricata; explicitly strips out and throws away octets that could be sensitive. For instance, it does not touch HTTP bodies,  and spends extra cpu cycles to strip out any parameter passed to GET/POST requests and cookies.

It is about our effort to tackle the increasing utilization of encryption by the recent cyber attacks to avoid detection:

https://www.wired.com/story/phishing-schemes-use-encrypted-sites-to-seem-legit/
https://www.thesslstore.com/blog/lets-encrypt-phishing/

However we also share your concern. We also agree that TLS code should be distributed in a more controlled way. This is why TLS will be part of the Enterprise edition.

Thank you for taking the time and comment.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 12, 2018, 05:51:59 pm
Hi @sol,

It looks like we've fixed the problem which in some cases leads to Sensei not stopping appropriately.

Fix will appear in 0.6.0-release, which will be released today US Pacific time.

Would be more than happy if you can give it a try.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: Nekromantik on September 13, 2018, 12:17:52 am
im interested in trying this out
I only have a 80/20 connection and am using a Celeron dual core mini pc with 4GB RAM.
Will this be too much for my hardware?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 13, 2018, 02:12:38 am
Hi @Nekromantik,

Thank you very much for your interest in Sensei.

Yes, unfortunately this hardware configuration will be insufficient for running the software. Sensei installer will refuse to start. You'll need at least 8GB RAM and a more modern CPU.

Please see this blog post to get more information:

https://www.sunnyvalley.io/blog/sensei-hw-requirements
Title: Re: Sensei on OPNsense - Application based filtering
Post by: samsonmcnulty on September 13, 2018, 02:50:10 pm
I just replied to your email with the download link to v .6 and didnt realize that the hardware requirements had changed.
Code: [Select]
This is Awesome! But I have one small request. I use a system with 12 GB ram now for my opnsense install. Previously, I was using 16 GB since sensei requires it but I never noticed my ram usage go over 8 GB. My environment is only about 4 users with maybe 20 total devices connected at once but rarely being used all at the same time (think SOHO network). Is there any way to add an option for a smaller network like mine or is there some way I can bypass the 16GB minimum requirement?
Am I totally tripping here? have they always been 8GB minimum? I could have sworn when I tried to install the last version it stopped me since I only had 12 GB... I'm probably crazy lol
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 13, 2018, 06:21:44 pm
Hi @samsonmcnulty,

Great to hear that it worked at your second try :) Yes, the check in the installer was for 8GB minimum RAM. I guess it was something else which went wrong.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: Alphakilo on September 15, 2018, 04:47:59 pm
Is it required to run the Elastic stack on the Firewall?
Why not split it into two packages: The "Firewall" part and then Elasticsearch, Logstash, etc...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 15, 2018, 07:24:13 pm
Hi @Alphakilo,

Many thanks for the input.

Currently it runs on the firewall. This was an important decision to make when we first started working on the plugin. All of the first users' feedback was to have it coupled with the firewall. Because the deployments were typical of a SOHO, SME, and they were not able to operate a separate deployment just for reporting.

So instead of starting with a distributed design, we started with this one, suggesting early users to increase the amount of memory they had. They were already using modern CPUs, so CPU was not a problem.

For a reference, with the current architecture, the largest deployment that we are reported is  700+ concurrent users and 500 Mbps/50 Mbps max, 300 Mbps sustained WAN throughput. HW: Dual-Core i5-2400 @3.10 GHz (4 threads) with 10GB RAM - OPNsense + Sensei. No IPS, No AV, No Caching. Use case is firewalling + application control + web security.

Looking forward, it looks like we'll offer this option. Since we see that more and more people want to see Sensei deployed in more large scale environments, with thousands of users.

For the time being, our focus is to have the software make super stable & make it cover the essential network security requirements of SOHO / SME users.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 16, 2018, 04:13:32 pm
Hi there,

Sensei 0.6.1 is released. This is a minor reliability release fixing a few issued reported for 0.6 release.


More on how to update to 0.6.1: https://www.sunnyvalley.io/blog/sensei-0-6-1



Title: Re: Sensei on OPNsense - Application based filtering
Post by: bulmaro on September 26, 2018, 12:00:18 am
Hi friends, thanks for the very interesting project work,
I'm testing version 0.6.1, my interface is vlan but I do not see Packets IN and Packets OUT, any settings I missed?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on September 26, 2018, 07:16:52 am
Hi @bulmaro,

@svn is working on your bug report. Hope to update you about this soon.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: bulmaro on September 26, 2018, 03:45:52 pm
thanks for your attention
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hyralak on September 29, 2018, 07:25:46 pm
I tested Sensei for a couple weeks. In that time I observed some unexpected behavior. First i need to say that I have had zero issues with opnsense in the year that i have been running it, rock solid. I am running it at home, my internet speed is 300/80. The hardware is a Dell Optiplex 8gb ram Intel(R) Core(TM) i5-3475S CPU @ 2.90GHz. Memory usage never exceeded 35% with sensei running and cpu usage was minimal. 
Issues I encountered after installing Sensei included web interface locking up, and unable to access opnsesne via ssh. I could still interact with the console. After this occurred i had to uninstall the plugin.
 
Also, I run a pi-hole for DNS poisoning which logged Sensei as the top domain. I was seeing 25,000-35,000 connection attempts to updates.sunnyvalley.io. I turned off auto updates but it continued to hammer away at updates.sunnyvalley.io. The screenshot below is from the last 24 hours. I uninstalled Sensei about 13 hours ago.

(https://i.imgur.com/nYv8rJw.jpg)

I liked the visibility and functionality that Sensei offered, but the instability was not acceptable. Perhaps my hardware is not adequate for the plugin?
Hopefully the information that the plugin sent back to sunnyvalley will provide insight into my web/ssh issues.
Keep up the great work and thanks for letting me try out the plugin. Perhaps I will try again at a later date.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 01, 2018, 08:29:22 pm
Hi @hyralak,

Many thanks for taking the time and reporting your issue. If you find value in Sensei, than it's our job to make it super stable.

Your Hardware configuration is just fine. CPU/memory utilization seems to be low & as expected.

Do you remember which Sensei version you installed first? Because the symptoms you're seeing, we had an issue which might be causing them, and was fixed at 0.6.1 release. I'm suspecting an upgrade issue.

Updates.sunnyvalley.io is being used by two purposes:

1. If you enabled Automated health-checks, it collects these info and sends them to the updates server, which we run a monitoring service with alerting capability (It's actually nagios). This way we instantly know that some Sensei instance has a problem, and try to diagnose it. Information that's sent:
    a) Check whether the packet engine is currently running
    b) Check whether the packet engine crashed and created any core files
    c) Check whether the Sensei engine has any issues with packet forwarding
    d) Check whether Elastic Search is running & healthy
    e) Check whether Sensei is utilizing any SWAP memory
    f) Check disk free space has at least 20% free.
    g) Check if Sensei is using excessive cpu/memory
    h) Check if Elastic Search is using excessive cpu/memory
    i) Check if overall load average is within safe limits
    j) Check if overall cpu/memory consumption is within safe limits
    k) Check if Sensei is put onto bypass mode because of a problem.

System health checks are done once a minute. Instead of collecting the information and sending in batch mode, health script connects to the server for every one of the checks. So this makes 11 connections for a minute. This is why you see some many connections. Yep, this is inefficient & we have an open JIRA issue to address this.

2. Software update checks. If you enable update checks, they are done once an hour.

Though the number seems to be double the number we should be seeing. Our guess is that there is a runaway cron job from previous versions.

I'd love to explore more, I'll be writing to you via a private message. I'd like to find the root cause relating to this. Than fix is the easy part :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hyralak on October 01, 2018, 08:39:12 pm
It appears that I installed sensei_installer_opnsense_0.6.1-release.sh
Title: Re: Sensei on OPNsense - Application based filtering
Post by: rhyse on October 02, 2018, 10:55:41 am
Hi

I am seeing an issue where the "Sensei Packet Engine" keeps stopping, clicking start makes it come back to life.

Enviro: VMware 6.7, 10GB RAM, 2 x vCPU's (host CPU 2 x  E5-2670) , disk space 2.2 gb used out of 18Gb , Sensie deployment size Small (I have just enabled " Enable Generation of Support Data:" ), Sensei version 0.6.1-release (installed from this version)

This is a test infra, so doesn't have much traffic going through it

Any ideas ?

Thanks
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 02, 2018, 04:00:12 pm
Hi @rhyse,

We did not have much users on VMware. Let's debug it together & make Sensei run there. I'm contacting you.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 02, 2018, 08:58:01 pm
Dear Sensei users,

@rhyse helping us debug his issue, we've spotted a bug with Netflow output formatter. If you're using Sensei with Netflow, better to disable it for now.

For the resolution, we'll issue a fix. Hopefully as 0.6.2.

Many thanks @rhyse !.




Title: Re: Sensei on OPNsense - Application based filtering
Post by: Csykes27 on October 16, 2018, 12:29:16 am
I am having an issue of when I Enable Cloud Reputation & Web Categorization all web traffic stops. all services are running and stay running from what I can tell.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: svn on October 16, 2018, 12:49:50 am
@Csykes27 thanks for reporting. We've heard this issue for the first time actually. Let's debug what is causing this together.

I shall be contacting you soon to resolve the issue.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: jjanzz on October 18, 2018, 10:48:08 am
During the initial installation, a dependency throws a 404 error:

Code: [Select]
pkg: https://updates.sunnyvalley.io/repo/libXtst-1.2.3.txz: Not Found
FAILED : Unable to install required packages. Please see install.log
Title: Re: Sensei on OPNsense - Application based filtering
Post by: svn on October 18, 2018, 07:11:19 pm
Hi @jjanzz,

We're currently working with OPNsense for the integration. As a part of this process, OPNsense started providing some of our dependencies. We'll update our installer and package to be compatible with the latest changes on OPNsense package repository, soon.

It seems that some of the dependencies not satisfied (namely, some configuration files of elasticsearch, and some java dependencies). We'll fix this urgently.

Right now, you can register for download and we'll send you a download link as soon as we fix the dependency issue.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: jjanzz on October 22, 2018, 04:10:32 pm
We're currently working with OPNsense for the integration. As a part of this process, OPNsense started providing some of our dependencies. We'll update our installer and package to be compatible with the latest changes on OPNsense package repository, soon.

Thank you for the reply. No problem; I'll gladly help you test it out as soon as it's possible :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 25, 2018, 12:09:10 am
@jjanz and community,

Elasticsearch5 was added to OPNsense packages as part of the 18.7.5 update. There was a problem in the FreeBSD elasticsearch package builds which was inherited by the OPNsense build system.

Because elasticsearch was problematic, Sensei installations were failing.

Today we fixed the problem. In the meantime, OPNsense will be removing the package from its repository in the upcoming release.

Starting 18.7.6, elasticsearch will be provided by Sunny Valley Package repository.

Long story short: We're resuming Sensei downloads. You can now download and install new Sensei version, which is 0.7.0-beta1 as of now.



Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 25, 2018, 12:10:55 am
Hello all,

As part of 0.7 release effort, we've launched Sensei Users' Manual & Documentation.

Please find it here:

https://guide.sunnyvalley.io/sensei/
Title: Re: Sensei on OPNsense - Application based filtering
Post by: wordsmith on October 25, 2018, 07:45:42 am
This plugin looks pretty interesting and I’d like to give you some non-technical feedback to consider. But first a question: will Sensei ever be open source?
See, the reason I ask is because to me it seems there is some confusing communication going on. I’m sure, some of it is non-intentional like:

Quote
For now I'm happy to tell that community edition for OPNsense will always be there and forever free.

"For now" and "always" don’t work well together. Basically, now you’re saying that this will always be the case, but later you might change your mind to “it isn’t free anymore”. I suspect that this was unintentional, but I just wanted to get it out of the way.

What rubs me a bit the wrong way is that the community edition is free, but not open source. According to your FAQ:
Quote
The Packet Engine coded in C++, and its source code is not open.

I think the reason there are community editions in the software space is precisely to indicate that a company/developer wants to build a trust model with others and, as a result, gives them the recipe so that they can build a community around it together. In short, it isn’t about getting something for free i.e. without having to pay, but to build trust.

Now, where your approach to marketing proofs to be rather problematic is with statements like this:

Quote
Empower your open source firewall with Next Generation features.

If you plan to keep parts of Sensei closed source, I’d suggest you’d drop the “open source” in your marketing, because it’s confusing at best, misleading at worst. Next, as long Sensei isn’t open source, I’d also reconsider the use of “community edition”: this is a rather well known way to describe the non-commercial version of a product that isn’t just for the community, but also by the community. If the community doesn’t have access to the code, it’s not a community edition, it’s a free edition.

The FLOSS community already suffers from a huge labeling problem (ever tried to explain to a non-technical user the difference between Free Software and Freeware?) so let’s not muddy the waters even more.

I don’t know about your business model, but for people who really care about open source it’s not about getting stuff for free, it’s to be able to verify the claims of company such as yours and, of course, to build a community around a solution that can be build by like-minded people without restrictions regarding code access.

Of course, at the end of the day there’s always the pragmatic side to consider and there will probably be a lot of users who are perfectly fine to run proprietary software on their open source OS, but for people like me who decided to use an open source solution not because it is free of charge, but precisely because it’s source code is available, Sensei won’t be the solution we’re looking for.

Now, with all that being said, I still appreciate your efforts.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on October 26, 2018, 08:33:09 pm
Hi @wordsmith,

Many thanks for taking the time and provide this valuable feedback. Now we become aware of a communication problem.

To clarify things:


As you’ve correctly pointed out, if there is any misunderstanding, it’s unintentional. Your comments shed a lot of light as to what needs to be adjusted in the messaging. We’ll be working on that.

Taking this chance, I’d like to give a little bit of background why we started with “open source firewalls”.

As Sensei team, we believe that we’ve created a powerful packet processing technology. We believe that better packet visibility means better decision making. Better decision making means better success rates in detecting malign traffic.

Sensei is the first of two products that we’re going to create for a large market.

We hope to make Sensei available for any network security equipment / product which needs application classification & web security features. L3-L4 firewalls, UTMs all fall into this category.

The thing we started with open source firewall space is that, it was a request by an MSP who was deploying open source firewalls onto customers and providing support services. Very happy with their current firewalls, they needed several features that we could provide. We quickly did an integration and voila! The resulting solution (OPNsense + Sensei) was found to be better than many of the current players in the UTM market.

This sparked a light for us. Why not deliver the product as a plugin instead of yet another full-blown firewall appliance? It’d be cost effective for us and we would than be able to relay this cost advantage for the benefit of our prospective users.

In this regard, open source firewalls is a delivery channel for us, though it’s not the complete target market. Via this initial channel, we learn very much from our users and improve Sensei. You can’t believe how much Sensei improved from the day we announced first beta up until this day. Then of course, we are looking for market visibility. It’s great to see people loving the solution and spreading the word.

A free of charge Sensei edition (maybe we should call this freemium edition) is a way of our giving back to the OPNsense community.

Having founded a local open source community (enderunix.org) and published some open source tools, I truly understand, appreciate and respect your stance.

Though we cannot make Sensei fully open source, I think the best we can do right now is to communicate what Sensei is and what it is not in a straight and open way. This way people would know what they will have and what they won’t; and will make an informed decision about using / not using it.

It's somewhat hard to figure out a way to communicate people that the current product is for “open source firewalls” without using the words “open source”. Because marketing wise, we would like to be as precise as possible so that people would know what it is for.

However I also see that it’s creating confusion. We’ll spend more time on this. I’d also like to consult you if you wouldn’t mind.

Again, many thanks for bringing this up to our attention.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 08, 2018, 02:24:10 pm
Dear Sensei users,

0.7.0-beta1 update is out for those who are on 0.6.x releases:

https://www.sunnyvalley.io/blog/0-7-beta1-update-available-for-0-6-x-users

0.7 Beta1 comes with the following functionality:
 
1. New Report - Blocked Connections Sessions Explorer and drill-down reporting
2. Reports enhancement: Daily executive reports. Selected reports delivered via a daily
    e-mail.
3. Customizable Landing Page for Blocked connections
4. Reports data retiring: disk space consumed by Elastic Search (Reports) is now
    configurable
5. Release Changelog is now displayed during Sensei updates
6. Shortcut to add Block/Allow rules based on fields (IP Address, Application, App
    Category etc.) via Session Explorer Reports. 
7. 350+ new applications identified.
8. Documentation: Sensei Users' Manual
9. Sensei speaks your language now, we added i18n support to match your OPNsense
    UI language. English & German are the two for now, more coming soon.
10. More performance & stability improvements


If you've downloaded & installed Sensei later than October 15, you should already be using 0.7.0 beta1. This is an update package for older versions.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: shrdlu on November 13, 2018, 06:15:37 pm
Not sure if this is the right place to post this, so if I am wrong please redirect me.

I have noticed with Sensei (BTW, it is working fine) that when I run a health audit in OPNsense I get the following (see attached screenshot) checksum mismatch for the nodes.csv file and was curious if this is normal or something is wrong.  Things appear to work fine and no matter what Cloud Threat Intel selections I make (not sure that is related but it might be) I get the mismatch and the Cloud Threat Intel is working fine regardless, or at least shows up and running.

And, on another note, in terms of processing when do the Sensei components process information in terms of order?  For example, I use the web proxy (squid) in OPNsense and was curious if Sensei process the packets before the proxy or after or somehow during, or frankly something completely different if I am misunderstanding the order of operations.

Thanks in advance.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 13, 2018, 06:27:42 pm
Hi @shrdlu,

You're in the correct place :) We're receiving feedback & comments and help requests here. You can also shoot a ticket if you think you've found an issue with the software:

https://gitlab.com/svn-community/opnsense-sensei-plugin/issues

The thing with Node.csv is not an issue. Web UI updates the contents of this file with the best servers available. I guess this creates a mismatch with the OPNsense File Integrity Checker. We'll handle that.

With regard to processing order: Sensei receives packets while they traverse from Network Adapter to the FreeBSD networking stack; which means it receives them before Squid and even before L3/L4 Filtering.

You're all welcome, and thanks for sharing your experience.

 
Title: Re: Sensei on OPNsense - Application based filtering
Post by: theq86 on November 15, 2018, 03:56:00 pm
I Installed sensei. When I was on the dashboard to configure the protected interfaces only my 2 vpn interfaces show up. Not WAN, not LAN, nor any other interface on my firewall.

current version as of writing (0.7.0-beta1)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mimugmail on November 15, 2018, 04:11:55 pm
Do you have IPS enabled on LAN or WAN?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: theq86 on November 15, 2018, 06:23:50 pm
Nope. Neither
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 15, 2018, 06:33:28 pm
Hi @nasq,

Any chances your LAN interface is virtio?

https://guide.sunnyvalley.io/sensei/support/faq#no-ethernet-interface-is-being-shown-in-the-interface-configuration

As quick workaround, select Intel E1000 as the adapter type.

As the final solution we're sponsoring a development which will ship the latest upstream netmap code into FreeBSD.

This will also fix lots of issues that you might be encountering with Suricata as well.

https://svnweb.freebsd.org/base?view=revision&revision=340436

It's us. Commit is done to HEAD, will be MFC'd to 11-STABLE in the following timeframe.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mimugmail on November 15, 2018, 06:36:53 pm
https://svnweb.freebsd.org/base?view=revision&revision=340436

It's us. Commit is done to HEAD, will be MFC'd to 11-STABLE in the following timeframe.

Really nice contribution Murat, thanks! :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 15, 2018, 06:44:59 pm
Hi @mimugmail,

Our pleasure. All welcome :) Super excited to see the changes land in 19.1.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: franco on November 16, 2018, 08:40:19 am
r340436 is indeed very nice. mb, please push these into my mailbox or open a src.git ticket for swift inclusion. we need the MFC for stable/11 to be committed first though.

for the csv, it's considered bad style to manipulate files shipped with the package. for that reason FreeBSD has the "sample" trick which creates a copy of the file and only checks in the unmodified file (suffix ".sample"). We use it in core in some places, too. Plugins don't support it yet, but they should eventually.


Cheers,
Franco
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 16, 2018, 05:24:23 pm
Hi @franco, thank you very much. I hope this will be of some help to the project.

We're still testing the code in HEAD. After we're confident, it's going to be MFC'd to 11-STABLE. I'll be pinging you once we're done with that. 

I've been informed that we actually have the unmodified file (.default) with the package. Engine reads a "processed" version of that file, which -indeed- do not need to be included with the package. We're removing it. I guess we're done then.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: franco on November 16, 2018, 06:33:35 pm
Yeah, that's all sorted then, great!  8)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: therec on November 18, 2018, 05:13:56 pm
Hello Murat,
I had a question around blocking. (i.e. adds, trackers, etc.). is there a way to allow a specific site? if i go to neweggs web site, the site is unreadable. if i disable the blocking, its ok again. i see the option to the right to unblock, but it wants to unblock the group (ad category) and not the site. forgive me if I've missed something simple. and thanks for the work, this is a wonderful product, I cant wait to see where you take it.

Thanks
Robert

If i posted this in the wrong place, let me know and ill move it

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 19, 2018, 02:51:31 pm
Hi Robert, @therec

Thank you very much for your feedback. Awesome to see you've found the plugin useful.

When you browse Reports -> Security->Session Explorer, see if the site is being blocked via Application filtering or Web filtering. You can differentiate it by looking at the "Block category" information. If by Application filter, it says "Application category", if via Web filtering, it reads "Web category".

To allow a specific "Application", just go to Application Control, find and expand the related category, find your specific application, and unblock it.

If the filtering is done via Web filtering, browse to Web Controls->User defined categories. Create a new category i.e. Whitelist, and put your websites which you want whitelisted here.

Click "Save Changes" and that should be it.

Thanks,
Murat
Title: Re: Sensei on OPNsense - Application based filtering
Post by: therec on November 20, 2018, 01:45:51 pm
Thanks, that makes a lot of sense. however it doesn't seem to be working. I've added

- https://www.newegg.com/
- secure.newegg.com/
- www.newegg.com/
- www.neweggbusiness.com/
- https://newegg.com

Maybe ive misses something?

as an alternate test i confirmed http://static.hotjar.com/ was blocked (webtracking site).
I added this to the web controls as requested (user defined group) and is had the green check (allow),
This site also remains blocked after whitelisting via web address.

I suspect im missing something, I have amatuer firewall skills at best. but i love this product and hope its a long term solution for me. thank you for the help

P.S. i just noticed https://flash.newegg.com works just fine.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 20, 2018, 09:42:23 pm
Hi @therec,

Let's dig a little deeper together. I'll be writing to you privately. I might need some logs. Let's see if there's something wrong or there is a configuration problem.

 
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on November 21, 2018, 08:04:50 am
Hi, Using Sensei plugin and its great. Need help in few thing:
1. Is there any live view to know which IPs are using the most bandwidth and then drill down to specific IP to check what sites / services it is using and which site / service is consuming the most. (I use ntopng and it has very nice view to tell which devices are consuming most bandwidth only)
2. I do not know why but when i check the "Table of Local / Remote hosts, it shows bytes in / out which is very low as compared to ntopng. and i have avg 25-30 devices running all the time out of 50 devices but it only show few so how can i list all of them.
3. Is there any way to get all the web history of a user or users ?
4. Is there any way to bind names to IP (local / LAN IP) as squid does in web proxy ?
5. It filter web traffic and works as transparent web proxy so is there any way to use it as cache server as well or if you are planning for it in future.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 21, 2018, 11:24:45 pm
Hi @manjeet,

Thank you for sharing your experience with Sensei. We very much appreciate that. Find the answer below:

Quote
Is there any live view to know which IPs are using the most bandwidth and then drill down to specific IP to check what sites - services it is using and which site - service is consuming the most

Yep. Navigate to Sensei -> Reports -> Connections. Look for the Chart named Top Local / Remote Hosts. But make sure to select the reporting criteria as "Volume" from the upper right hand corner of the reports page. Default is by sessions. You can do "Session based", "Packet based" or "Volume based" reporting.

When you left click on any IP, a submenu appears. Click "Drill-down" and all reports will be automagically filtered according to this IP address you've selected.


Quote
I do not know why but when i check the "Table of Local / Remote hosts, it shows bytes in / out which is very low as compared to ntopng. and i have avg 25-30 devices running all the time out of 50 devices but it only show few so how can i list all of them.

My guess is that you might be viewing the "Session" reports. Make sure you've selected "Volume" as the reporting criteria.

All devices currently active should be listed though. My guess would be that you might be viewing reports for the last 15 minutes. Make sure you've selected a longer time frame from the right hand corner.

 
Quote
Is there any way to get all the web history of a user or users ?

Yes. You can do that from the Web / TLS reports. You have the drill-down capability for every report type.


Quote
Is there any way to bind names to IP (local / LAN IP) as squid does in web proxy ?

Actually, packet engine automatically maps DNS names to IP addresses if it can find a matching DNS transaction. Soon there will be Active Directory / LDAP integration which you'll be able to see the user / group names.

Quote
It filter web traffic and works as transparent web proxy so is there any way to use it as cache server as well or if you are planning for it in future.

In theory, packet engine is capable of doing that. But we chose to focus on complementing features that are currently not existing. Squid is a great caching proxy. Indeed caching is its original reason of existence. That being said, Sensei roadmap does not have "caching" as a feature.

Many thanks for reporting your experience with us.



Title: Re: Sensei on OPNsense - Application based filtering
Post by: maekar on November 22, 2018, 02:09:46 pm
Hi,

The maximum of 1000 concurrent users is an approximation for better hardware performance or an strict software limit?

thanks

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 23, 2018, 02:02:50 am
Hi @maekar ,

This is the current field-tested maximum. Software arranges several tunables (e.g. cache sizes, connection table sizes etc.) according to the user size.

Current focus is to make the software super stable for SME use cases (which generally means user populations below 1000)

Looking forward, engine is able to scale to hardware resources, which makes it possible to secure thousands of users.

Hope this answers your question.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: johjoh on November 23, 2018, 11:57:10 am
Good morning, will Sensei one day consume less resources in terms of RAM and CPU?
For example an Atom CPU or a Celeron with 4GB or 8GB of ram?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 23, 2018, 02:47:31 pm
Hi @johjoh,

Yes :)

A big portion of the resource requirement come from the Reporting engine (Elasticsearch). The core packet engine has been tested to run on low resource systems: e.g. Celeron  < 1GB RAM.

A roadmap feature - remote reporting - allows to run packet engine on the firewall itself, and reporting on another more powerful server.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: bobbythomas on November 24, 2018, 07:19:02 am
Hi Murat,

Couple of questions? Is there anyway to find the current installation or patch status? Where are the Sensei logs installation logs stored and how can we view that? I received an rc1 update and it's about 36MB, but it's been more than an hour since I started the installation, I would like to know the status. While installing Sensei some packages took a lot of time to get downloaded and I suspect something like that. I believe there is some latency reaching some of the repositories. Could you help me troubleshoot this issue?

Thank you,
Regards,
Bobby Thomas
Title: Re: Sensei on OPNsense - Application based filtering
Post by: shrdlu on November 24, 2018, 07:35:55 am
Not sure if this is just my setup but after upgrading to OPNsense 18.7.8 I get stuck in a loop that won't complete.  Because it reset my configuration of Sensei* after the OPNsense 18.7.8 upgrade, I have to go through the config wizard again and when I click finish, it attempts to configure everything but kicks out the attached error.  Essentially it tells me, "error indices could not be created," and I am stuck in that loop as it returns me to the beginning of the config wizard.

So, #1, is it just me?
and #2, assuming it is not me and before I simply try to uninstall/reinstall, any ideas?

Thanks
 
*Is it normal for an OPNsense upgrade to reset my Sensei configuration?  If the answer is yes, that is fine but also if there is a way to backup a config and restore it that would help me retain settings.  Either way, love the solution and reconfiguration is actually a minor thing in the grand scheme of things so if the answer is no here then that is fine as I still find huge value in the software.

Thanks
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 24, 2018, 09:02:07 am
Hi @bobbythomas,

/tmp/sensei_update.progress should have more detail regarding the update process. 36MB download shouldn't take that long.

We rolled back rc1 update in case there is something we miss with the update process.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 24, 2018, 09:12:40 am
Hi @shrdlu,

It was unfortunate that both OPNsense & Sensei got updated at the same time. Looks like while OPNsense was upgrading, we shipped 0.7.0-rc1. OPNsense update manager also updated Sensei, a case which we did not handle.

Sorry for the inconvenience. We rolled back 0.7.0-rc1.

A final fix will be out shortly.

For a workaround, I'll be contacting you. We'll try to recover the old configuration.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 24, 2018, 11:26:30 pm
Dear Sensei users,

0.7.0-rc1 upgrade is back.

A quick update on 0.7.0-rc1 upgrade:

If you encountered any Sensei issues while upgrading your OPNsense to 18.7.8, this was due an unhandled case in our package updater when the upgrade process is triggered from the OPNsense firmware updater, not from the Sensei Status Page. This is fixed now in the upcoming 0.7.0-rc1.

But the fix will be in effect starting from 0.7.0-rc1.

So, If you’re on 0.7.0-beta1, and do NOT want to upgrade to 0.7.0-rc1 immediately we strongly recommend running the following command to avoid any issues with the OPNsense system updater.

pkg upgrade os-sensei-updater && pkg lock os-sensei

The command will upgrade your Sensei updater to the latest version and also put a lock on os-sensei package so that OPNsense package update utility will not try to update Sensei.

If you also want to upgrade  to 0.7.0-rc1: Navigate to Sensei -> Status -> Check Updates, and you’ll be guided to upgrade to 0.7.0-rc1.

pS: 0.7.0-rc1 introduces several minor bug-fixes both on the updater and the UI. If we do not hear any issues, we’ll hopefully release 0.7.0 in the coming week.

ppS: Thanks to increasing number of Sensei beta users, it looks like we need to increase bandwidth for Sensei Updates server (updates.sunnyvalley.io). Cool indeed  ;) This will be done in the following weeks. In the meantime, if you encounter slowdowns while installing / upgrading, we’d very much appreciate your understanding.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: samsonmcnulty on November 25, 2018, 08:54:10 am
Sounds fantastic! Good to see the adoption rate increasing at a healthy rate. I did encounter this error but it seems you are already aware of the issue:


***ERROR: Indices could not be created! Reporting may not work***



Is there a temp workaround? I assume uninstalling the package and reinstalling would work?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 25, 2018, 05:59:44 pm
Hi @samsonmcnulty

Yep, that would work.

Can you run the following commands. Basically it'll uninstall & install sensei

service eastpect onestop
service elasticsearch onestop
pkg delete elasticsearch5
pkg delete os-sensei
rm -rf /var/db/elasticsearch/nodes/*


You can also do that by selecting "Uninstall elasticsearch & Remove elasticsearch data" options while uninstalling from Web UI.

then to re-install it:

pkg install os-sensei

Sorry for the inconvenience.

One question: did that happen after you've done an OPNsense 18.7.8 upgrade? We're aware of this problem & hopefully fixed.

I wonder if there are other cases.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: dragon2611 on November 25, 2018, 10:05:50 pm
I'd like to try sensei but I suspect i'd run into problems with lack of RAM and also I have an opensense HA pair with one physical and one virtual (KVM) so I think i'd run into the KVM/VIRTIO issue.

I'm wondering if I'd be better off starting another virtual firewall and stuffing it in the traffic path for the machines i'd want to put behind sensei.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 26, 2018, 02:38:53 pm
Hi @dragon2611,

Good idea :) Let us know if you encounter any issues. On the virtual FW, you can use E1000 as network the adapter type:

https://guide.sunnyvalley.io/sensei/support/faq#can-i-run-sensei-on-a-virtualized-environment-like-proxmox-virtualbox-kvm

Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on November 27, 2018, 07:42:10 pm
Hi, Sunnyvalley.

The first hit and miss: try to block youtube used via google chrome...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 28, 2018, 05:56:32 am
Hi @Antaris,

Thanks for reporting this.

It's because of QUIC: Google's new protocol suite, a replacement for TCP + TLS + HTTP/2. Chrome defaults to QUIC when you browse Google services. Other browsers use TCP so Sensei is be able to identify & block.

Sensei is able to identify QUIC, though its detailed protocol parser is under development. When we're done with it, it'll be able to identify protocols which are transported through QUIC. We hope to have it with 0.8.0 release.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 29, 2018, 04:03:49 pm
Dear Sensei users,

An update to: https://forum.opnsense.org/index.php?topic=9521.msg47488#msg47488 :

If you got stuck in Sensei Configuration Wizard,  here is a quick fix for you:

open /usr/local/sensei/scripts/installers/opnsense/18.1/sensei-init.sh file with an editor, and locate this part. It should be line 64.

if [ "$INDICES_COUNT" -lt 6 ]; then

Update this line to read like:

if [ "$INDICES_COUNT" -lt 5 ]; then

Save the file and re-run the configuration wizard.

0.7.0-rc2 will come with a more intelligent provisioning script which will try to diagnose any inconsistencies with the backend database and try to fix them automatically.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on November 30, 2018, 02:22:27 am
Dear Sensei users,

An update to: https://forum.opnsense.org/index.php?topic=9521.msg47653#msg47653

We've decided to merge some of the code from the QUIC branch to 0.7.0.

Looks like we have a working app identification & filtering for applications running through QUIC :) e.g. Google Services + Chrome browser.

Update will be introduced with 0.7.0-rc2.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 01, 2018, 11:16:31 am
Dear Sensei users,

An update to: https://forum.opnsense.org/index.php?topic=9521.msg47653#msg47653

We've decided to merge some of the code from the QUIC branch to 0.7.0.

Looks like we have a working app identification & filtering for applications running through QUIC :) e.g. Google Services + Chrome browser.

Update will be introduced with 0.7.0-rc2.

Thanks guys, looking forward to it. Can we hope for an optimisation to reduce hardware requirements, especially about RAM?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 01, 2018, 10:36:03 pm
Hi @Antaris,

Many thanks for bringing this into our attention. Looks like with 0.7.0-rc2, Sensei is one of the first in the industry to offer granular control for QUIC based applications.

Currently, big vendors are advising to completely block QUIC protocol, thus forcing browsers to fall back to TCP+TLS. This is slower.

As for memory requirements, actually yes. We're planning a limited reporting option, which will require way less memory than we require today. This will still provide reporting but most probably will lack some advanced features like Drill-down and per-connection details. Other than reporting, all features will be there.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: opnsenseuser on December 02, 2018, 08:13:37 am
when will sensei appear in opnsense as a plugin?
which hardware requirement is necessary?
Is the latest generation Atom processor (c3558) with 8 gb of ram sufficient for sensei?

thx
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 02, 2018, 03:15:16 pm
There is an update Engine: 0.7.0-rc2, but when trying to update it, the system returns:  "No update is available
There are no updates available for you. You are using the latest version. " and stays on 0.7.0-rc1
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 03, 2018, 05:04:26 am
Hi @noname12123,

when will sensei appear in opnsense as a plugin?
which hardware requirement is necessary?
Is the latest generation Atom processor (c3558) with 8 gb of ram sufficient for sensei?

thx

We have a few small items left for the final OPNsense integration.  Then Sensei will be an OPNsense plugin which can installed from the OPNsense Plugins menu. If anything big does not come up, I guess we'll all be finished with them by the end of this month.

I'd expect that latest generation Atom would be ok. Might be a little bit slow to start Elasticsearch but when it warms up, it should be all fine. Crucial thing is RAM and 8GB is perfectly fine.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 03, 2018, 05:08:21 am
Hi @antaris,

There is a small blog post coming related to that. We'll need to use the command-line updater for the rc2 update. GUI code is missing a "pkg update -f".

Can you try to update via command line?

As the root user, just run:

sensei-updater

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 03, 2018, 07:32:42 pm
Dear Sensei users,

After testing 0.7.0-rc2 update with a few of Sensei users, it looks like 0.7.0-rc2 is ready to go.

We'll need to use the command-line updater for this update. GUI code is missing a "pkg update -f".

Login to the firewall console as the root user; and run:

sensei-updater

It'll take care of the rest, and you'll be updated to 0.7.0-rc2. You'll need to manually start the Sensei engine from Sensei->Status.

0.7.0-rc2 introduces fine grained application identification & filtering for Google Services through Chrome browser (QUIC protocol update); as well as several other reliability fixes for the sensei-updater.

If we do not see any issues reported; 0.7.0 will be finally released Thursday this week :)


Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 03, 2018, 07:33:50 pm
Thanks a lot:

"Sensei has been updated successfully."

Just have to start Sensei Packet Engine manually...

It's runnig as guest on Proxmox btw...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 03, 2018, 07:38:10 pm
Hi @antaris,

Glad that it went well. Thanks for the notice about starting Sensei. I've updated the message accordingly.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 04, 2018, 06:25:33 pm
Do i miss Web 2.0 controls and TLS Visibility menus as seen on advertisement video?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 05, 2018, 03:12:55 am
Hi @Antaris,

Web 2.0 Controls / Cloud Application Controls depend on port agnostic TLS Inspection functionality. TLS Inspection will be made available with Sensei Premium Edition.

Should you like to give an early try, I'll be happy to provide a trial license for you.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 05, 2018, 05:19:01 pm
It's too early i guess, and my Sensei is not ot production enviroment. When it's ready and the prices are known, will give it a try in one of the schools that i support. I can test it in network with up to 1500 devices and 1gbps symmetrical internet.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 06, 2018, 03:31:44 pm
Hi @Antaris,

Sounds great. Will get back to you when we have more progress with that.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 07, 2018, 01:21:42 pm
Hi, I just reinstalled the OPNsense and trying to install the Sensei plugin but script is timing out.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 07, 2018, 03:22:29 pm
Hi @manjeet,

Update server is operational again.

Make sure you're following the latest install instructions:

https://guide.sunnyvalley.io/sensei/getting-started/setup

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 10, 2018, 08:10:17 am
Thanks. It is installed and working.

I still have one same issue as mentioned before.. In "Table of Local Assets" and "Table of Remote Hosts" i only see 10 devices / hosts. I have changed criteria to volume, time interval 24 hours. I have also checked it by increasing settings. I am currently running more then 30 devices. So is there a limit on no of showing hosts or anything ?? I need to provide bandwidth usage (upload / download / total) reporting of all the devices in network daily basis. Any other way to do so ??
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sagem2004 on December 10, 2018, 04:02:06 pm
Good evening,
we can filter the site in safesearch " picture "
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 10, 2018, 06:15:16 pm
Hi @manjeet,

Glad that installation went smooth.

Thanks. It is installed and working.

I still have one same issue as mentioned before.. In "Table of Local Assets" and "Table of Remote Hosts" i only see 10 devices / hosts. I have changed criteria to volume, time interval 24 hours. I have also checked it by increasing settings. I am currently running more then 30 devices. So is there a limit on no of showing hosts or anything ?? I need to provide bandwidth usage (upload / download / total) reporting of all the devices in network daily basis. Any other way to do so ??

Ah, now I see what you're trying to accomplish. All charts on the report tabs will show top 10 records. For table reports, I guess we can put an option to provide the full list.

Stay tuned. We'll pass an update.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 10, 2018, 06:19:32 pm
Good evening,
we can filter the site in safesearch " picture "

Hi @sagem2004,

I don't think I was able to fully understand the question. Can I request that you rephrase it?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sagem2004 on December 10, 2018, 06:41:06 pm
can have blocked pornographic images via safesearch

exemple : https://support.opendns.com/hc/en-us/articles/227986807-How-to-Enforcing-Google-SafeSearch-YouTube-and-Bing

Merci.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Misant on December 10, 2018, 07:27:49 pm
Installed Sensei today on a Qotom. seems to be working fine. Setup is just for a small household with me and my girlfriend, but we are going to expand to a dog and 2 kids. So torture tests will have to wait for some time.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 10, 2018, 09:15:03 pm
Hi @sagem2004,

Thank you for the clarification.

Google Safesearch enforcement is in Sensei's short-term roadmap. Should arrive with Sensei 1.0, which is the first production release for Sensei (ETA Q1 2019).
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 10, 2018, 09:16:33 pm
@Misant, Good to hear that :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: sagem2004 on December 10, 2018, 09:34:59 pm
Hi @sagem2004,

Thank you for the clarification.

Google Safesearch enforcement is in Sensei's short-term roadmap. Should arrive with Sensei 1.0, which is the first production release for Sensei (ETA Q1 2019).

very Good news thank you :) :) :) :) :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 11, 2018, 11:12:54 am
Thanks for it..

Hi @manjeet,

Glad that installation went smooth.

Thanks. It is installed and working.

I still have one same issue as mentioned before.. In "Table of Local Assets" and "Table of Remote Hosts" i only see 10 devices / hosts. I have changed criteria to volume, time interval 24 hours. I have also checked it by increasing settings. I am currently running more then 30 devices. So is there a limit on no of showing hosts or anything ?? I need to provide bandwidth usage (upload / download / total) reporting of all the devices in network daily basis. Any other way to do so ??

Ah, now I see what you're trying to accomplish. All charts on the report tabs will show top 10 records. For table reports, I guess we can put an option to provide the full list.

Stay tuned. We'll pass an update.

I do not how it calculate the top 10 but i think you have an issue here.. I was looking at "Insight" for current network usage and find out that one of the system has consumed 4GB of data since morning. I checked it in Sensei and it showed the same 4GB data usage for that IP.

But when i checked the top 10 list in "dashboard" and in "reports" (No filters, cross-checked) (it showed me that same report), this IP with 4GB usage was not there. Even some other IPs which Insight showed were not also there.

It showed me list of top 10 which i think is better match with the last night usage but not since this morning. Its been 6 hours and i do not see those IP in this list.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 11, 2018, 03:58:09 pm
Hi @manjeet,

I see. Let's dig deeper. Can you reach us through sensei -at- sunnyvalley.io?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on December 11, 2018, 04:17:14 pm
Hello, mb

Is there a way to clear all the logs in Sensei?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 11, 2018, 08:52:23 pm
Hi @cgwork,

Use the following two scripts to delete and re-create all reporting indices:

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py


This will be also a Web UI menu item with the upcoming Sensei update (0.7.0-rc3).
Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on December 12, 2018, 04:35:05 pm
Hi @cgwork,

Use the following two scripts to delete and re-create all reporting indices:

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py


This will be also a Web UI menu item with the upcoming Sensei update (0.7.0-rc3).

Awesome Thank you ... also have you thought of getting the reports to be printed or converted to .pdf format? i also noticed when i get the emails and "click to download and view the detailed reports" are blank see attachment. Did i miss an check in the box so i get them? I'm currently selected only Sessions but it would be nice if i could get all of them or select the once i would like to have.

Thank you again for the hard work.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 13, 2018, 02:37:55 am
Hi @cgwork,

You're all welcome. We had introduced PDF export previously.  It's being re-worked and will be available shortly.

You shouldn't receive an empty html file. Looks like a problem. Can you share which e-mail provider you are using? It's been tested with major ones like Gmail & Outlook. Let's try with yours.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on December 13, 2018, 01:53:12 pm
sure i'm using gmail for this setup
Title: Re: Sensei on OPNsense - Application based filtering
Post by: kagou on December 13, 2018, 02:06:17 pm
Hi. I'v some problems with sensei (look at the picture).
I'v tried first with my system but after some problems i'v rebuilt my interface assignments, removing bridge system.
Now i'v a WAN/DMZ/WIFI/LAN on my 4 ethernet ports.
I'v stoped and used the "You can restore all Sensei packet engine configuration to their original defaults by clicking 'Reset' button."
Set just ma LAN to be supervised, but look at the picture
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 14, 2018, 05:51:11 pm
Hi @kagou,

Looks like a problem with the backend indexes.

Can you try these if they fix the problem?

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py


It it does not, can you share your /var/log/elasticsearch/elasticsearch-2018-12-13.log log file to sensei - at - sunnyvalley.io ?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 14, 2018, 06:45:57 pm
Hi @cgwork,

sure i'm using gmail for this setup

Gmail should be fine. Can you forward the email to sensei - at - sunnyvalley.io ? If you can forward as an attachment, that'd be perfect.

Are you using Gmail through a browser, or through an email client?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 14, 2018, 06:53:21 pm
Update to @manjeet's post: https://forum.opnsense.org/index.php?topic=9521.msg48451#msg48451

Spotted the problem. A typo avoided reporting criteria to be reflected for some reports.

Fix should arrive with 0.7.0 release.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 14, 2018, 07:15:33 pm
Dear Sensei users,

We know you’re looking forward to seeing 0.7.0 release. We also do indeed.

Yet, we decided to ship another release candidate before the actual release because some updates to the code base might have more impact than we originally planned. These code updates are preliminary work related to an effort to minimize external library dependencies and compiling Sensei engine as a Position Independent Executable (PIE).

Minimizing external library dependencies will allow Sensei to be able to run on embedded platforms which run on very low resources.

PIE is a nice feature which will be default for OPNsense@HardenedBSD and will provide mitigation capabilities against exploit attempts to the packet engine. (Note: PIE is not enabled yet)

So there we have 0.7.0-rc3 publicly available for you to test. This is the Changelog from rc2 to rc3:

New features (from 0.7.0-rc2 to 0.7.0-rc3).
* More lightweight core packet engine
* Option to delete all reporting data
* Mobile web browsers compatibility. You’ll be able to view Sensei reports through a mobile device.
* Prevented scheduled jobs from submitting unnecessary emails.
* HW requirements check has been made available for the UI initial configuration wizard.
* Some stability improvements. 

0.7.0-rc3 has been under testing for about a week now, but if you’re running Sensei on a more production like environment, you might want to wait till we ship 0.7.0 final release, which should arrive in a week if we do not see any issues with 0.7.0-rc3.

To update to 0.7.0-rc3, login to OPNsense UI, navigate to Sensei -> Status and click Check for Updates. You should see an update reported. Click Update to proceed with the update. Sensei updater should take care of the rest.

Best
Sensei team
Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on December 18, 2018, 02:12:03 pm
Great News mb,

In my personal opinion RC (Release Candidate) are like the actual gold image, as it progress and other clients testing it will become better with the final release.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 19, 2018, 08:08:22 am
Hello MB, I can see the option in "Table of local / remote assets" to select different top users. Can you also add another option to sort it ascending or descending so that we can check the top user in top list rather then going to the entire list to find one.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 19, 2018, 02:55:22 pm
Another idea about "Session details": give the user ability to restrict begin and end date and time fields to reduce search results to concrete time period.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 19, 2018, 07:17:09 pm
@cgwork, @manjeet, @Antaris,

Many thanks for the suggestions. Feature requests have been added to 0.8 workload. We'll do a more general re-visit to table reports. Please feel free to reach out for more ideas.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: the-mk on December 19, 2018, 08:01:51 pm
a question from a maybe future sensei user:
since this elastic search module needs a lot of diskspace and sure does a lot of writing - is there a possibility to divide the installation into an "OS"-disk (binaries; usually on a SSD) and a "data"-disk (storage intensive data, lots of writes; usually on a HDD)?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: nikkon on December 20, 2018, 12:13:10 am
Hi all,

After upgrading to version 0.7.0-rc3 none of my dashboards or reports are loading anymore

That's an error example:
{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "resource.type": "index_or_alias",
        "resource.id": "conn_all",
        "index_uuid": "_na_",
        "index": "conn_all"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index",
    "resource.type": "index_or_alias",
    "resource.id": "conn_all",
    "index_uuid": "_na_",
    "index": "conn_all"
  },
  "status": 404
}

Any clue?

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 20, 2018, 06:23:27 am
Thanks @MB for considering this.

I have an another thing to ask. I am not if that is 100% possible or is it already implemented because i did not find it in any details.

In report we can see the source address, destination address or host, app category and protocol it is used. It gives us huge information about who has download / uploaded to where and how much data, also time stamp of session etc. But i do not see any ways to check what exactly the user has downloaded. For e.g one of my user used 5GB data in one day which is used by google services and it gives us the list of when and where, but no info about what exactly which for now we have to ask the user. This could be useful because if user is downloading / uploading something not allowed to server / account which they are allowed to access then they probably will deny it.

Also can you add option to export reports (excel or pdf) including custom / filtered reports so that we can provide report to management whenever needed rather then filling mail box with auto reporting.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: maekar on December 20, 2018, 02:25:15 pm
Hi,

Is there anything special to do with VLAN?

We have interfaces tagged and untagged. When I activated Sensei and configured just a few web categories to test, everything worked well with the untagged interface but all VLAN networks lost connectivity, devices in all VLAN not even get IP address by DHCP. And the problem persisted even when I deselected those interfaces to get managed by Sensei, I had to stop it and uninstall it to get VLAN networks working again.

Thanks!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 20, 2018, 05:38:55 pm
Hi @the-mk,

Thank you very much for the suggestion: We get this request quite many times. People who’d like to see this functionality seem to be either running on the low end - the device is very weak and lack the resources to run reporting on the device itself, or they run on the high end - throughput & number of users are quite high (>1K users) and it makes  sense to put reporting on a separate device.

In addressing this requirement, we’ll offer an option - in the initial configuration wizard - asking the user whether s/he wants the reporting on the device itself, or on a remote server.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 20, 2018, 05:44:05 pm
Hi @nikkon,

Looks like alias indexes are messed up. By any chance, did you do any "reset to factory defaults" ?

We'd like to dig deeper. Can you share your /var/log/elasticsearch/elasticsearch-2018-12-19.log through sensei - at - sunnyvalley.io ?

For a workaround, you can run these two commands to reset the indexes: (beware: this will erase your reporting history)

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py


Let us know if this does not fix the problem.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 20, 2018, 05:53:57 pm
@manjeet, you’re all welcome.

If the connection is clear-text (e.g. HTTP), you can see the individual downloaded files from Web Reports: Web - Table of URIs. For the TLS encrypted sessions (e.g. HTTPS), this will be possible with the all ports TLS Inspection feature - though it’s going to be available for Premium Subscriptions.

For the Table reports, development & tests have been completed, and it’s ready to ship with 0.7.0 release.
I’ve sent you a link today to try it and see if there are any more issues.

Reports - PDF export - its’ on the short term roadmap. Probably it will ship with 0.8.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 20, 2018, 06:07:27 pm
Hi @maekar,

Thanks for reporting this. Yes, we’re aware of this problem. Unfortunately part of the solution required some development on the Operating System itself (FreeBSD netmap implementation).

Good news is that hopefully it’ll be fixed with OPNsense 19.1. On the FreeBSD side, we’ve sponsored a development which fixes this and some other issues with the netmap implementation on FreeBSD (https://svnweb.freebsd.org/base?view=revision&revision=340436)

We’ve been testing the 11.2-STABLE MFC code for some time and it looks good to be finally integrated with OPNsense.

We’re working very closely with the OPNsense team on this. I’ll be posting an ETA after we sync with @franco.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: nikkon on December 21, 2018, 02:57:34 pm
@mb thanks for replying
I did execute the 2 scripts.

please check the log below:

cat /var/log/elasticsearch/elasticsearch-2018-12-
elasticsearch-2018-12-16.log  elasticsearch-2018-12-20.log
root@Skynet:~ # cat /var/log/elasticsearch/elasticsearch-2018-12-20.log
[2018-12-20T01:05:36,849][INFO ][o.e.n.Node               ] [yCObJMR] stopping ...
[2018-12-20T01:05:36,889][INFO ][o.e.n.Node               ] [yCObJMR] stopped
[2018-12-20T01:05:36,889][INFO ][o.e.n.Node               ] [yCObJMR] closing ...
[2018-12-20T01:05:36,911][INFO ][o.e.n.Node               ] [yCObJMR] closed
[2018-12-20T01:07:19,550][INFO ][o.e.n.Node               ] [] initializing ...
[2018-12-20T01:07:19,707][INFO ][o.e.e.NodeEnvironment    ] [yCObJMR] using [1] data paths, mounts [[/var (tmpfs)]], net usable_space [1.9gb], net total_space [2.4gb], spins? [unknown], types [tmpfs]
[2018-12-20T01:07:19,707][INFO ][o.e.e.NodeEnvironment    ] [yCObJMR] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-12-20T01:07:19,711][INFO ][o.e.n.Node               ] node name [yCObJMR] derived from node ID [yCObJMRsQcSMKeQy7KNhyA]; set [node.name] to override
[2018-12-20T01:07:19,711][INFO ][o.e.n.Node               ] version[5.6.8], pid[32322], build[688ecce/2018-02-16T16:46:30.010Z], OS[FreeBSD/11.1-RELEASE-p17/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_172/25.172-b11]
[2018-12-20T01:07:19,711][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/local/lib/elasticsearch]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [aggs-matrix-stats]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [ingest-common]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-expression]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-groovy]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-mustache]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-painless]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [parent-join]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [percolator]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [reindex]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [transport-netty3]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [transport-netty4]
[2018-12-20T01:07:21,819][INFO ][o.e.p.PluginsService     ] [yCObJMR] no plugins loaded
[2018-12-20T01:07:25,240][INFO ][o.e.d.DiscoveryModule    ] [yCObJMR] using discovery type [zen]
[2018-12-20T01:07:26,419][INFO ][o.e.n.Node               ] initialized
[2018-12-20T01:07:26,420][INFO ][o.e.n.Node               ] [yCObJMR] starting ...
[2018-12-20T01:07:26,927][INFO ][o.e.t.TransportService   ] [yCObJMR] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-12-20T01:07:30,078][INFO ][o.e.c.s.ClusterService   ] [yCObJMR] new_master {yCObJMR}{yCObJMRsQcSMKeQy7KNhyA}{QHCtod64RcOkM74GkkvW-g}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-12-20T01:07:30,120][INFO ][o.e.h.n.Netty4HttpServerTransport] [yCObJMR] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-12-20T01:07:30,121][INFO ][o.e.n.Node               ] [yCObJMR] started
[2018-12-20T01:07:30,140][INFO ][o.e.g.GatewayService     ] [yCObJMR] recovered

in Gui i got this:
Error at /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php:74 - fsockopen(): unable to connect to 127.0.0.1:4343 (Operation timed out) (errno=2)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 21, 2018, 03:01:24 pm
Hi @Nikkon,

Is this the log after you executed the delete/create scripts, or the one with the errors?

Looks like the former? Did the scripts resolve the problem?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: nikkon on December 21, 2018, 03:16:43 pm
yes. this is before i executed both scripts
it's not solved.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 21, 2018, 05:06:43 pm
Hi @nikkon, understood. Let's do some more debugging together. I'll contact you.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 23, 2018, 06:02:32 pm
Very often i see remote hosts in local table and vice versa. Is something wrong with my setups?
And sometimes i see comunication between two local ip addresses and one of them is marked as remote...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 23, 2018, 08:03:06 pm
Hi @Antaris,

Do you have multiple interfaces configured for Sensei? Are these IP addresses multicast / broadcast addresses?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 23, 2018, 10:53:34 pm
I have only LAN selected in Sensei with only one IP and no VLANs on it. The adresses are known internal hosts. Not broadcast or net addresses.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 26, 2018, 09:56:25 pm
Dear Sensei & OPNsense users,

Happy new year to all. Here is a humble new year present from Sensei team.

We're happy to announce the availability of Sensei 0.7.0 release. It was ready since last Friday, but we wanted to make sure everyone had a calm Christmas holiday, spending time with friends and family instead of doing Sensei deployments :)
 
This is the full list of features that this release brings (from 0.6.x):

1. 350+ new applications identified.
2. Google applications browsed via Chrome are now being identified (QUIC over UDP protocol support).
3. Mobile browser compatibility: you can view reports from your mobile browser
4. Reports enhancement: Data retirement option introduced. With this option you can define how long to keep your reports (days)
5. Reports enhancement: Option to erase all reporting data
6. Reports enhancement: Drill-down in Security reports is now available
7. Reports enhancement: Daily executive reports. Selected reports delivered via a daily e-mail.
8. You can easily add block/allow rules within Session Explorer based on Application and Application Category or SNI / hostname
9. User's Manual in English.
10. More deployment options for Home and Large scale users
11. Changelog between updates
12. Fixed Rebellion Theme compatibility issues.
13. Better Cloud Nodes availability
14. Better & smoother updates
15. We speak your language now, we added i18n support to match your OPNsense UI language. English and German are there for now, more coming soon.
16. Removed some large dependencies in preparation for embedded devices & PIE (Position Independent Executable) support.More performance & stability improvements.

To update your installation, simply navigate to Sensei -> Status and you should see 0.7.0 update being reported and an option to install it. If you do not see the update notification, just click "Check for updates" and you'll be guided through the update process.
 
A quick note: Although this is marked "release", Sensei is still under BETA development. We strongly advise to test the software on one of your test-beds to see if it fits your requirements. When we finally release Sensei 1.0, the BETA program will cease and the software will be publicly available for all users. We expect to release Sensei 1.0 in Q1 2019.
 
If you find any issues or you want to reach out for comments and feedback, please do not hesitate to contact us through sensei -at- sunnyvalley.io or through this forum thread.
 
Happy new year to all

Sensei team
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on December 27, 2018, 07:18:12 am
Thanks for the update @MB.

For Table Reports update is working as expected.

As @Antaris mentioned, i also see remote host in local table but no local host in remote table except OPNsense LAN IP which i think, in one way, is not an issue because firewall itself generates traffic for interface access etc..
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on December 27, 2018, 09:04:14 pm
Also thanks from me for the update.

"12. Fixed Rebellion Theme compatibility issues."

In session details the headers of the columns are still with white text on white background:

https://www.dropbox.com/s/0v72em2bch0rk0q/Reb.jpg?dl=0 (https://www.dropbox.com/s/0v72em2bch0rk0q/Reb.jpg?dl=0)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on December 28, 2018, 01:02:32 am
Can't tell if this is a new issue or not as I only installed of of .7.0-rc3. When the packet engine is running unbound overrides are being ignored.

My nslookup results show "UnKnown" in the server spot and are forwarding my overrides to public servers.
As soon as I stop packet engine this works again.
I was able to add my root domain to the "local domain to override" section and it fixed that one issue there but I have overrides for other hosts. Am I missing a setting where Sensei is overriding DNS?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 28, 2018, 03:18:43 am
Thanks for the update @MB.

For Table Reports update is working as expected.

As @Antaris mentioned, i also see remote host in local table but no local host in remote table except OPNsense LAN IP which i think, in one way, is not an issue because firewall itself generates traffic for interface access etc..

Hi @manjeet, you're very welcome. Can you share with me a screenshot of  the remote hosts table (you know my email). Would like to see how they look like. Normally you should only see local hosts behind the firewall there.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 28, 2018, 03:20:06 am
Hi @Antaris,

You're all welcome & thx for the pointer. We'll fix it.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 28, 2018, 03:32:44 am
Hi @donatom3,

Actually this is an expected behavior. We're utilizing DNS override for Web Reputation & Threat Intel. Since DNS occurs before the actual connection attempt, we gather prior threat intelligence & reputation about the remote IP & host.

For a quick workaround you can disable Cloud Reputation & Web Categorization from Sensei -> Configuration. Then you'll still have reputation data for the top 1Million domains from the local database, but not for +140M :(.

We're exploring ideas to do this in parallel. This way you'll still be able to do your DNS through your DNS server and Sensei will do a parallel query for its intelligence.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on December 28, 2018, 05:04:55 am
@mb this is good to know.
So if I'm in an environment where I'm using Windows domain controllers for DNS to get the full effect of Sensei would I need to have the opnsense router be the DNS forwarder?

Also does this mean if I just hand out public DNS servers via DNS am I not getting the full advantage of Sensei?

P.S. I do want to add that I am liking Sensei so far.
I am still able to download at 1gbps on my i5-5250u but thinking of picking up a box that has an i5-6500.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on December 29, 2018, 07:29:00 am
Hi @donatom3,

For application control, dns does not play any role there, so you'll be utilizing Sensei at its full potential in any ways.

For security & web filtering, yes, you'll lose some data there, provided that you do not enable Web Reputation & Cloud Threat Intel, which requires DNS override.

If your DNS transactions are traversing through an interface which is protected by Sensei & you have Cloud enabled, dns override will be in place (like unbound). Because we are way earlier in the process, we'll also override unbound.

That being said, if you place the DC in the same broadcast domain with your clients, they'll be utilizing your DC (this way they do not traverse through Sensei to reach the DNS server). -- Local DNS server will still be subject to the same behavior if its traffic passes through Sensei. -- Here you have your local queries untouched.

We've been asked (by some more users) of a possibility of using both Sensei Cloud database & local dns servers. We're evaluating methods to do it. Most probably we'll be back into this in the second or third quarter of 2019.

Very happy to hear that you've attained gigabit speeds and happy with the software ;)

Title: Re: Sensei on OPNsense - Application based filtering
Post by: jinn on January 02, 2019, 09:17:50 am
I tested sensei last week. after I activated it, however, access to the internet was barely possible (eg google was not available at all). since it was a productive system, I deactivated sensei for now and did not continue to use it. Nevertheless, I would like to know why it was located, how should I proceed best for analysis here?

In addition, I wanted to ask whether it is even possible, what I want to achieve: I would like an evaluation for special services (social media, online gaming, ...). Is sensei able to give me an evaluation of how much time / data was used for special services?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on January 02, 2019, 12:03:23 pm
Hello @MB, I need another favor from you if possible.

Can you please work with OPNsense team to add an option for Sensei "Dashboard" and "Reports" in "Assigned Privileges" for users/groups. Well, I need to create few users/group so that they can only check the reports of team assigned to them. I do not want to provide root user access level to them to avoid them poking around and change my configuration or delete any logs or data..
Title: how to work with local hostnames?
Post by: the-mk on January 02, 2019, 07:45:19 pm
I finally decided to install Sensei on my box with several network interfaces.
I also have some servers running at those interfaces where I configured different hostname suffixes (configured with dhcp reservations and the checkbox to register the names in unbound dns). i.e. server1.lan, server2.home, server3.iot,...
before running sensei I was able to resolve all hostnames fine.
I guess the setting "local domain name to exclude" in the cloud threat intel tab has something todo with it? tried to enter here one servername for testing - did not work for me... is saving the setting enough or do I need to restart something?
how to tell sensei to honor local servernames when tried to resolve local hostnames?

EDIT: after reading the post of donatom3 and the suggestion of mb to turn off cloud threat intel I can resolve my local hostnames again!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on January 03, 2019, 06:54:05 am
the-mk,

In my case I left that feature turned on. All I did is put my domain in the local domain section of the cloud threat intel section.

Now my local domain is ad.xxxx.com, but I have entries for domain xxxx.com, so I put in xxxx.com into the local domain and all subdomains are passed through correctly to my custom names in unbound.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 03, 2019, 07:21:25 am
@manjeet,

This is a cool feature request. Thanks. Added to roadmap.

A quick note on remote IP addresses on "local assets table": We've had a look at the screenshots. 169.254.x.x is actually a local ip address. Your PC is automatically assigned an IP address, if it cannot get an IP address from a DHCP server. More on this: https://www.techrepublic.com/forums/discussions/where-did-ip-16925451183-come-from/

Screenshots show that some PCs (or a PC) wanted to communicate with the outside world, but it did not get any replies (Incoming packets all zero).

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on January 03, 2019, 11:00:29 am
Thanks @MB and Thanks for the update.

Can you also add one option in reports for looking a live reports without manually refresh time. When in Dashboard / Reports -> Filter (Reports Interval) -> When selecting Custom interval there is "Start time" and "End time".

It will be great if you can add another option or select box there to select "End time" as ongoing.

For e.g: If i want to see current reports from a specific time let says since morning and wants to check the reports after every 10 or 15 min gap then every time i have to select the option "Go to today" in End time. It would be better if there is an option as ongoing which will automatically change time in some specific interval of time or select "refresh interval" as time to refresh and update the time in 'Reports Interval"
Title: Re: Sensei on OPNsense - Application based filtering
Post by: dp on January 03, 2019, 08:02:06 pm
I see that shaping at layer 7 is on the roadmap for sensei. Is there any time table on that feature? Has it even started? I am looking to use it in a 1500-2000 user environment to replace some aging equipment if it is slated for the near future.

Also I have several ideas that I would like to see implemented as I have used application shapers for over 10 years in our environment.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 04, 2019, 06:09:15 am
@manjeet, you're right. They are already in the workload for 0.8 ;)

Hi @dp, correct. Shaping is on the roadmap. Our plan is to feed the currently existing shaping infrastructure on OPNsense. Sensei development is quite booked with IPv6 support nowadays. Though, you should see it implemented like Q2 or Q3 2019. We'd like to keep in touch about ideas on that ;)
Title: Sensei on OPNsense - Cloud Node Status
Post by: lmwalker71 on January 04, 2019, 07:44:40 pm
Under Cloud Node Status, The Nodes are always showing Down, with a count down runs with a 'Check Now" button. If the count down runs its cource the status changes to up for about 15 seconds or if I click 'Check Now' is this the normal??? :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 04, 2019, 08:01:18 pm
Hi @lmwalker71,

Not quite ;)

If you're based in USA, make sure you have the "US - Central" Cloud nodes checked & in green color (Sensei -> Configuration -> Cloud Threat Intelligence). (If in Europe, Europe nodes should be active)

If that's already the case, can you reach out to us through sensei - at - sunnyvalley.io so that we can dig deeper together?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on January 09, 2019, 09:26:35 am
Services are randomly (?) stopping.

I read somewhere that services will stop, when there is less performance, to save power for opnsense native tasks, but I run Sensei on a Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (8 cores) with 24 GB ram which should be quite enough power.

Since I have lacp interfaces for lan (lagg0) and wan (lagg1), each with 2x1g and vlans on lan interface and due to some remarks in this thread that vlans are not supported yet (due to FreeBSD netmap) and will be fixed with OPNsense 19.1, I added an additional, plain interface and just connected 1 pc.

Then I added this single interface with 1 pc as protected interface in Sensei. I even reduced the deployment size from x-large (what I would need if vlans would work) to small in hope that memory footprint will be reduced (actually just 1 user/pc is connected).

But the packet engine still stops randomly. I blocked shopping categories to verify whether engine is just shown as stopped or really stopped. And it is really stopped, since I can open shopping page when service is marked as down.

Same with the ElasticSearch service. Sometimes when I open reports or dashboard, I get a pop-up that ElasticSearch service has to be started first and whether I want this. When I deny and check status page, then both services are down - means whenever packet engine stops, ElasticSearch stops, too.

Any hints what could be the reason for the stopping services.

The service crashes pretty soon. 1-2 minutes after starting up.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: jinn on January 09, 2019, 09:52:04 am
Quote
I tested sensei last week. after I activated it, however, access to the internet was barely possible (eg google was not available at all). since it was a productive system, I deactivated sensei for now and did not continue to use it. Nevertheless, I would like to know why it was located, how should I proceed best for analysis here?

In addition, I wanted to ask whether it is even possible, what I want to achieve: I would like an evaluation for special services (social media, online gaming, ...). Is sensei able to give me an evaluation of how much time / data was used for special services?

currently sensei works with deactivated cloud threat intel.
Unfortunately, "Egress New Connections by APP Over Time" and "Egress New Connections by Source Over Time" show no data:"no egress new connection" what do I have to configure to make it work?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 09, 2019, 03:56:00 pm
Hi @jinn,

Thank you for giving Sensei a try. I see your quoted message did not get response. Sorry for that. I looks like we missed it.

I guess you've been able to figure out the first part yourself. But I wonder why Cloud Threat Intel did not work for you. I'll write to you about this.

For reporting about application categories, yes you can do it. I guess you've started using it.

As for the egress connections report does not show anything. Is it just a single report or all reports which shows egress connections (i.e. local assets, remote assets, eggress conns by source ) do not show anything at all.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 09, 2019, 04:34:43 pm
But the packet engine still stops randomly. I blocked shopping categories to verify whether engine is just shown as stopped or really stopped. And it is really stopped, since I can open shopping page when service is marked as down.

Same with the ElasticSearch service. Sometimes when I open reports or dashboard, I get a pop-up that ElasticSearch service has to be started first and whether I want this. When I deny and check status page, then both services are down - means whenever packet engine stops, ElasticSearch stops, too.

Any hints what could be the reason for the stopping services.

The service crashes pretty soon. 1-2 minutes after starting up.

Hi @hbc,

Thanks for reporting this. After services stop, and when you look at Status-> Services page, do you also see that both services are disabled at boot time?

If yes, most probably this is because Sensei's Health Check subsystem. Because Sensei is in BETA now, checks are more sensitive to problems. Even if it finds a small problem it disables both services in an effort to keep  network connectivity up & running.

Can you try disabling Health Check and see if services are running persistently?

If they do and it turns out because of Health checks, I'd still recommend investigating this. While running Sensei & ES, can you do 'top' on OPNsense console and see if any processes (not necessarily Sensei (eastpect) processes) are consuming much CPU/Memory?

Performance-wise, your system looks pretty decent. We've been reported a similar system handling 700 concurrent users.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on January 10, 2019, 07:52:01 am
Hi @mb,

you are right, I just set ElasticSearch to start on boot and left packet engine disabled for auto-start. I'll try to set both to start on boot.

But I already had try with health check disabled and after a while, no traffic passed at all. But I'll re-check it again.
First with both starting on boot and then with health check disabled.

Update:
The start on boot was not the reason. Whenever packet engine stopped for unknown reason, the option was automatically disabled. I tried it 3 times and reenabled start on boot. But within 5 minutes service stopped again.

As next option I disabled Health Check. Currently the service runs for 20 minutes which is 4 times longer then ever before. I'll keep an eye on it.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 11, 2019, 02:38:11 am
Hi @hbc,

Thank you for further information. Let us know if anything weird comes up.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on January 11, 2019, 07:50:52 am
Hi @MB, I had a similar issue for "Sensei Packet Engine" stops within 5min everytime I enable it. It didn't fix with the reboot as well. But since "health check" is disabled (its been more than 24 hours and reboot few times), service is running without an issue.

I only faced this issue after updated OPNsense to 18.7.10.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: jinn on January 11, 2019, 01:47:37 pm
hey mb, ty for reply!

Quote
For reporting about application categories, yes you can do it. I guess you've started using it.

Not yet. At least not as detailed as I would like (facebook, online shopping, ...)


Quote
As for the egress connections report does not show anything. Is it just a single report or all reports which shows egress connections (i.e. local assets, remote assets, eggress conns by source ) do not show anything at all.

in fact, several do not work: Egress New Connections by App Over Time, Egress New Connections by Source Over Time, Egress New Connections Heatmap, Top Destination Locations Heatmap, Table of Apps (maybe this one is what im really looking for?), Table of Local Assets, Table of Remote Hosts
Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on January 11, 2019, 02:59:16 pm
Good Morning, mb

is it possible to incorporate and additional "TAP" for  Hostname in your tab-bar see picture attachment
Title: Re: Sensei on OPNsense - Application based filtering
Post by: l0rdraiden on January 13, 2019, 10:25:38 pm
What are the plans between sensei and opensense? it will be embedded in opnsense or it will be available as a pluging at some point?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 14, 2019, 06:09:04 am
@hbc, @manjeet: thanks for your update. We're fine-tuning health check auto-bypass.

@cgwork, how about this: likewise destination hostnames, the default will be hostnames for source addresses, and when you move the mouse over the src hostname field, IP address is displayed as a tooltip. Adding another column would make the table more crowded. We'd like to use the space allowance for the new-coming "username" column.

@jinn, are you running Sensei on your LAN? Any chances that it might be on your WAN interface?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 14, 2019, 06:13:58 am
What are the plans between sensei and opensense? it will be embedded in opnsense or it will be available as a pluging at some point?

Hi @l0rdraiden,

It'll be a plugin.

Currently, we're working together to address some issues related to netmap (e.g. virtio). Once it's done, whole integration will be completed, and you'll be able to install it from OPNsense plugin manager.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: cgwork on January 14, 2019, 07:17:53 pm
Quote
@cgwork, how about this: likewise destination hostnames, the default will be hostnames for source addresses, and when you move the mouse over the src hostname field, IP address is displayed as a tooltip. Adding another column would make the table more crowded. We'd like to use the space allowance for the new-coming "username" column.

That sound even better thank you
Title: Re: Sensei on OPNsense - Application based filtering
Post by: OPNsenseN00b on January 15, 2019, 09:36:30 am
Hello there,
Shortly I've registered on the beta program to obtain the required Downloadlink but ssh is rejecting the provided download link after I login into opnsense. The link is slithly different than in the tutorial.
Could you update the Installer-URL please. Many thanks.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: jinn on January 15, 2019, 09:39:33 am
@jinn, are you running Sensei on your LAN? Any chances that it might be on your WAN interface?

it is currently on LAN. The WAN interface is not displayed to me under available interfaces.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 15, 2019, 02:45:44 pm
Hi @OPNsenseN00b,

The command to install Sensei is:

curl https://updates.sunnyvalley.io/getsensei | sh

I checked again. It should be the same both Users' guide (https://guide.sunnyvalley.io/sensei/getting-started/setup) and Website.

Can you copy/paste the error message you get when you run the command on the OPNsense console?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 15, 2019, 02:46:48 pm
Hi @jinn,

Got it. Will send you a few commands to diagnose the issue.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: OPNsenseN00b on January 16, 2019, 02:04:57 pm
Hi mb,
Thanks for your respnose. This time the firewall comes with the message of not having enough memory (8GB required, I have only 4 GB). Yesterday it said something different. I'll come back to you when I can replicate the previous error.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on January 17, 2019, 06:45:10 am
Hi mb,
Thanks for your respnose. This time the firewall comes with the message of not having enough memory (8GB required, I have only 4 GB). Yesterday it said something different. I'll come back to you when I can replicate the previous error.

8GB is currently required to run Sensei. It checks for that when you first initialize it.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: xames on January 17, 2019, 02:18:41 pm
ssl_error_syscall

I attach
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 17, 2019, 06:33:08 pm
ssl_error_syscall

I attach

Hi @xames,

Looks like everything is ok on the server side. Can you try with fetch:

# fetch https://updates.sunnyvalley.io/getsensei
# sh getsensei


Title: Re: Sensei on OPNsense - Application based filtering
Post by: Space on January 27, 2019, 10:18:35 pm
Hi,

I have Sensei running on my OPNsense and I wondered why big part of the traffic did not show up and I see in the FAQ that IPv6 support is still work in progress.

Do you have an ETA for that feature already?

Thanks and looks great so far!

Space
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on January 28, 2019, 09:16:05 pm
Hi @Space,

Many thanks for trying Sensei. Yep, 0.7 is IPv4 only.

Good news is that IPv6 will be coming very shortly with 0.8. It's been under testing for the past months. Looks like it's good to go for a test ride by BETA users.

We'll ship 0.8-beta1 this week or early next week :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on February 02, 2019, 10:20:23 am
19.1 goes in production now, and we that have sensei have a problem:
https://forum.opnsense.org/index.php?topic=11400.msg51520#msg51520 (https://forum.opnsense.org/index.php?topic=11400.msg51520#msg51520)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 02, 2019, 10:27:54 am
Hi @Antaris,

Thanks for reporting this. Looking into it now.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 02, 2019, 11:24:53 am
19.1 goes in production now, and we that have sensei have a problem:
https://forum.opnsense.org/index.php?topic=11400.msg51520#msg51520 (https://forum.opnsense.org/index.php?topic=11400.msg51520#msg51520)

Looks like there was a typo in that command. Correct command should be: (from https://forum.opnsense.org/index.php?topic=11400.msg51521#msg51521)

For OpenSSL:

# opnsense-update -fp -n "19.1\/latest"

Or LibreSSL:

# opnsense-update -fp -n "19.1\/libressl"



Just did an OPNsense 19.1 upgrade on two of our firewalls. Looked good. 

Anyone who had any other issues upgrading to 19.1 ?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on February 04, 2019, 09:08:21 am
Quote
Anyone who had any other issues upgrading to 19.1 ?

Update did not work with sensei nor without. Update started and just installed two kernel/base files, then restarted with 18.7.10. Even when sensei was uninstalled, update did not work. I tried GUI and console.

So I saved config, installed 19.1 clean from image and restored backup and reinstalled sensei.

Now with 19.1, sensei finally works with tagged vlan interfaces  8)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 06, 2019, 02:55:31 am
Hi @hbc,

Thanks for sharing you experience. We're looking into the upgrade problem if it's something related to Sensei repository.

Glad to see that you're enjoying it now :)

Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on February 06, 2019, 02:23:14 pm
Yes, works pretty nice. Just the cloud nodes seem a bit flappy. Most time at least one is displayed down.

One hint:

Traffic to local squid proxy on port 3128 is categorized as "Generic TCPIP". I think it is intention that not labeled as 'Proxy' which would properly cause problems when blocking 'Proxy' category.

But maybe you can label it category 'Web Browsing', application 'Squid Proxy'
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 07, 2019, 02:43:32 am
Hi @hbc,

Thank you very much for the feedback. With regard to Cloud servers, we have a fix for that in 0.8.

Thanks for the suggestion. You're right, and suggestion sounds good ;)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 07, 2019, 05:48:33 pm
Dear Sensei users,

Regarding https://forum.opnsense.org/index.php?topic=11477.0;

To be able to utilize the new functionality that comes with the new netmap - enabled kernel, we'll need to ship Sensei 0.8-beta1 which will re-enable virtio interfaces.

Actual ETA was this week. Still working on a few issues reported. Stay tuned for updates. 
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on February 12, 2019, 10:28:26 am
Hi!

Quote
utilize the new functionality that comes with the new netmap - enabled kernel

One question. I had opnsense 19.1 (fresh install) active with shipped kernel and tagged vlans already worked in sensei (what they did not with 18.7). I assume the new c4ec367c3d9(master) kernel is just for virtio interfaces?
Well, I updated kernel and it still works.

Will there ever be the possibility to set different policies for different interfaces? I have interfaces where I would like to be more restrictive and just allow productive things and interfaces where social media, gaming, etc. would be ok.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 13, 2019, 02:38:07 am
Hi hbc,

Yep, correct. VLANs were also broken, but it looks like it was fixed with the FreeBSD 11.2 update. My note was about virtio support. Sensei 0.7 filters out any virtio interfaces. 0.8 will remove this filtering so that they will be presented in the Interface Selection.

Source Interface/Network Address/IP Address/VLAN/User/Group filtering is a feature of Policy based enforcement, which will be showing up with the Premium Subscription.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on February 13, 2019, 09:42:35 pm
Hi hbc,

Yep, correct. VLANs were also broken, but it looks like it was fixed with the FreeBSD 11.2 update. My note was about virtio support. Sensei 0.7 filters out any virtio interfaces. 0.8 will remove this filtering so that they will be presented in the Interface Selection.

Source Interface/Network Address/IP Address/VLAN/User/Group filtering is a feature of Policy based enforcement, which will be showing up with the Premium Subscription.

My advice is to consider exchange "Source Interface/Network Address/IP Address/VLAN/" for volume of users above 1000 or so... It's vital for usability and development at all IMHO.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on February 14, 2019, 03:22:24 am
@Antaris, Thanks for your input. We'll definitely make use of your feedback.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Kruemel on March 01, 2019, 11:39:29 am
Hi,

greetings from germany.  :)
Great so see such a powerful addon for OPNSense. It was the reason to migrate my APU2C4 to VMWare on HPE ProLiant Xeon CPU, to fulfill the Sensei requirements.

However, it's working great. But I miss a feature: If something is blocked, it's just not loading, right? But the user is not aware, if it's a not working webpage (or parts on it) or if it's blocked. It would be great, if Sensei delivers some kind of block page, something like "This page has been blocked - block category is xxx. Please contact abc@def.de for further information".

Did I miss something in the settings or this feature currently missing?

Keep on the good work!
Cheers
Marco
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 02, 2019, 02:38:46 pm
Hi Kruemel,

From Sunnyvale, California, greetings to you too :) Glad to hear that Sensei is of value to your OPNsense installation. Many thanks for sharing your experience.

We hope to bring some news with regard to less demanding hardware requirements. We're planning to employ an alternative less resource-intensive database engine for reporting.

Quote
But I miss a feature: If something is blocked, it's just not loading, right?

Yep. This is so because, your Sensei policy configuration hits a TLS SNI or application rule. TLS and some app detection jump into the scene way too early before the HTTP protocol starts being conversed back and forth between your browser and the server. 

So when we decide that we need to apply filtering, neither server nor client does not yet know how to talk HTTP. They just know how to talk TCP. This is why we just do a TCP RST, and you see a blank page in your browser.

We'll have a feature called "delayed action" (requires TLS inspection) where we'll flag a particular connection as being blocked and will let them talk a little bit more so that they can handle a HTTP response. As soon as we get a HTTP request from the client, we'll send the landing page and just close the connection at that particular time.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: astoklas on March 03, 2019, 10:27:51 am
Hi,

I just installed Sensei on my OPNsense and I think it's working great.
I found in the dashboard an interesting "HotSpot" I'd like to investigate further. However, the "Top Destinations Locations Heatmap" does not allow for a Drill Down, nor is there a geo location filter available.

Can you please advise on how to investigate on such hotspots?
Is it possible to retrieve DNS/IP for a certain geo location hotspot?

Regards
Alexander
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 04, 2019, 06:46:19 am
@astoklas,

Many thanks for the feedback. Currently, drill-down is not possible with the map. We'll take this as a feature request. Will get you updated.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 04, 2019, 06:56:07 am
Dear Sensei users,

After several months of field testing, we are super happy to announce the availability of Sensei 0.8.0 Beta.

Release 0.8 introduces long awaited support for IPv6 and virtual ethernet adapters. Below is the full list of features that are coming along with this release (from 0.7.0)


For more information: https://www.sunnyvalley.io/blog/sensei-0-8-beta1-is-released

Currently we're shipping 0.8.0 beta1 from a separate package repository. So, if you are on 0.7, you'll not be able to see the software update as of now. When 0.8.0 rc1 is released, we'll move the packages to the main repository and you'll then be able to update to 0.8.0.

The reason behind this is that we want to allow 0.8.0 a bit more field testing before we make it an update for 0.7 stable users.

ETA for 0.8.0.rc1 is March 18, 2019.

If you don't want wait and want to see 0.8 in effect now, just uninstall Sensei from the UI and use the following one-liner command to re-install:

# curl https://updates.sunnyvalley.io/getsensei8 | sh
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on March 04, 2019, 08:46:19 pm
Thanks, mb, and keep up with good work!

Is "VLAN child interfaces support *with OPNsense 19.1.x" means that filtering on VLANs work without netmap kernel?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 04, 2019, 09:15:24 pm
Hi @antaris,

Many thanks. You're correct. It looks like FreeBSD 11.2 default kernel had some fixes with regard to that.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on March 06, 2019, 06:45:49 am
I'm having a problem where elasticsearch won't start after a reboot. I have to clear the settings completely and re setup sensei to get elasticsearch to start.

Just seeing the below in the general log.

Code: [Select]
root: /usr/local/etc/rc.d/elasticsearch: WARNING: failed to start elasticsearch
This is in the backend log and it keeps adding to it.
Code: [Select]
Mar 5 21:44:55 configd.py: [7d62e2b1-bcce-48d3-a80b-4b665aed6cb4] read sensei stats
Mar 5 21:44:54 configd.py: [a4351d00-f929-466b-a18d-1752f72e0a8c] read sensei stats
Mar 5 21:44:53 configd.py: [40ea2e8d-6574-4662-a135-a4c817bf7f0c] read sensei stats
Mar 5 21:44:52 configd.py: [86399ab0-e991-4493-b62f-d6a2b29d88b3] read sensei stats
Mar 5 21:44:51 configd.py: [b8bfc148-83a2-407f-91d3-7091c77b7832] read sensei stats
Mar 5 21:44:50 configd.py: [baf1dddc-39c6-49e4-aad3-f6d87d29a0da] read sensei stats
Mar 5 21:44:49 configd.py: [f08d4d14-f236-4d25-8011-8b25a848eeec] read sensei stats
Mar 5 21:44:48 configd.py: [571d2e9b-d0cb-402c-b5ac-8bf7ff72d811] read sensei stats
Mar 5 21:44:47 configd.py: [e77883ce-8f8b-4a2b-aebb-7c4125ed7e17] read sensei stats
Mar 5 21:44:46 configd.py: [18dd5adf-9437-4e15-90ba-1ee6e08c4bff] read sensei stats
Mar 5 21:44:45 configd.py: [105c9ddc-960b-4bff-98fa-3e202c9ac49e] read sensei stats
Mar 5 21:44:44 configd.py: [87cb6f2f-e3ca-42b0-8040-4cfacd647de8] read sensei stats
Mar 5 21:44:43 configd.py: [4228579b-7e43-4138-8ea8-414fc9ec1c1a] read sensei stats
Mar 5 21:44:42 configd.py: [a755740c-45d8-438c-99e4-a232bd02c661] read sensei stats
Mar 5 21:44:41 configd.py: [024f64e4-2fa6-4558-8482-d8330cbc7742] read sensei stats
Mar 5 21:44:40 configd.py: [327c339b-b0b2-484c-92f9-3c9e9364820e] read sensei stats
Mar 5 21:44:39 configd.py: [396bb45c-c1f1-4728-91d0-e33bbcaea1f5] read sensei stats
Mar 5 21:44:38 configd.py: [d6b674d1-dd2f-494b-927d-ad55791063e4] read sensei stats
Mar 5 21:44:37 configd.py: [40338097-db55-4b60-b45f-877a1ae76b7c] read sensei stats
Mar 5 21:44:36 configd.py: [304857d4-7d26-45aa-ae75-6c520958fba9] read sensei stats
Mar 5 21:44:35 configd.py: [13675e7f-5dc6-4457-b5c9-c4b4c21e8a58] read sensei stats
Mar 5 21:44:34 configd.py: [4f0f6ae9-f39f-48ae-a799-876c86cb3164] read sensei stats
Mar 5 21:44:33 configd.py: [f4a1bb7f-8d12-47bd-b7d3-403d159450b4] read sensei stats
Mar 5 21:44:32 configd.py: [9c67445c-4ffe-444e-ba3c-a5f444ffbf21] read sensei stats
Mar 5 21:44:31 configd.py: [1cfc4b5a-c263-4240-b627-938197d72afe] read sensei stats
Mar 5 21:44:30 configd.py: [adbefd78-9c10-45e9-9cad-8d6495388773] read sensei stats
Mar 5 21:44:29 configd.py: [ad4176d3-1c8a-4890-a90c-c9b734979673] read sensei stats
Mar 5 21:44:28 configd.py: [22ff41e4-fc8f-4ba7-9f27-63d6c2b23b7e] read sensei stats
Mar 5 21:44:27 configd.py: [1fe553d1-06c5-4db6-b950-7a71e5af7bd4] read sensei stats
Mar 5 21:44:26 configd.py: [c3252f98-b238-448a-af02-d311a6f75e49] read sensei stats
Mar 5 21:44:25 configd.py: [09153632-0bff-46ad-ad98-c45319cd5ff8] read sensei stats
Mar 5 21:44:24 configd.py: [0bbec0b1-6e86-4930-a57c-f57be9e83008] read sensei stats
Mar 5 21:44:23 configd.py: [dcf30e51-763b-4df9-9f53-239615912384] read sensei stats
Mar 5 21:44:22 configd.py: [49c214e7-9b60-44c8-9ded-b22ac257f02c] read sensei stats
Mar 5 21:44:21 configd.py: [463b3e7f-c8d6-48ae-8064-08a414fa7e5d] read sensei stats
Mar 5 21:44:20 configd.py: [6ead17e8-53b9-48aa-a6b7-a644d5f170d2] read sensei stats
Mar 5 21:44:19 configd.py: [12378048-9b6d-4c5c-852d-6575fab78706] read sensei stats
Mar 5 21:44:18 configd.py: [bc415b0c-fe6c-404e-a5fb-a99e6b2646bc] read sensei stats
Mar 5 21:44:17 configd.py: [2b46da7d-1325-4e1c-aba0-20bc12e7e4b3] read sensei stats
Mar 5 21:44:16 configd.py: [720bebee-2387-4735-b794-085b94f5b505] read sensei stats
Mar 5 21:44:15 configd.py: [829b4c54-6629-4ae1-81fc-5a3255ba1c91] read sensei stats
Mar 5 21:44:14 configd.py: [80d84ec1-5cee-4f60-9290-bcaba50a351d] read sensei stats
Mar 5 21:44:13 configd.py: [6b233cd4-81d2-4569-99f6-2989332cb14b] read sensei stats
Mar 5 21:44:12 configd.py: [31706105-d805-41bf-b201-8f75e72fe5b3] read sensei stats
Mar 5 21:44:11 configd.py: [e0f1c395-db7e-4ee1-bdd7-e20ee8ff1dfa] read sensei stats
Mar 5 21:44:10 configd.py: [3f704530-859b-4e1f-95dd-136f85219d4b] read sensei stats
Mar 5 21:44:09 configd.py: [ab29e24e-2146-49e3-9bb6-fb6064233ff2] read sensei stats
Mar 5 21:44:08 configd.py: [645ca172-5629-4ea5-ad1f-8538c1b1ea06] read sensei stats
Mar 5 21:44:07 configd.py: [f8b70f86-0bee-4880-9306-bb4450d7db4d] read sensei stats
Mar 5 21:44:06 configd.py: [8bd95d71-bd13-4ec0-8f27-ed3932579bd3] read sensei stats
Mar 5 21:44:05 configd.py: [be4feb64-ef8e-4756-9e0c-0bbe00f5d4d0] read sensei stats
Mar 5 21:44:04 configd.py: [1aa6cf3a-da0e-473c-b710-553aa1287d69] read sensei stats
Mar 5 21:44:03 configd.py: [12d70d27-8724-477b-a274-99e795bcac42] read sensei stats
Mar 5 21:44:02 configd.py: [91adebc2-e1ee-4cf8-87c2-e1d8a5e8eee1] read sensei stats
Mar 5 21:44:01 configd.py: [ac505fe1-4ebb-4c68-99a7-a684c7f43a99] read sensei stats
Mar 5 21:44:00 configd.py: [7acfc145-9a17-40eb-be37-841d034621e7] read sensei stats
Mar 5 21:44:00 configd.py: [92b767af-81f1-4a5e-9e00-25219f89c715] check sensei engine health
Mar 5 21:43:59 configd.py: [d32f3278-e509-4969-b4a8-7ae7c79c700c] read sensei stats
Mar 5 21:43:58 configd.py: [ad2a102f-b1e0-4bb5-a593-09df77d04bac] read sensei stats
Mar 5 21:43:57 configd.py: [b92813e9-1cef-4b7f-8480-87b49d02d4f6] read sensei stats
Mar 5 21:43:56 configd.py: [d54e5bf2-f367-428a-a8d6-831488f4023e] read sensei stats
Mar 5 21:43:55 configd.py: [189af746-8852-4feb-bc24-2a13da1ff032] read sensei stats
Mar 5 21:43:54 configd.py: [dc2193ce-51c2-451e-917e-ebd56814ad1a] read sensei stats
Mar 5 21:43:53 configd.py: [08950c34-f59e-4fa5-95d5-0af61c02bdd1] read sensei stats
Mar 5 21:43:52 configd.py: [ea882489-9044-4768-b09c-ed6a0d5edd6d] read sensei stats
Mar 5 21:43:51 configd.py: [a4beae9e-0848-46df-bfd2-9e884d455d64] read sensei stats
Mar 5 21:43:50 configd.py: [66bc19f1-867a-4cff-bd31-e21221374c82] read sensei stats
Mar 5 21:43:49 configd.py: [1cff607f-dfba-4adb-8839-82dc49b1b83f] read sensei stats
Mar 5 21:43:48 configd.py: [7fee0851-b848-48d8-8d26-bc84b8bdce1b] read sensei stats
Mar 5 21:43:47 configd.py: [a5261abd-d409-4b27-921c-4f7f7ec41b90] read sensei stats
Mar 5 21:43:46 configd.py: [b8b7127a-5d56-408d-b7dd-902dd95e9ea2] read sensei stats
Mar 5 21:43:45 configd.py: [48a32138-cf91-4641-be4f-045f04ec7af6] read sensei stats
Mar 5 21:43:44 configd.py: [8c4ef497-2b33-4144-ba5b-4ef31a654070] read sensei stats
Mar 5 21:43:43 configd.py: [37cfb408-8ef5-408b-9348-53bcbb5bd089] read sensei stats
Mar 5 21:43:42 configd.py: [939282e0-234c-4b5f-ab00-9113bd803c96] read sensei stats
Mar 5 21:43:41 configd.py: [2989a365-034b-4aa6-b69f-a11ad3bd61c9] read sensei stats
Mar 5 21:43:40 configd.py: [5264a79b-1cf0-4d63-83a7-01129eead1ce] read sensei stats
Mar 5 21:43:39 configd.py: [3a8b90d3-46eb-494f-a19f-78817048cd12] read sensei stats
Mar 5 21:43:38 configd.py: [950f188d-26bd-4e9c-ac76-d65cdb48e212] read sensei stats
Mar 5 21:43:37 configd.py: [cea553fe-507d-492d-ab6d-f4318a600400] read sensei stats
Mar 5 21:43:36 configd.py: [f5b111b5-b585-4843-83bb-0a1bbfb2c1cd] read sensei stats
Mar 5 21:43:35 configd.py: [606ca68b-d3c0-4331-b410-afd4fef1a96c] read sensei stats
Mar 5 21:43:34 configd.py: [995954f6-fa00-4a3a-b32a-5638fa5eaffc] read sensei stats
Mar 5 21:43:33 configd.py: [3a856c39-6a60-4c23-83d7-15e7a00c2472] read sensei stats
Mar 5 21:43:32 configd.py: [3cfda134-4227-4c55-bcca-8ee10229e527] read sensei stats
Mar 5 21:43:31 configd.py: [9e43feed-c461-47fa-b692-8d445f317f4f] read sensei stats
Mar 5 21:43:30 configd.py: [02568a2b-6285-4431-bd2e-081b6bc3d77e] read sensei stats
Mar 5 21:43:29 configd.py: [72dbb649-88a3-4991-b51a-47c698256ce4] read sensei stats
Mar 5 21:43:28 configd.py: [1473e74d-fce9-4173-a6fa-bf54eb577778] read sensei stats
Mar 5 21:43:27 configd.py: [4a6222fc-465d-4528-9dcc-c906a5de1855] read sensei stats
Mar 5 21:43:26 configd.py: [b82dd2a5-8c9a-4a02-be10-6ad52bbaac5e] Show system activity
Mar 5 21:43:26 configd.py: [670749ac-91e3-4643-a9c4-5b9fd44f94da] read sensei stats
Mar 5 21:43:25 configd.py: [30d3970c-86fe-4d91-bca6-7353c654df63] read sensei stats
Mar 5 21:43:25 configd.py: [9a8daded-b8e5-4f51-bc56-d016e8ac7c02] read sensei stats
Mar 5 21:43:24 configd.py: [ebb18255-5159-4ab9-b641-b88821bf1e7d] read sensei stats
Mar 5 21:43:24 configd.py: [5120fa8d-e8ef-48a4-96e9-ffe553f81d30] read sensei stats
Mar 5 21:43:23 configd.py: [b727b40c-13ef-4d1e-b251-bf71c98a5b2f] read sensei stats
Mar 5 21:43:23 configd.py: [3634a274-5368-48a6-8867-b9932cd4809d] read sensei stats
Mar 5 21:43:22 configd.py: [0fb20dcf-c03b-4582-9c36-535207c9fa7f] read sensei stats
Mar 5 21:43:22 configd.py: [7d93ab3c-e1d8-452a-9863-c048ca11e7ff] view elasticsearch disk size
Mar 5 21:43:22 configd.py: [f09b62e6-cbf1-41be-97ae-56cce24ed05f] control services
Mar 5 21:43:22 configd.py: [e52be1cb-68be-4eea-b9e1-6c7b0f4e583c] check sensei ui version
Mar 5 21:43:22 configd.py: [02277005-468d-418c-aeea-5f26e03a016a] check sensei db last modified
Mar 5 21:43:22 configd.py: [5d851b8a-fda4-41cc-9967-7fe8ac178622] check sensei db version
Mar 5 21:43:22 configd.py: [99541288-f562-4f59-aa05-8a9b326cac81] check sensei db last modified
Mar 5 21:43:22 configd.py: [a29ac723-7f8f-41c0-8f73-26d60fc2493e] check sensei db version
Mar 5 21:43:22 configd.py: [37de4a96-014a-47fb-b12c-9c6c6aef5f37] check sensei last modified
Mar 5 21:43:22 configd.py: [7b58d2c8-5505-4df3-8a36-c4a6cf63c70b] check sensei version
Mar 5 21:43:22 configd.py: [9f2677fa-a66d-4e81-9d48-3191f60db682] control services
Mar 5 21:43:21 configd.py: [271b39f0-44fd-4ca1-9a0d-57e074e2ac8c] read sensei stats
Mar 5 21:43:20 configd.py: [8be4d78e-c447-4ff4-92b9-8d2de2a0b9a1] view license
Mar 5 21:43:20 configd.py: [ed3ffc6c-13a6-4468-b09d-2c2cba7469d6] read sensei stats
Mar 5 21:43:19 configd.py: [8483e0c4-6b9e-4cb6-a9ff-ac0cceed2488] read sensei stats
Mar 5 21:43:19 configd.py: [eb9e9a55-1aa1-4ece-a8cb-f71a0b1e3d0c] control services
Mar 5 21:43:18 configd.py: [caaf4bb7-d2af-4258-bba1-960e1b3b3bcb] read sensei stats
Mar 5 21:43:17 configd.py: [77b7f220-2a12-4238-a4f4-622639abb5a2] read sensei stats
Mar 5 21:43:16 configd.py: [fbb0669d-a17f-4918-b158-f28d2cc86aae] read sensei stats
Mar 5 21:43:15 configd.py: [f22ac12a-fdbe-45aa-9e2e-cd75abbc5c68] read sensei stats
Mar 5 21:43:14 configd.py: [04bf4e69-7021-48d4-a14c-429bad0bcd9e] read sensei stats
Mar 5 21:43:13 configd.py: [7f0bca65-1c34-45a5-9816-192eedcadc21] read sensei stats
Mar 5 21:43:13 configd.py: [cde48204-6443-48be-93b8-5c57c8d3cb4b] read sensei stats
Mar 5 21:43:12 configd.py: [d9669127-1ec6-482b-9800-34bf1090604d] read sensei stats
Mar 5 21:43:12 configd.py: [9fd1971a-e907-4704-b0b6-9ef8c193b4a0] read sensei stats
Mar 5 21:43:11 configd.py: [7e084ad4-bd04-40b7-a269-f86b030d470b] read sensei stats
Mar 5 21:43:11 configd.py: [e2f40c45-1449-4eaa-adad-392535ab65b9] read sensei stats
Mar 5 21:43:10 configd.py: [c06c00d0-29c3-424c-805a-624b8bb86c2c] read sensei stats
Mar 5 21:43:10 configd.py: [d44777a5-aede-4403-9963-65f5caf835f8] read sensei stats
Mar 5 21:43:09 configd.py: [5d031005-ce3b-4ddb-b119-c15818b64d7c] read sensei stats
Mar 5 21:43:09 configd.py: [4aaab29d-dd26-499b-8a94-114f728d447c] read sensei stats
Mar 5 21:43:08 configd.py: [32811901-60a5-41fb-8a70-23df003b409a] read sensei stats
Mar 5 21:43:08 configd.py: [e7f2cf0d-5ba4-4b5e-bb0f-6483884c55a7] read sensei stats
Mar 5 21:43:07 configd.py: [7e830b6f-f83d-417e-ad4c-a9ed577644dc] read sensei stats
Mar 5 21:43:07 configd.py: [997cb509-1145-43ea-a461-ed291432856c] read sensei stats
Mar 5 21:43:06 configd.py: [54e86060-313f-4c37-b7c8-ce55f24c5363] read sensei stats
Mar 5 21:43:06 configd.py: [b580155d-f96d-4c35-a94a-19b784208558] read sensei stats
Mar 5 21:43:05 configd.py: [eeddf8f5-89b1-491e-a627-aa879133e63a] read sensei stats
Mar 5 21:43:05 configd.py: [4beb04bf-4103-48ae-86ed-98c9ee7f96d0] read sensei stats
Mar 5 21:43:04 configd.py: [08eac025-5388-4807-9da7-f1d6004c4926] read sensei stats
Mar 5 21:43:04 configd.py: [106e18d5-ee88-4dba-b5e7-6d0d4921d065] read sensei stats
Mar 5 21:43:03 configd.py: [3532ac59-95e9-4439-9837-7a1ab5188a8a] read sensei stats
Mar 5 21:43:03 configd.py: [966fa7d7-c5f7-4809-b72f-fafd7e230bf0] read sensei stats
Mar 5 21:43:02 configd.py: [c87d2a2b-3b5c-44be-8e78-5fc89b1ee7b4] read sensei stats
Mar 5 21:43:02 configd.py: [fbc26fe4-dfc6-4991-bf26-6fa726d28c13] read sensei stats
Mar 5 21:43:01 configd.py: [2cfd5f28-21ce-4651-8a6f-68d7bc4ee5bf] read sensei stats
Mar 5 21:43:01 configd.py: [ad503b54-302c-4534-961b-7f4ffd830022] read sensei stats
Mar 5 21:43:00 configd.py: [edd42365-060e-4e8f-8bfb-9022ae8630e2] read sensei stats
Mar 5 21:43:00 configd.py: [9dc39d58-07bd-443d-bd2d-781a88573d10] read sensei stats
Mar 5 21:43:00 configd.py: [bf2bdcc2-2775-40c7-98c9-512ff7032409] check sensei engine health
Mar 5 21:42:59 configd.py: [ef64a92c-1456-4c26-92fd-72d259adfb70] read sensei stats
Mar 5 21:42:59 configd.py: [bd987828-89f8-46c4-8104-1f78e2c395da] read sensei stats

I attached the elasticsearch log. This only happens after a reboot with sense .8 beta 1 installed.

Here is the error I get when I start elasticsearch from the shell

Code: [Select]
root@OPNsense:~ # service elasticsearch start
Starting elasticsearch.
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME
/usr/local/etc/rc.d/elasticsearch: WARNING: failed to start elasticsearch


Looks like the java env variable isn't being saved in the elasticsearch file or getting overwritten on a startup.

I ran this part of the sensei-init.sh script manually and elasticsearch started with no error now.

Code: [Select]
echo -n "Setting up elasticsearch..."
mkdir -p /usr/local/lib/elasticsearch/plugins
chmod -R 755 /usr/local/lib/elasticsearch/plugins
sysrc elasticsearch_login_class="root" >/dev/null 2>&1
sed -i '' -E '/auto_create_index/d' /usr/local/etc/elasticsearch/elasticsearch.yml
echo "action.auto_create_index: false" >> /usr/local/etc/elasticsearch/elasticsearch.yml
/usr/bin/sed -i '' 's/opt\/eastpect\/run\/elasticsearch/var\/run\/elasticsearch/g' /usr/local/etc/rc.d/elasticsearch
/usr/bin/sed -i '' 's/Xms512m/Xms2g/g' /usr/local/etc/elasticsearch/jvm.options
/usr/bin/sed -i '' 's/Xmx512m/Xmx2g/g' /usr/local/etc/elasticsearch/jvm.options
echo 'elasticsearch_enable="YES"' > /etc/rc.conf.d/elasticsearch
echo 'elasticsearch_env="JAVA_HOME=/usr/local/openjdk8"' >> /etc/rc.conf.d/elasticsearch
echo "done"
I'm fairly certain it's the second to last line that's fixing elasticsearch. Just why that isn't surviving past a reboot is beyond my skill set with this.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 06, 2019, 02:09:11 pm
donatom, thanks for the detailed report.

You are right, it's:

echo 'elasticsearch_env="JAVA_HOME=/usr/local/openjdk8"' >> /etc/rc.conf.d/elasticsearch

that's fixing it. JAVA_HOME variable should be set to openjdk8 directory.

We're having a look at it why it is not persisting.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on March 06, 2019, 03:57:19 pm
Mb,

Beyond the elasticsearch issue everything else is working so far. IPv6 is definitely working and blocking categories.
With .7 my ram usage would hover around 4.8gb. With .8 it started around 4.8 but when I went in this morning dropped down to 2.7gb. The only time ram dropped on .7 was when elasticsearch had crashed.

I don’t know if it’s from enabling ipv6 again on my lan or something with .8 but web pages are loading quicker by a noticeable margin as well. I did also turn on cloud threat intel so it could be that too.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 07, 2019, 02:28:48 am
Hi donatom3,

Many thanks for the detailed feedback. Very good to see 0.8 with IPv6 is running good.

We've fixed a bug with regard to the Elasticsearch rc script. Our configuration manager was overriding it under a condition. Now elasticsearch starts on boot with no problem.

Wait for 0.8.0.beta2 update. It should be arriving momentarily.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: cfsl1994 on March 09, 2019, 02:27:36 am

Good day to all  :),

Recently I'm trying out the sensei package at OPNsense and I thought it was very good, it left me surprised. My questions are:

I would like to know if the primium subscription option is available?

How can I apply filtering for certain IPs?

Thanks
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 09, 2019, 02:46:22 am
Hi cfsl1994,

Many thanks for sharing your feedback. Great to see that Sensei is up to your expectations.

Yep, premium subscription will be available and will come with source IP/network based filtering. You'll be able to create custom policies and apply them to different user groups.

We expect to have Sensei 1.0 in early April and will start offering Premium subscription beginning early May.

Beginning with 1.0 version, Sensei will be directly instalable from OPNsense plugin manager.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: opnsenseuser on March 09, 2019, 10:10:53 am
Hi cfsl1994,

Many thanks for sharing your feedback. Great to see that Sensei is up to your expectations.

Yep, premium subscription will be available and will come with source IP/network based filtering. You'll be able to create custom policies and apply them to different user groups.

We expect to have Sensei 1.0 in early April and will start offering Premium subscription beginning early May.

Beginning with 1.0 version, Sensei will be directly instalable from OPNsense plugin manager.

I would wish to incorporate a function that may have fewer features, but also works on low end cpu's better or at all works.
Because in order to really use sensei you need a cpu that consumes a lot of electricity and therefore generates a lot of costs for the private user.
I would be very happy about such a feature and certainly others as well.

Thanks for the great product! Regards rené
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 09, 2019, 02:22:25 pm
Hi rené,

Many thanks for sharing your suggestion.

I'd like to happily tell that we have two ongoing projects which involve:

1. To make Sensei run on very low end devices, which have weak CPU and memory under 1GB. 
2. To make Sensei run on very large deployments e.g. sites with thousands of users.

For the former, the hurdle is the backend database. Although it's very efficient for medium to large settings, Elasticsearch is heavy for small deployments. It simply does not successfully run under 4GB memory. We're currently evaluating and testing several other databases which will do the job for small settings.

Expect to hear more on this late fall this year.

With regard to the latter, also this year, we'll announce a solution which will be able to handle many thousand concurrent users.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: opnsenseuser on March 09, 2019, 04:03:52 pm
Hi rené,

Many thanks for sharing your suggestion.

I'd like to happily tell that we have two ongoing projects which involve:

1. To make Sensei run on very low end devices, which have weak CPU and memory under 1GB. 
2. To make Sensei run on very large deployments e.g. sites with thousands of users.

For the former, the hurdle is the backend database. Although it's very efficient for medium to large settings, Elasticsearch is heavy for small deployments. It simply does not successfully run under 4GB memory. We're currently evaluating and testing several other databases which will do the job for small settings.

Expect to hear more on this late fall this year.

With regard to the latter, also this year, we'll announce a solution which will be able to handle many thousand concurrent users.

if you really want to do that and really do it, some users would be very grateful to you. Me too of course! ;-)

How can I or how can others keep an eye on the development of this feature?
Is there a kind of roadmap or something similar?

Greetings René
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 10, 2019, 12:27:05 am
Quote
if you really want to do that and really do it, some users would be very grateful to you. Me too of course! ;-)

How can I or how can others keep an eye on the development of this feature?
Is there a kind of roadmap or something similar?

Hi René,

We will do it :) You're all welcome.

To keep up with the development, roadmap etc, best is to keep following this forum thread and also following company web site and twitter account:

https://twitter.com/sunnyvalley

Beginning April, we'll share more information about the upcoming feature set and more about the technology.

For now, I can tell that the technology at the heart of Sensei is a powerful packet analysis engine which is aimed at providing contextual network visibility, protection at all ports for all devices and also protection against encrypted threats which are gaining momentum.

Utilizing this core tech, our mission is to provide enterprise grade cyber protection for everyone, let it be a household, a small business or an enterprise with thousands of users.

From this perspective, making Sensei run on any scale is our priority.

 
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on March 10, 2019, 05:20:55 am
And you start working on getting it to run on lower end machines after I order the new qotom case with 6 built in intel nics and a lga 1151 slot for 6th of 7th gen core desktop processors.

It's the Qotom Q600G6 for anyone interested.
https://www.aliexpress.com/item/Qotom-DIY-Powerful-Firewall-Router-Appliance-Q600G6-Barebone-System-Support-6th-7th-Gen-Processor-DDR4-RAM/32967092263.html?spm=a2g0s.9042311.0.0.154d4c4d2CNERH
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on March 16, 2019, 01:44:03 pm
HI, I Can not open report in either Dashboard or Reports giving me an error "An error occurred while report is being loaded!".

In view error message it says:
{
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": []
  },
  "status": 503
}

Both "Sensei Packet Engine" and "Elasticsearch" are running. I have restarted the system and error is still there.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 16, 2019, 04:41:36 pm
Hi manjeet,

Thanks for reporting this. Are you on 0.7?

We've got two more reports for the same problem and currently investigating it.

We'd like to dig deeper. Can you share your relevant elasticsearch.log ( located at /var/log/elasticsearch/ ) through sensei - at - sunnyvalley.io ?

For a workaround, you can run these two commands to reset the indexes: (beware: this will erase your reporting history)

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py

Let us know if this does not fix the problem.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: ltb76 on March 17, 2019, 04:41:31 pm
Hi,

I'm new to OPNsense and Sensei, testing it to replace my soon expering PaloAlto home firewall.

Just did a default install and it seems to be working well (I see several blocked add sites under "Blocked Sites Explorer").
I might be missing something though. I tried adding "Bing" under "App Controls" - however I can still access bing.com. (I then tried adding Facebook - and that blocks Facebook). might the "bing" app be broken or am I missing something?

Another question, I looked in the manual but did not find the answer. Initially I added all my interfaces (WAN, LAN, LAN2 and DMZ) under "Protected Interfaces". dooing that seems to block DNS.
With the WAN interface protected, DNS trafic seems to be blocked with "Network Management category is administratively restricted" - even if does not appear to be blocked under "App Controls". Should I only add "LAN" interfaces to "protected"?

Is there a way to "not protect" an IP on a protected interface? Lets asume I have a device / client on the LAN interface that I for some reasone want to bypass all checks - is that posible?

I'm running
Sensei: 0.8.0.beta4
OPNsense: 19.1.4
Running ontop of VMware, 4 vCPU (D1540), 12GB RAM, vmxnet3 NICs
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on March 17, 2019, 05:58:19 pm
Quote
Should I only add "LAN" interfaces to "protected"?
AFAIK Sunnyvalley recommends not to block WAN and use suricata for this instead.

Quote
Is there a way to "not protect" an IP on a protected interface?
Not in the free version. That is a feature of the premium edition.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on March 19, 2019, 09:08:30 am
Thanks @MB. This fixed the issue.

I am currently running 0.7 & I am sending you the email for logs and screen shot error.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on March 19, 2019, 09:35:04 pm
I have a question about the VLAN feature.
I use some VLAN on OPNSense and added all my interfaces to the "protected interfaces".
After that all connected VM´s inside the VLAN´s are offline and unable to access the opnsense (which means they are offline for all networks)

If i remove the "LAN" interface from the "protected interfaces" which is my physical interface,
the access from the VM´s inside the VLAN´s is ok again.
I have clients connected to "LAN" as well and would like to protect them, too.

Here is a overview:

LAN (em0) is my physical device and all VLAN are added to this interface:

Code: [Select]
10_DMZ (em0_vlan10) -> v4: 172.16.10.254/24
                    v6/t6: 2003:f2:63c9:63e1:4c1f:32ff:fe6d:4ae/64
 20_VPN (em0_vlan20) -> v4: 172.16.20.254/24
 30_Pentest (em0_vlan30) -> v4: 172.16.30.254/24
                    v6/t6: 2003:f2:63c9:63e3:4c1f:32ff:fe6d:4ae/64
 40_WifiGuest (em0_vlan40) -> v4: 172.16.40.254/24
                    v6/t6: 2003:f2:63c9:63e4:4c1f:32ff:fe6d:4ae/64
 50_IoT (em0_vlan50) -> v4: 172.16.50.254/24
                    v6/t6: 2003:f2:63c9:63e5:4c1f:32ff:fe6d:4ae/64
 60_Dev (em0_vlan60) -> v4: 172.16.60.254/24
                    v6/t6: 2003:f2:63c9:63e6:4c1f:32ff:fe6d:4ae/64
 70_WiFi (em0_vlan70) -> v4: 172.16.70.254/24
                    v6/t6: 2003:f2:63c9:63e7:4c1f:32ff:fe6d:4ae/64
 80_Server (em0_vlan80) -> v4: 172.16.80.254/24
                    v6/t6: 2003:f2:63c9:63e8:4c1f:32ff:fe6d:4ae/64
 90_Clients (em0_vlan90) -> v4: 172.16.90.254/24
                    v6/t6: 2003:f2:63c9:63e9:4c1f:32ff:fe6d:4ae/64
 LAN (em0)       -> v4: 172.16.17.254/24
                    v6/t6: 2003:f2:63c9:63e0:4c1f:32ff:fe6d:4ae/64
 PIA_VPN (ovpnc1) -> v4: 10.56.10.6/32
 WAN (igb0)      -> v4: 192.168.217.2/24
                    v6/DHCP6: 2003:f2:63c9:6300:6eb3:11ff:fe1b:aedf/64


I´m on Sensei 0.8.0.beta4 and OPNsense 19.4.1

Do you need some more informations ?
Thank you!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 19, 2019, 09:41:29 pm
Hi BeNe,

We're aware of this issue. There's another Sensei deployment exactly the same setting with yours and experiencing the same problem.

Looks like something weird with em-vlan-netmap trio. We're on this. Will update the thread when it's done.

One question: are you fine when you remove the trunk interface and just protect vlan child interfaces?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on March 19, 2019, 10:21:31 pm
Hi mb,

thanks for that fast information.

Yes, if i remove the trunk Interface (LAN em0 in my case) from the protected interfaces list, the machines inside the VLAN 's are reachable again.

Gesendet von meinem Pixel 2 mit Tapatalk

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 20, 2019, 06:12:34 pm
Hi Bene,

All welcome. Thanks for the information. Can I ask a favor? Can you try the new netmap kernel to see if your current setup works? (child interfaces protected, trunk not protected).

Here's how to do it:

https://forum.opnsense.org/index.php?topic=11477.msg55261#msg55261


Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on March 20, 2019, 08:09:48 pm
Hello Murat,

of course  ;) But the problem is still the same. I installed the new Kernel:
Code: [Select]
# uname -a
FreeBSD surtur.my-network.de 11.2-RELEASE-p9-HBSD FreeBSD 11.2-RELEASE-p9-HBSD  4ea457eb7b8(master)  amd64
If i add "LAN (em0)" to the protected interfaces, the VLAN´s are offline.
So revert back to the stock kernel. Added a screenshot from my OPNsense Console after adding the interface.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 21, 2019, 08:57:19 pm
Hi Bene,

Messages in the screenshot are ok: netmap telling you it was able to open the ethernet port.

I can confirm that there's something weird with the trunk interface when we bridge hw <-> sw rings. After a while packet transmission stalls for the child interfaces:

Code: [Select]
658.955704 [2909] netmap_transmit           igb3 from_host, drop packet size 541392904 > 2048
683.531482 [2909] netmap_transmit           igb3 from_host, drop packet size 541392904 > 2048

Looking into that.

For now our advise is - if you're using VLANs -:


Our plan is to be able to process the trunk interface directly and for all VLANs and you'll not need to separately select child interfaces. Will get you updated on this.

For now, if you can carve out the untagged traffic from the trunk port, you're ok.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 23, 2019, 01:05:46 am
Dear Sensei users,

An update on broken Elasticsearch indices:

After digging together with users who have reported the issue, it looks like the indices were broken because some index file integrity got broken.

This is usually because of abrupt shutdown of the firewall. If power goes off suddenly, before Elastic does a full write of its in-memory buffers, than we have a broken index.

So, not to experience this issue try to turn off your system gracefully.

If in any case this happens, Sensei 0.8.0.beta6 has a "Fix Elastic indices" button under Sensei -> Configuration -> Reporting & Data menu. Just click on the button and Sensei will reset only the broken indices.

0.8.0.beta6 is available for update for 0.8 users.

0.8 looks stable enough to offer as an update for existing 0.7 installations. If we do not see any outstanding issues, we'll move 0.8 to the general repo in a few days.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on March 23, 2019, 02:21:56 am
MB,

I'm using dhcpv6 with track interface. Anytime Sensei starts after a reboot or an upgrade my ipv6 stops working until I do a release and renew of the entire WAN interface. It just did it to me again on the beta 6 upgrade.


Code: [Select]
Mar 22 18:13:25 opnsense: /usr/local/etc/rc.newwanipv6: Dynamic DNS: updatedns() starting
Mar 22 18:13:25 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Mar 22 18:13:25 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
Mar 22 18:13:25 opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Mar 22 18:13:25 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
Mar 22 18:13:25 opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt4'
Mar 22 18:13:25 opnsense: /usr/local/etc/rc.newwanip: On (IP address: X.X.X.X) (interface: XXXXX[opt4]) (real interface: ovpnc2).
Mar 22 18:13:25 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ovpnc2'
Mar 22 18:13:25 kernel: ovpnc2: link state changed to UP
Mar 22 18:13:24 opnsense: /usr/local/etc/rc.newwanipv6: Dynamic DNS: (Success) X.X.X updated to X.X.X.X
Mar 22 18:13:24 opnsense: /usr/local/etc/rc.newwanipv6: Dynamic DNS: updating cache file /var/cache/dyndns_wan_X.X.X_0.cache: X.X.X.X
Mar 22 18:13:21 kernel: ovpnc2: link state changed to DOWN
Mar 22 18:13:21 opnsense: /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.

Code: [Select]
Mar 22 18:15:55 dhcp6c: dhcp6c REQUEST on igb0 - running newipv6
Mar 22 18:15:55 dhcp6c[89888]: add an address 2605:X:X:36:1de7:22c5:7284:90a5/128 on igb0
Mar 22 18:15:55 dhcp6c[89888]: add an address 2605:X:X:a900:4262:31ff:fe00:7873/64 on igb1
Mar 22 18:15:55 dhcp6c[89888]: add an address 2605:X:X:a9ec:4262:31ff:fe00:7874/64 on igb2_vlan55
Mar 22 18:15:55 dhcp6c[89888]: add an address 2605:X:X:a9ef:4262:31ff:fe00:7874/64 on igb2_vlan200
Mar 22 18:15:55 dhcp6c[89888]: Received REPLY for REQUEST
Mar 22 18:15:55 dhcp6c[89888]: Sending Request
Mar 22 18:15:55 dhcp6c[89888]: Sending Solicit
Mar 22 18:15:54 opnsense: /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route
Mar 22 18:15:54 opnsense: /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route
Mar 22 18:15:54 opnsense: /usr/local/etc/rc.linkup: ROUTING: no IPv6 default gateway set, assuming wan
Mar 22 18:15:54 opnsense: /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan
Mar 22 18:15:54 opnsense: /usr/local/etc/rc.linkup: ROUTING: entering configure using 'lan'
Mar 22 18:15:54 dhcp6c[89888]: failed to remove an address on igb1: Can't assign requested address
Mar 22 18:15:54 dhcp6c[89888]: remove an address 2605:X:X:a9ec:X:31ff:fe00:7874/64 on igb2_vlan55
Mar 22 18:15:54 dhcp6c[89888]: remove an address 2605:X:X:a9ef:X:31ff:fe00:7874/64 on igb2_vlan200
Mar 22 18:15:54 dhcp6c[89888]: Sending Release
Mar 22 18:15:54 dhcp6c[89888]: Start address release
Mar 22 18:15:54 dhcp6c[89888]: remove an address 2605:X:X:X:1de7:22c5:7284:90a5/128 on igb0
Mar 22 18:15:54 dhcp6c[89888]: Sending Release
Mar 22 18:15:54 dhcp6c[89888]: Start address release
Mar 22 18:15:54 dhcp6c[89888]: restarting
Mar 22 18:15:54 opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface lan
Mar 22 18:15:54 opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for lan
Mar 22 18:15:54 kernel: igb1: link state changed to UP
Mar 22 18:15:50 opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for lan
Mar 22 18:15:50 eastpect[42809]: nm2::igb1^: permanently promiscuous mode enabled
Mar 22 18:15:50 eastpect[42809]: nm1::igb1:1: permanently promiscuous mode enabled
Mar 22 18:15:50 kernel: 750.076995 [2219] netmap_ioctl got 10000 extra buffers
Mar 22 18:15:50 kernel: 750.069849 [ 736] netmap_extra_alloc allocate buffer 24583 -> 24582
Mar 22 18:15:50 kernel: 750.062915 [ 736] netmap_extra_alloc allocate buffer 24582 -> 24581
Mar 22 18:15:50 kernel: 750.055985 [ 736] netmap_extra_alloc allocate buffer 24581 -> 24580
Mar 22 18:15:50 eastpect[42809]: nm0::igb1:0: permanently promiscuous mode enabled
Mar 22 18:15:50 kernel: 750.049074 [ 736] netmap_extra_alloc allocate buffer 24580 -> 24579
Mar 22 18:15:50 kernel: 750.042410 [ 736] netmap_extra_alloc allocate buffer 24579 -> 0
Mar 22 18:15:50 sshlockout[10974]: sshlockout/webConfigurator v3.0 starting up
Mar 22 18:15:50 kernel: 750.035617 [2216] netmap_ioctl requested 10000 extra buffers
Mar 22 18:15:50 kernel: igb1: link state changed to DOWN
Mar 22 18:14:06 dhcp6c[89888]: no responses were received
Mar 22 18:14:06 dhcp6c[89888]: no responses were received
Mar 22 18:14:04 dhcp6c[89888]: no responses were received
Mar 22 18:14:03 dhcp6c[89888]: no responses were received
Mar 22 18:13:49 dhcp6c[89888]: Sending Release
Mar 22 18:13:49 dhcp6c[89888]: Sending Release
Mar 22 18:13:48 dhcp6c[89888]: Sending Release
Mar 22 18:13:48 dhcp6c[89888]: Sending Release
Mar 22 18:13:41 dhcp6c[89888]: Sending Release
Mar 22 18:13:41 dhcp6c[89888]: Sending Release
Mar 22 18:13:40 dhcp6c[89888]: Sending Release
Mar 22 18:13:40 dhcp6c[89888]: Sending Release
Mar 22 18:13:37 dhcp6c[89888]: Sending Release
Mar 22 18:13:37 dhcp6c[89888]: Sending Release
Mar 22 18:13:37 dhcp6c[89888]: Sending Release
Mar 22 18:13:37 dhcp6c[89888]: Sending Release
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 23, 2019, 05:12:06 am
Hi donatom3,

Thanks for reporting this. Having a look now.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on March 24, 2019, 01:23:42 am
MB,

Another issue I've been having is the pan interface randomly disconnecting completely and I have to reboot to ping the interface again.

This is something that started since opnsense 19.1 for me. It happened on sensei 7.0 as well.

It happened on my old hardware and new. Both bare metal installs with Intel nics using the igb drivers. I can't find anything meaningful in the logs.

Im using the stock kernel now. Not sure if the test kernel will help with this lockup of the interface.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 24, 2019, 04:39:02 pm
Hi donatom3,

Thanks for reporting the issue in detail. I'll reach out to you to investigate further together.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: astoklas on March 25, 2019, 04:48:53 pm
Dear Sensei users,

An update on broken Elasticsearch indices:

After digging together with users who have reported the issue, it looks like the indices were broken because some index file integrity got broken.

This is usually because of abrupt shutdown of the firewall. If power goes off suddenly, before Elastic does a full write of its in-memory buffers, than we have a broken index.

So, not to experience this issue try to turn off your system gracefully.

If in any case this happens, Sensei 0.8.0.beta6 has a "Fix Elastic indices" button under Sensei -> Configuration -> Reporting & Data menu. Just click on the button and Sensei will reset only the broken indices.

0.8.0.beta6 is available for update for 0.8 users.

0.8 looks stable enough to offer as an update for existing 0.7 installations. If we do not see any outstanding issues, we'll move 0.8 to the general repo in a few days.

I just had a power outage on my opnsense, after the reboot the reports could not be displayed. The "Fix Indices" shows all good, but the report still does not show up. I still have the system in a "broken" state if you want to investigate further...

OpnSense 19.1.4
Sensei 0.8beta6
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 25, 2019, 05:07:08 pm
astoklas,

Thanks for the report. Reaching out to you now.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on March 26, 2019, 05:04:01 pm
Hello Murat,

is there an option to sync/export the collected data to another ELK Stack ?

Background:
I´m already running a ELK Stack in my network and i want to add the Sensei Data to it,too.
Sensei has much more information than the default syslog infos from OPNSense.

Benefit:
- long time archive
- own correlations searchs with other logs from the network/apps/devices
- build own dashboards and searches
- faster results than on the firewall itself

Thanks
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on March 27, 2019, 09:53:49 pm
Hi!

I use Sensei in couple of opnsense system. Works well so far.
I was wondering is there any way to run in a low memory board?
I have a pcengine APU2 board with 2GB memory, but i have a fast V-NAND msata SSD.
I setup 8GB swap file on the opnsense so i have 2GB physical and 8GB swap. The access speed not much differ since the SSD is very fast.
Im removed the memory checking row from the installation script so sensei installed succesfully.
I can configure too, it warns me the physical ram is low but i can continue.
However when i try to start the engine it says: Sensei detected swap usage is too high
And its stopped. Yes i know the swap usage is high but i dont think it can cause any issue since i use the fast ssd. Is there any way to override this? Let sensei use the swap file, i take the risk.

Thanks!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on March 28, 2019, 07:22:20 am
SunnyValley evaluating lightweight backend database engines to provide a lighter version for home users with low spec hardware. When they are ready, there will not be a need of such swap tricks...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 29, 2019, 02:12:04 am
Hello Murat,

is there an option to sync/export the collected data to another ELK Stack ?

Background:
I´m already running a ELK Stack in my network and i want to add the Sensei Data to it,too.
Sensei has much more information than the default syslog infos from OPNSense.


Hi BeNe,

Many thanks for your suggestion. This feature - along with syslog and netflow streaming - is in the roadmap.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on March 29, 2019, 02:23:18 am
SunnyValley evaluating lightweight backend database engines to provide a lighter version for home users with low spec hardware. When they are ready, there will not be a need of such swap tricks...

Hi Archanfel80,

As Antaris recommends, you might think of waiting for the alternative db backend work.

Sensei uses in-memory caching so I would worry that swap usage might degrade your system performance bad -- even if you are using SSD.

Still, if you want to go for it, Disable Health Check from Sensei: Configuration: Updates & Support, and you're all set.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on March 29, 2019, 05:35:52 pm
Thank You!
Both of you :)
I probably wait for the light version but i give it a try for the ssd swap just for testing. Its a low bandwidth system, just a few users, it might will be no problem. If yes we know its no good :)
Regards, Peter
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mdurkin on March 30, 2019, 09:05:01 am
Anyone having problems blocking YouTube using 0.8.0.beta7? I used app control but it has no effect. Other controls seem to work fine. It's a shame as its the reason I installed was to try this out!
Anyone else tried blocking YouTube?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mayo on March 30, 2019, 12:32:18 pm
Archanfel80 could you please make a step by step guide? I will try Sensei on my apu2c4...

thank you!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on March 31, 2019, 11:10:35 pm
Archanfel80 could you please make a step by step guide? I will try Sensei on my apu2c4...

thank you!

Hi!

On a 2c4 which have 4GB ram enough to use the default 2GB swap file. Just enable in the system-miscellaneous.
Make sure you have limited Sensei to 100 user maximum, and you have no problem.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mayo on April 01, 2019, 12:16:16 pm
Archanfel80 could you please make a step by step guide? I will try Sensei on my apu2c4...

thank you!

Hi!

On a 2c4 which have 4GB ram enough to use the default 2GB swap file. Just enable in the system-miscellaneous.
Make sure you have limited Sensei to 100 user maximum, and you have no problem.

Thank you so much! Will try in the afternoon!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on April 01, 2019, 06:23:07 pm
In version 0.8 beta 7 on netmap kernel i experience tremendous slowdown in DNS resolving and packet loss to internet resources.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: ict-guy on April 02, 2019, 11:05:24 am
i have the same problem for over a week now, at the moment i'm using sensei in xlarge mode and have set dhcp lease time for 8 hour default and 10 hour max.

this seems to help stablilize the occurends
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on April 03, 2019, 07:55:42 pm
What common on earth have DHCP lease time with packet loss ?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: SchylgeICT on April 03, 2019, 09:03:14 pm
I just started testing and noticed the slowdown. In my case disabling cloud threat intel solved this.
maybe this helps.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on April 03, 2019, 09:08:10 pm
I just started testing and noticed the slowdown. In my case disabling cloud threat intel solved this.
maybe this helps.

I can confirm that, cloud threat intel cause noticable delay in the dns query. Its seems the cloud servers not stable enough, since i see packet loss. In a workaround use the opnsense builtin intrusion detection with ET Pro telemetry (can be installed as a plugin). Its free if you let your firewall send anonymous statistics (why not?).
Other than that sensei is an amazing product!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on April 04, 2019, 12:10:02 am
I just started testing and noticed the slowdown. In my case disabling cloud threat intel solved this.
maybe this helps.

I can confirm that, cloud threat intel cause noticable delay in the dns query. Its seems the cloud servers not stable enough, since i see packet loss. In a workaround use the opnsense builtin intrusion detection with ET Pro telemetry (can be installed as a plugin). Its free if you let your firewall send anonymous statistics (why not?).
Other than that sensei is an amazing product!

I can confirm too ;) We'll be shipping 0.8.0.beta8 tomorrow. It has several fixes which we expect to address this issue.

Plus, it has tagged (trunk) vlan interface support :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on April 04, 2019, 12:12:46 am
Anyone having problems blocking YouTube using 0.8.0.beta7? I used app control but it has no effect. Other controls seem to work fine. It's a shame as its the reason I installed was to try this out!
Anyone else tried blocking YouTube?

Hi mdurkin,

Many thanks for reporting this. I checked with several deployments now. It looks like it's blocking. Let me contact you, there might be something in your environment which might trigger this.

 
Title: Re: Sensei on OPNsense - Application based filtering
Post by: OPNsense4ever on April 04, 2019, 04:00:20 am
Hello! 4 of my graphs are suddenly showing nothing. "Egress New Connections by App Over Time" and "Egress New Connections by Source Over Time" say "No Egress New Connection." "New Connections & Unique Remote Hosts" says "No New Connection & Unique Remote Host" and "Unique Local Hosts over Time" says "No Local Host." I just updated to 0.8.0.beta7 as well as stopping and starting the Sensei Packet Engine and Elasticsearch services. Any thoughts on what might have gone wrong or how to fix it?

Thanks!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on April 04, 2019, 07:02:19 am
Hi OPNsense4ever,

Many thanks for trying Sensei & reporting the issue.

We changed a field type in Elasticsearch. New query format is not compatible with the data type in old indexes. This is why you cannot see any data with those "histogram"s.

When you have some activity over time, they'll get back to normal, at most in a couple of days.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on April 05, 2019, 02:57:41 pm
Dear Sensei users,

We've shipped 0.8.0.beta8 yesterday. This update brings vlan tagged interface support and fixes several issues with beta7. All beta7 users are encouraged to update to beta8.

With regard to Cloud infrastructure, we decided to take following steps to improve the availability:

1. Independent cloud queries:

Currently we're utilizing DNS infrastructure to communicate with our Cloud backend systems. Since we're redirecting dns traffic, this means for the cloud systems, we have to also act like a DNS recursive server. On the recursion side, since this is not within the scope of Sensei project, we cannot always guarantee the best DNS response time.

This is why, starting with 0.8.0.beta9, we'll be doing the cloud threat intelligence lookups with an independent to-the-purpose query. 

2. New cloud servers for US-West, US-East and Asia.

To improve cloud response time and distributing load, we'll be introducing new servers for Asia, US-West and US-East regions.

This change will have the following benefits:

1. Improved the availability
2. Improved response times (from avg 100ms to as low as 5ms)
3. You'll be able to continue using your local DNS servers.
4. You'll be able utilize other DNS based solutions (like Pi-hole) - in conjunction -  with Sensei.

We plan to have this before 0.8 rc1 so, hopefully we'll ship this with beta9 in two weeks.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on April 05, 2019, 06:25:57 pm
Hi!

Just a curious question. Did you consider using Apache Lucene as the db backend instead of Elasticsearch?
I use lucene in several projects (mostly bitnami) and its a very scalable and fast backend. There is an option to use as a "lightweight" scenario and also like as an "enterprise". It may solve the low memory hw problem.
Im just thinkin loudly :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on April 06, 2019, 03:15:29 pm
Hi Archanfel80,

Many thanks for the suggestion. Actually didn't consider this as an option - wasn't aware that lucene had a lightweight option.

Currently we're evaluating Timescaledb and Influxdb. We'll also have a look at lucene lightweight option. Any pointers on this for me?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on April 07, 2019, 09:21:15 am
Hi!

I mostly played with heap sizes and buffer sizes. Lower values results lower memory usage in the cost of performance (slower queries) because the increased disk IO.
TimescaleDB is a good choice too. Im not sure about the Influxdb, i had to use it in the past but cause too much headache. Its not easy to operate.
Elasticsearch memory consumption also can limited. If i use in a low users <100 scenario and does not store more than 3 days data, the whole system memory usage is below 2GB. I run sensei in a 2GB board for almost a week now, small office 8 user only stored 3 days. The boss just want to see what the workers do so he check sensei reports in the end of the day. The whole system memory consumption is below 2GB. I use the default 2GB swap in opnsense but not a single byte used on that. I had to disable the sensei health check because its stopped the engine from time to time, but no issues so far. Also i have a bigger system, college with students, much more user much more data, stored 3 days history, the memory is just a bit above 4GB. I think the 8GB minimum recommended ram is a bit high. I dont have any system what eat this much.

What if sensei will detect the available system memory with the optional swap file too and gray out the big scenarios like 500 user and limit the maximum data history time limit, etc. So the user cant use a big scenario what break down the system?
For example with 2GB system, 25 users max, 3 days history
4GB system 100 users max, 7 days history
etc. And you can limit elasticsearch memory usage too.

And a quick report, after the beta8 the cloud threat query time a bit better but still cause delay what the user noticed.

Keep up the good work :)

Hi Archanfel80,

Many thanks for the suggestion. Actually didn't consider this as an option - wasn't aware that lucene had a lightweight option.

Currently we're evaluating Timescaledb and Influxdb. We'll also have a look at lucene lightweight option. Any pointers on this for me?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on April 08, 2019, 06:48:31 am
Hi Archanfel80,

Many thanks for sharing your experience. Indeed, we found this very helpful.

Now I'm thinking we might be over optimizing. We were trying to keep the memory usage for the Sensei and DB below 1GB for small deployments, like 25 users. And also we are trying to provide at least a month of history.

If the median minimal RAM size for OPNsense small deployments are 2GB, your suggestion looks very viable.

Let's do a quick twitter poll:

https://twitter.com/sunnyvalley/status/1115109250479476737

With regard to beta8, glad to hear that it looks better. We've received similar feedback from several other users. Hopefully, we will be solving the remaining issue with Cloud with beta9.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on April 08, 2019, 09:12:03 am
Hi!

I think keep the ram usage below 1GB would be a bit hard.
This is my smallest scenario, very low activity, sensei active only in one IF, around 8-10 users.

https://imgur.com/a/t8Bk8qg

This is a VM actually, the ram usage is below 2GB, but higher than 1GB. I cant keep below that. Of course this is the OS+Sensei RAM usage together. OPNSense eat 300-800MB RAM depending on scenario, so the 2GB usage with sensei means sensei use 1-1.5GB RAM with a low end settings.
A 2GB board should handle this, even with a swap file.
I think you can try to reach the ~1GB ram usage for a small scenario, that should be satisfy the low end HW users :)

Hi Archanfel80,

Many thanks for sharing your experience. Indeed, we found this very helpful.

Now I'm thinking we might be over optimizing. We were trying to keep the memory usage for the Sensei and DB below 1GB for small deployments, like 25 users. And also we are trying to provide at least a month of history.

If the median minimal RAM size for OPNsense small deployments are 2GB, your suggestion looks very viable.

Let's do a quick twitter poll:

https://twitter.com/sunnyvalley/status/1115109250479476737

With regard to beta8, glad to hear that it looks better. We've received similar feedback from several other users. Hopefully, we will be solving the remaining issue with Cloud with beta9.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: SchylgeICT on April 09, 2019, 09:18:48 pm
Hi MB,

With beta7 i was able to add OPT1(vlan interface) to the protected interfaces. I can still do this with beta 8. What did actually change with
Quote
"We've shipped 0.8.0.beta8 yesterday. This update brings vlan tagged interface support and fixes several issues with beta7. All beta7 users are encouraged to update to beta8."
. I think I'm overlooking something.
It would be nice to add different Sensei rules for different Vlans. I was hoping this was one of the things that changed in beta8.
I can confirm faster DNS lookups now with cloud threat intel enabled!
Best regards.
Ruud

Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on April 11, 2019, 09:47:13 am
Yeah, different rules on different interfaces would be a great feature, as also a scheduling function.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: opnsenseuser on April 14, 2019, 12:28:43 pm
a nice feature would be synonymous if you could install the plugin as standalone on an external bsd or linux computer and could use the plugin as an analyzer.

The firewall could be relieved. especially in the home user area an advantage but certainly also in the business area a welcome feature.

is there any news on the topic sensei for low power hardware optimization?

Thank you

Regards, rene
Title: Re: Sensei on OPNsense - Application based filtering
Post by: thg0432 on April 15, 2019, 08:27:42 pm
Hi,

Is it possible to have parental controls or per device/group filtering?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: rb_newbie on April 18, 2019, 09:49:44 pm
Just installed 0.8.0.beta8 and did an audit of the packages and found that an outdated library is being used that is vulnerable.  Any way I can manually update this w/o breaking anything or will it be fixed in the stable release?

***GOT REQUEST TO AUDIT SECURITY***
vulnxml file up-to-date
libXdmcp-1.1.2_2 is vulnerable:
libXdmcp -- insufficient entropy generating session keys
CVE: CVE-2017-2625
WWW: https://vuxml.FreeBSD.org/freebsd/1b6a10e9-4b7b-11e9-9e89-54e1ad3d6335.html

1 problem(s) in the installed packages found.
***DONE***
Title: Re: Sensei on OPNsense - Application based filtering
Post by: timota on April 22, 2019, 09:30:27 pm
Im keen to check your plugin, but installer complains on

"Unfortunately Celeron is not supported by Sensei."

i cant say that my CPU is weak, it peforms good on most of tasks.

What will happened if i remove this check from installer ? do you have any other cheks that will prevent to install it ?

Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on April 23, 2019, 02:14:23 pm
Yes! If you have less than 4GB ram the installer will also fail. You can remove this check too. The ram is not problem, i have sensei with 2GB apu board without problem, but that board have a quad core intel processor, and the cpu usage is kinda heavy. Im not sure the celeron processor can handle this.

Im keen to check your plugin, but installer complains on

"Unfortunately Celeron is not supported by Sensei."

i cant say that my CPU is weak, it peforms good on most of tasks.

What will happened if i remove this check from installer ? do you have any other cheks that will prevent to install it ?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: timota on April 24, 2019, 04:45:47 pm
great thanks.

will try anyway.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: the-mk on May 09, 2019, 06:17:52 pm
Hi,

is anyone using the scheduled reports in reports&data section of the configuration (Sensei 0.7)?
Is it just me or is the "click to download and view detailed reports" link not working within the email (getting a blank html file with 0 bytes)?
It does not work for me in MS Outlook (Office 365) and Thunderbird.
If I access that mails through the webmail of my GMX (my mail provider) I can see that there's a html file attached. I can see the attachment in the app on my iPhone, but that's not my favorite "view that report" device.

Are there any updates on Sensei 0.8? since that thread fell asleep ;)

Thanks!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: the-mk on May 09, 2019, 06:35:18 pm
@mb: is there an update about this issue?
I ran into this issue on my physical test hardware with only two ethernet ports, where one is running several VLANs. After reading your post I removed the parent adapter from the protected interfaces and it is working again. But that was on Sensei 0.8.0.beta8...
Thanks!

Hi Bene,

Messages in the screenshot are ok: netmap telling you it was able to open the ethernet port.

I can confirm that there's something weird with the trunk interface when we bridge hw <-> sw rings. After a while packet transmission stalls for the child interfaces:

Code: [Select]
658.955704 [2909] netmap_transmit           igb3 from_host, drop packet size 541392904 > 2048
683.531482 [2909] netmap_transmit           igb3 from_host, drop packet size 541392904 > 2048

Looking into that.

For now our advise is - if you're using VLANs -:

  • Stay with the stock kernel which comes default with the OPNsense release, we need more work in new kernel with regard to VLANs
  • Do not put any untagged traffic to your VLAN trunk port and you should be able to protect vlan child interfaces just fine

Our plan is to be able to process the trunk interface directly and for all VLANs and you'll not need to separately select child interfaces. Will get you updated on this.

For now, if you can carve out the untagged traffic from the trunk port, you're ok.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: malac on May 11, 2019, 06:32:35 pm
Hi @donatom3,

For application control, dns does not play any role there, so you'll be utilizing Sensei at its full potential in any ways.

For security & web filtering, yes, you'll lose some data there, provided that you do not enable Web Reputation & Cloud Threat Intel, which requires DNS override.

If your DNS transactions are traversing through an interface which is protected by Sensei & you have Cloud enabled, dns override will be in place (like unbound). Because we are way earlier in the process, we'll also override unbound.

That being said, if you place the DC in the same broadcast domain with your clients, they'll be utilizing your DC (this way they do not traverse through Sensei to reach the DNS server). -- Local DNS server will still be subject to the same behavior if its traffic passes through Sensei. -- Here you have your local queries untouched.

We've been asked (by some more users) of a possibility of using both Sensei Cloud database & local dns servers. We're evaluating methods to do it. Most probably we'll be back into this in the second or third quarter of 2019.

Very happy to hear that you've attained gigabit speeds and happy with the software ;)


Would be great if i could use Cloud database & local dns!

Do you have a pricing idea for premium edition for home user?

thx
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 11, 2019, 07:43:48 pm
Dear Sensei users,

An update on the low-resource systems:

Below is the results of the poll "How much memory do you have on your OPNsense firewall"

Many thanks to those who attended the poll. According to the results, 2/3 of the OPNsense users have either 4GB or more memory.

So, as per Archanfel80's suggestion, enabling for 4GB will allow another 40% to be able to start using Sensei. We thought that this is a huge number and lowered the minimum memory requirement to 4GB (Elastic is configured accordingly).

So, practically, if you have 4GB RAM, than starting with beta9 (coming this weekend), you'll be able to enjoy Sensei for up to 100 users.

I'd like to thank Archanfel80 for his awesome suggestion. It's in the works now.

Alternative database backend work (which will enable Sensei for 2GB or less memory) is continuing, but might take a little longer than we originally planned -- most probably post 2019. (due to other high priority work).

Note: I see that we missed some messages unanswered here. Apologies for that: we're recovering quite a loaded timeframe, and will be getting back to you shortly.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 11, 2019, 07:54:34 pm
a nice feature would be synonymous if you could install the plugin as standalone on an external bsd or linux computer and could use the plugin as an analyzer.

The firewall could be relieved. especially in the home user area an advantage but certainly also in the business area a welcome feature.

Yes, we have some good news about this. Part of our overload was due to this feature actually. With 0.8.0.beta9 (coming this weekend), you'll notice in Configuration page that we have introduced another deployment option:

L2 transparent bridge.

In this mode, Sensei literally bridges two of your ethernet interfaces.

This way, we can scale to the number of Rx/Tx ethernet queues, thus making maximum use of the multiple CPU cores in the system.

We introduced this to be able to support sites which have thousands of users.

This also helps you to keep your existing firewall and still enjoy the functionality offered by OPNsense & Sensei as an additional layer of defense.

A live deployment for 5000 users was done; and looks quite promising.

is there any news on the topic sensei for low power hardware optimization?

Yep, please see my above answer: https://forum.opnsense.org/index.php?topic=9521.msg58741#msg58741
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 11, 2019, 07:59:34 pm
Would be great if i could use Cloud database & local dns!

Yes, it's implemented :) Look for 0.8.0 beta9, coming up this weekend.

Do you have a pricing idea for premium edition for home user?

Pricing and premium subscription plan details are almost complete. Hope to announce it very soon.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: malac on May 11, 2019, 08:07:27 pm
Would be great if i could use Cloud database & local dns!

Yes, it's implemented :) Look for 0.8.0 beta9, coming up this weekend.

Do you have a pricing idea for premium edition for home user?

Pricing and premium subscription plan details are almost complete. Hope to announce it very soon.

GREAT!!! looking forward...THX
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 11, 2019, 08:10:15 pm
@mb: is there an update about this issue?
I ran into this issue on my physical test hardware with only two ethernet ports, where one is running several VLANs. After reading your post I removed the parent adapter from the protected interfaces and it is working again. But that was on Sensei 0.8.0.beta8...
Thanks!

Hi @the-mk,

Yes, we have updates on this. Sensei is is now able to process VLAN trunk interfaces.

So, if you're using VLANs -, the latest advise is:

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 11, 2019, 08:15:30 pm
Just installed 0.8.0.beta8 and did an audit of the packages and found that an outdated library is being used that is vulnerable.  Any way I can manually update this w/o breaking anything or will it be fixed in the stable release?

libXdmcp-1.1.2_2 is vulnerable:
libXdmcp -- insufficient entropy generating session keys
CVE: CVE-2017-2625
WWW: https://vuxml.FreeBSD.org/freebsd/1b6a10e9-4b7b-11e9-9e89-54e1ad3d6335.html

Hi rb_newbie, many thanks for pointing this out. This is a dependency package required by Elasticsearch/java. We'll go ahead and update it.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on May 11, 2019, 08:19:07 pm
@mb: is there an update about this issue?
I ran into this issue on my physical test hardware with only two ethernet ports, where one is running several VLANs. After reading your post I removed the parent adapter from the protected interfaces and it is working again. But that was on Sensei 0.8.0.beta8...
Thanks!

Hi @the-mk,

Yes, we have updates on this. Sensei is is now able to process VLAN trunk interfaces.

So, if you're using VLANs -, the latest advise is:

  • Stay with the stock kernel which comes default with the OPNsense release, we need more work in new kernel with regard to netmap
  • You can now protect untagged (trunk) vlan interfaces. Sensei will process both tagged and untagged frames at the same time. This is the advised & performant method.
  • Or, you can still choose to protect vlan child interfaces or vlan parent interfaces. The important thing to be careful here is do not have them at the same time, or you'll hit a bug present in current netmap code
MB,

Are you saying if I move my 2 vlans off their own interface back to my main trunk I should stop seeing that netmap crash that was causing sensei to stop all traffic?

Sent from my Pixel 3 XL using Tapatalk

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 11, 2019, 08:59:12 pm
Hi Ruud,

With beta7 i was able to add OPT1(vlan interface) to the protected interfaces. I can still do this with beta 8. What did actually change with beta8? I think I'm overlooking something.

Correct. The difference is; beta7 did not actually process tagged frames, they were just forwarded; whereas beta8 does process both tagged and untagged frames.

It would be nice to add different Sensei rules for different Vlans. I was hoping this was one of the things that changed in beta8.

We're addressing this with Policy based filtering (Interface, VLAN, Subnet based policies) which will appear in Premium subscription.

I can confirm faster DNS lookups now with cloud threat intel enabled!

Many thanks for this update. 0.8.0.beta9 should be slightly better.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 11, 2019, 09:07:44 pm
is anyone using the scheduled reports in reports&data section of the configuration (Sensei 0.7)?
Is it just me or is the "click to download and view detailed reports" link not working within the email (getting a blank html file with 0 bytes)?
It does not work for me in MS Outlook (Office 365) and Thunderbird.
If I access that mails through the webmail of my GMX (my mail provider) I can see that there's a html file attached. I can see the attachment in the app on my iPhone, but that's not my favorite "view that report" device.

Hi @the-mk,

Gmail web/iPhone looking good. It looks like a problem embedding the report for Office365/Thunderbird,

Having a look at it. Many thanks for reporting.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: the-mk on May 12, 2019, 11:21:45 am
@mb - thanks!
tested adding the trunk interface only to the protected interfaces - and it processes all VLANs that are on that trunk interface - that's ok for me!
looking forward to beta9! I guess we get a notification here in the forums as soon as it is available?
scheduled reports - the embedded report problem also exists in 0.8 beta8...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 12, 2019, 05:53:00 pm
Hi @tk-mk,

Glad to hear that vlans are working for you. beta9 is reporting vlans & interfaces. Final tests are run for it & should arrive late today (PST) or tomorrow.

Got it. Not able to make the fix for beta9, hopefully with the next beta.
Title: Sensei on OPNsense - Application based filtering
Post by: shijo on May 13, 2019, 04:19:59 pm
Hi there,

Is there any possible way to block  Ultrasurf client proxy by using Sensei. Ultrasurf sets up a local proxy on the user’s computer, and then configures Internet Explorer’s proxy settings to run all Internet requests through that local proxy. The default port is 9666. Since the traffic between Ultrasurf and IE is entirely on the localhost, it never goes to the network and can’t be blocked by a firewall. Ultrasurf then sets up an encrypted connection with a remote server in its network of proxy servers. The connection to the remote proxy server is made over port 443. Hopefully someone out there can help me with this.

Thanks in advance !  :)  :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 13, 2019, 07:36:04 pm
Hi @shijo,

Thank you very much for trying out Sensei.

The pre-requisite for filtering an application is the identification of that application in the first place. Once its traffic is correctly identified, filtering is the easiest part.

It looks like we're not able to identify this traffic as Ultrasurf Proxy.

We've had requests for Ultrasurf and its identification is on the roadmap.

In the meantime, if you'd like to give that a pace, you can share pcap of a "test" ultrasurf session, that would be really helpful.

Then it'd be faster for us to write the signature for identifying the application.

And once it's identified, filtering is automatically in place.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 14, 2019, 12:57:45 am
Dear Sensei users,

0.8.0.beta9 is out now. Below are the updates against 0.8.0.beta8:

Support for Large Settings (More than 1000 users)

Thanks to the newly introduced L2 Transparent Bridge Mode, you can run Sensei for thousands of users.

In this mode, where Sensei literally bridges two of your ethernet interfaces, we can scale to the number of Rx/Tx ethernet queues, thus making maximum use of the multiple CPU cores in the system.

This also helps you to keep your existing firewall and still enjoy the functionality offered by OPNsense & Sensei as an additional layer of defense.

Practically, what this means is that, if you deploy Sensei on an 8-core server with a --say 64GB of memory, you can serve 8000 users behind this configuration.

Please be noted that we'll need a small integration with OPNsense to be able to fully provide this functionality. We'll keep you posted.

Support for 4GB RAM

In an effort to be able to provide Sensei for people who have less than 8GB memory, and as per Archanfel80's suggestion, we've enabled Sensei to run for deployments with 4B of RAM.

Please note that if you have 4GB memory, maximum number of users will be 100.

Improved application signatures


Cloud

New Cloud Query Infrastructure

Filtering

Fixed a bug where auto-whitelisting a host does not immediately take effect / requiring a restart of engine.

Integrations


Better Reporting



How to update?

For 0.8 users, in the OPNsense Web UI, you should have already seen Sensei reporting 0.8.0.beta9 update. Just click on "Update" and Sensei will take care of the rest.

For 0.7 users, please wait for an announcement for 0.8.0.rc1; when it's out, you should also see 0.8 update in the OPNsense UI. We'll announce it from here and our twitter page.

Hope you enjoy this one.

--
Sensei team
Title: Re: Sensei on OPNsense - Application based filtering
Post by: shijo on May 14, 2019, 12:46:17 pm
Hi @mb,

Thank you very much for the reply. As you suggested I'm attaching the pcap file for your reference.

Thanks in advance !
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 14, 2019, 01:58:45 pm
Hi @shijo,

That's awesome. Thank you. This'll help a lot.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on May 14, 2019, 02:53:39 pm
Im glad i can help :)

Dear Sensei users,

0.8.0.beta9 is out now. Below are the updates against 0.8.0.beta8:

Support for Large Settings (More than 1000 users)

Thanks to the newly introduced L2 Transparent Bridge Mode, you can run Sensei for thousands of users.

In this mode, where Sensei literally bridges two of your ethernet interfaces, we can scale to the number of Rx/Tx ethernet queues, thus making maximum use of the multiple CPU cores in the system.

This also helps you to keep your existing firewall and still enjoy the functionality offered by OPNsense & Sensei as an additional layer of defense.

Practically, what this means is that, if you deploy Sensei on an 8-core server with a --say 64GB of memory, you can serve 8000 users behind this configuration.

Please be noted that we'll need a small integration with OPNsense to be able to fully provide this functionality. We'll keep you posted.

Support for 4GB RAM

In an effort to be able to provide Sensei for people who have less than 8GB memory, and as per Archanfel80's suggestion, we've enabled Sensei to run for deployments with 4B of RAM.

Please note that if you have 4GB memory, maximum number of users will be 100.

Improved application signatures

  • Browsec VPN
  • Microsoft Updates
  • Office Updates
  • Fixed a bug in Web based applications classification module which -in some cases- might lead to a crash.

Cloud

New Cloud Query Infrastructure

Filtering

Fixed a bug where auto-whitelisting a host does not immediately take effect / requiring a restart of engine.

Integrations

  • Improved CLI access API
  • First bits of Active Directory Integration

Better Reporting

  • New report: Ethernet interface reports. You can now see which ethernet interfaces carry the most bandwidth and drill down to per-interface detailed reports.
  • New report: VLAN reports. You can filter out a VLAN and drill down as deep as session details.
  • New report: User reports. When the OPNsense captive integration is finished, you’ll be able to view user-based reports.
  • All live session reports now have VLAN, Interface, Username columns.
  • All live session reports now have auto-refresh / refresh interval options
  • Fixed a bug where charts were refreshed randomly causing excessive page loads
  • Fixed a bug where setting Elasticsearch not to start at boot causing reporting to cease.
  • Introduced an option to be able to reset all Elasticsearch Indexes.
  • Introduced Elasticsearch Index Health Checker, where you can check and do a fix-up on an index basis
  • Elasticsearch shards are now single. Not requiring a replica. All indexes can be seen green now.
  • Fixed a bug in Elasticsearch data retiring module, which -in some cases- would result in more disk space consumption


How to update?

For 0.8 users, in the OPNsense Web UI, you should have already seen Sensei reporting 0.8.0.beta9 update. Just click on "Update" and Sensei will take care of the rest.

For 0.7 users, please wait for an announcement for 0.8.0.rc1; when it's out, you should also see 0.8 update in the OPNsense UI. We'll announce it from here and our twitter page.

Hope you enjoy this one.

--
Sensei team
Title: Re: Sensei on OPNsense - Application based filtering
Post by: malac on May 14, 2019, 02:56:37 pm
Hi updated from beta8 to 9, everythings looks fine so far.
Also local DNS an Cloud Threat Intel is working, GREAT!

Only: I cannot set deployment size, drop down is empty....but thats it
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on May 14, 2019, 04:24:26 pm
Im glad i can help :)

How does it help to just quote the complete previous text without any sensful addition?  ::)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on May 14, 2019, 04:36:01 pm
I referred for this: "In an effort to be able to provide Sensei for people who have less than 8GB memory, and as per Archanfel80's suggestion, we've enabled Sensei to run for deployments with 4B of RAM."

Im glad i can help :)

How does it help to just quote the complete previous text without any sensful addition?  ::)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: ruffy91 on May 15, 2019, 09:38:45 am
I installed Sensei 0.8p9 on 19.1.6 (which I now updated to 19.1.7).
I get the following error when accessing the Dashboard or any sensei page:
Warning: fopen(/usr/local/sensei/log/active/Senseigui.log): failed to open stream: No such file or directory in /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php on line 73 Can't open log file at '/usr/local/sensei/log/active/Senseigui.log'

The folder /usr/local/sensei/log does not exist.

After manually creating /usr/local/sensei/log/active the plugin does seem to work.

The interface selection unfortunately does not show any tagged VLAN interfaces. Is this correct? I tought tagged VLANs are supported now?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on May 15, 2019, 09:52:15 am
Login to the firewall through SSH:
mkdir -p /usr/local/sensei/log/active
mkdir -p /usr/local/sensei/log/archive

reboot

I installed Sensei 0.8p9 on 19.1.6 (which I now updated to 19.1.7).
I get the following error when accessing the Dashboard or any sensei page:
Warning: fopen(/usr/local/sensei/log/active/Senseigui.log): failed to open stream: No such file or directory in /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php on line 73 Can't open log file at '/usr/local/sensei/log/active/Senseigui.log'

The folder /usr/local/sensei/log does not exist.

After manually creating /usr/local/sensei/log/active the plugin does seem to work.

The interface selection unfortunately does not show any tagged VLAN interfaces. Is this correct? I tought tagged VLANs are supported now?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 15, 2019, 02:04:01 pm
The folder /usr/local/sensei/log does not exist.

After manually creating /usr/local/sensei/log/active the plugin does seem to work.

The interface selection unfortunately does not show any tagged VLAN interfaces. Is this correct? I tought tagged VLANs are supported now?

Hi ruffy,

Having a look at log folder creation. Thanks for reporting this.

As for the VLAN tagged interface, any chances that you did not enable the trunk interface from OPNsense Interfaces menu?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on May 15, 2019, 02:40:48 pm
Im using tagged vlan interfaces and all shown correctly. See attached image.

The folder /usr/local/sensei/log does not exist.

After manually creating /usr/local/sensei/log/active the plugin does seem to work.

The interface selection unfortunately does not show any tagged VLAN interfaces. Is this correct? I tought tagged VLANs are supported now?

Hi ruffy,

Having a look at log folder creation. Thanks for reporting this.

As for the VLAN tagged interface, any chances that you did not enable the trunk interface from OPNsense Interfaces menu?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on May 15, 2019, 04:34:03 pm
Im using tagged vlan interfaces and all shown correctly. See attached image.

Yes, but you had these interfaces already active before you upgraded sensei. If you remove them, you will not be able to readd them again unless you edit the right file to disable the display filter.

mb:
Quote
[...] since we started supporting vlan trunk interfaces, we are filtering child interfaces now. Because netmap was causing problems when there are more than 2-3 vlan child interfaces monitored at the same time. [...]

You will have to edit /usr/local/opnsense/mvc/app/controllers/OPNsense/Sensei/Api/ToolsController.php
and change $filterflag = true; to $filterflag = false; in line #51 where is the comparision with 'vlan'.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on May 15, 2019, 06:01:53 pm
True!
I can confirm that, i dont see the vlan interfaces unless i add manually to the config.xml (Sensei section) or do the same what you mentioned.

Im using tagged vlan interfaces and all shown correctly. See attached image.

Yes, but you had these interfaces already active before you upgraded sensei. If you remove them, you will not be able to readd them again unless you edit the right file to disable the display filter.

mb:
Quote
[...] since we started supporting vlan trunk interfaces, we are filtering child interfaces now. Because netmap was causing problems when there are more than 2-3 vlan child interfaces monitored at the same time. [...]

You will have to edit /usr/local/opnsense/mvc/app/controllers/OPNsense/Sensei/Api/ToolsController.php
and change $filterflag = true; to $filterflag = false; in line #51 where is the comparision with 'vlan'.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: opnip on May 15, 2019, 10:57:37 pm
Cloud Node Status is always DOWN (see attachment). I can klick "Check Now" and after that, the status changes to "UP". But after a few seconds it goes back to "DOWN" and stays at is. Is this normal?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 16, 2019, 03:03:29 am
@opnip,

As a private message, can you share your firewall's IP address with me? Let's do a trace.

Hi updated from beta8 to 9, everythings looks fine so far.
Also local DNS an Cloud Threat Intel is working, GREAT!

Only: I cannot set deployment size, drop down is empty....but thats it

@holger, fixed for beta10.

I get the following error when accessing the Dashboard or any sensei page:
73 Can't open log file at '/usr/local/sensei/log/active/Senseigui.log'

@ruffy, fixed for beta10.


@Archanfel80, @hbc, @ruffy,

Please watch for beta10. We removed the filter for VLAN child interfaces.

So the latest situation:

You can either

- Add the parent/tagged ethernet interface and protect the whole tagged/untagged
   traffic passing through the interface

or

- Add each vlan child interface seperately to the protected interfaces. The thing
  to note here is do NOT add both the parent and the child interfaces at the same
  time, or you'll hit a netmap bug.

Any Sensei users who are using more than two VLAN child interfaces at the same
time? Any issues so far?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on May 16, 2019, 03:49:12 am



Any Sensei users who are using more than two VLAN child interfaces at the same
time? Any issues so far?

Ive got one parent and two vlans interfaces on the same trunk all working fine. Same issue as others where the vlan interfaces don't show up as selectable but just adding the parent gets all 3

Sent from my Pixel 3 XL using Tapatalk

Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on May 16, 2019, 03:50:13 am



Any Sensei users who are using more than two VLAN child interfaces at the same
time? Any issues so far?

Ive got one parent and two vlans interfaces on the same trunk all working fine. Same issue as others where the vlan interfaces don't show up as selectable but just adding the parent gets all 3

Sent from my Pixel 3 XL using Tapatalk
Just saw you said more than 2 I can add a third one just for fun.

Sent from my Pixel 3 XL using Tapatalk

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on May 16, 2019, 06:11:22 am
Hi MB, In App Control, we can block an entire protocol / type of service. Is there any way to block one user and allow everyone else OR allow one user and block rest in network either by IP or MAC address. Thanks
Title: Re: Sensei on OPNsense - Application based filtering
Post by: malac on May 16, 2019, 06:26:46 am
Cloud Node Status is always DOWN (see attachment). I can klick "Check Now" and after that, the status changes to "UP". But after a few seconds it goes back to "DOWN" and stays at is. Is this normal?

i have exact same behavior!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 16, 2019, 02:16:38 pm

Ive got one parent and two vlans interfaces on the same trunk all working fine. Same issue as others where the vlan interfaces don't show up as selectable but just adding the parent gets all 3

Just saw you said more than 2 I can add a third one just for fun.

Hi @donato,

Thanks, much appreciated. Please note that problem seem to arise when you add more than two "child" vlan interfaces. Haven't beed reported of a problem with tagged/trunk interfaces, although curious to know if there are any.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 16, 2019, 02:22:48 pm
@manjeet,

This is addressed via policy based filtering coming up with Premium subscription. Details almost complete. Hope to announce it very soon.

@malac,

Please send your public IP address to sensei - at - sunnyvalley.io. We'll run a trace.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on May 17, 2019, 06:31:17 am
Thanks @MB for the update. Looking forward to it.

Also, Yesterday i enabled the email reporting and today i got this message "Scheduled reports could not be generated. Probably elasticsearch service is not running or not working properly. Please check elasticsearch service manually."

Elastic search is working fine, reports in dashboard and reports section looks all good. Do not understand what could be the issue..
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 17, 2019, 03:52:32 pm
Hi @manjeet,

We're having a look at Scheduled Reports now, let's also check this.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: the-mk on May 18, 2019, 12:55:13 pm
@mb: when I look to the reporting mail - how is that number of "unique local hosts" of the "quick facts" derived? I do not have that many hosts in my network...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: N0_Klu3 on May 18, 2019, 01:05:41 pm
So would this work at replacing pfblockerng?
As in AD Blocking?

Also I read stuff about VLANs, basically I have 2 VLANs running on my main LAN Ethernet port.
Would Sensei work?

I'm planning on rebuilding to OPNSense hopefully today, but I'd really like some sort of ad blocking to replace pfblockerng.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 18, 2019, 02:04:53 pm
Hi @the-mk,

Do you see different statistics in the UI, or are they the same?

If they are: we saw this happen when Sensei was being run for a WAN interface. In that, LAN/WAN directions are being reversed for Sensei. So you see remote host count in place of locals and vice versa.

If not, let's have a look if we're missing something.


Hi @N0_Klu3,

You can try for yourself. It's easy to try out Sensei.

Yep, if you just add the parent LAN interface to the protected interfaces, than you're good to go.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: N0_Klu3 on May 18, 2019, 06:14:01 pm
@mb do you still need an invite or install link?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 18, 2019, 06:16:04 pm
Hi @N0_Klu3,

You can use this command to install 0.8:

curl https://updates.sunnyvalley.io/getsensei8 | sh

Title: Re: Sensei on OPNsense - Application based filtering
Post by: Space on May 19, 2019, 10:15:09 am
Hi,

are these files needed? Took most of my disk space ...

Code: [Select]
root@OPNvirt:/usr/local/sensei/log # du -sm * | sort -n
1 active
14156 archive

These logs contain statistics for all interfaces per second ... but I did not find an option to disable these logs ...

Thanks and best regards,

    Space
Title: Re: Sensei on OPNsense - Application based filtering
Post by: malac on May 19, 2019, 11:54:43 am
@manjeet,

This is addressed via policy based filtering coming up with Premium subscription. Details almost complete. Hope to announce it very soon.

@malac,

Please send your public IP address to sensei - at - sunnyvalley.io. We'll run a trace.

Have you found something?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 19, 2019, 04:16:50 pm
are these files needed? Took most of my disk space ...
These logs contain statistics for all interfaces per second ... but I did not find an option to disable these logs ...

Hi @Space,

Within this beta period, in times of troubleshooting, they can be very valuable for us to point out the location of some of the problems.

Nearing 1.0, we'll cease  to archive logs. In the meantime, adding a functionality to automatically purge logs older than 10 days.

Thanks for pointing this out.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 19, 2019, 04:19:02 pm
Have you found something?

Hi @malac,

Yep, it looks like engine is still a little bit too sensitive for response times. We've lowered the thresholds a bit. Coming with beta10.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: the-mk on May 19, 2019, 04:48:13 pm
Hi @the-mk,

Do you see different statistics in the UI, or are they the same?

If they are: we saw this happen when Sensei was being run for a WAN interface. In that, LAN/WAN directions are being reversed for Sensei. So you see remote host count in place of locals and vice versa.

If not, let's have a look if we're missing something.

when comparing the quick facts from the last report mail with the conns facts from the dashboard - they are pretty much the same when having the report interval set 05/18/2019 00:00 to 05/19/2019 00:00.
I'd expect that the number of unique local hosts are about the same numbers as IP-addresses are listed in the table of local assets from the dashboard.
protected interfaces on the firewall in question with sensei 0.7.0 are 6 vmx-network cards to different LANs and one vmx to WAN.
but maybe my understanding if unique local hosts is wrong here?
could it be that i.e. a host talking on the network of interface #1 is talking to another host on the network interface #2 and the same source hosts also talks to the internet (WAN)?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 20, 2019, 05:49:22 pm
Hi @the-mk,

Thank you very much for providing additional information.

Whether we decide if some IP address is local or remote depends on the flow direction.

A little bit of background info how Sensei works & decides the flow direction:

Sensei deploys between the ethernet adapter and the host operating system, bridging the two, forwarding packets back and forth, and at the same time doing the inspection. Typically we are deployed on inner-facing interfaces.

It assumes that ethernet side of the bridge is LAN and Operating System side is WAN. So flows initiated from the LAN side is considered they are egress, and flows which are initiated from the WAN side are ingress.

For eggress connections, the source IP address who initiated the connection is tagged as "Local", whereas for ingress connections, it's the destination IP address.

So, in your scenario, I'd expect that you having a protected interface on the WAN side might complicate things, since this time sensei will regard all outgoing connections as Ingress (for that interface) and regard the remote IP addresses as local.

Might worth removing that interface from protected interfaces and try to see if this changes things.

If that's not the case, please let us know so that we can have a look at it together.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: kaviraj on May 21, 2019, 09:26:44 am
Hello,

Been testing sensei 0.8.0.beta9 since some days now and since yesterday am facing some strange problems. Some clients are unable to resolve DNS. If i change the client IP everything start to work again. I tried to uninstall and reinstall but still the same.

OPNsense is running over virtualised environment (Proxmox) with kernel 19.1.4 having netmap support as am using virtio.

Test case:
1. I have a client with IP 10.249.10.228/24. When i run a dig it returns a timed-out. A tcpdump on the hypervisor shows that the request was forwarded over the OPNsense interface but a dump on OPNsense interface shows nothing.

2. I stop sensei engine dig starts to work. But as soon as i start it, the client is unable to resolve DNS.

3. Same client but i change IP to 10.249.10.11/24. Dig works.

I may provide remote access if needed.

Thanks for your help.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 21, 2019, 01:46:56 pm
Hi @kaviraj,

Many thanks for reaching out. Please watch for 0.8.0.beta10 which will be coming out today. We have a fix for this.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 21, 2019, 06:15:02 pm
Dear Sensei users,

Sensei 0.8.0.beta10 is out. This brings back VLAN child interfaces and fixes a bug with Cloud Threat Intel. You should now see much better uptimes.

Also addressed: libXdmcp, an Elasticsearch dependency package, is updated to version 1.1.3, fixing a security issue.

Complete list is as follows:

Enjoy :)

Sensei team
Title: Re: Sensei on OPNsense - Application based filtering
Post by: the-mk on May 21, 2019, 08:51:30 pm
@mb: thanks for the clarification - I need to do a deeper check it on the weekend...
Title: Re: Sensei on OPNsense - Application based filtering
Post by: OPNsense4ever on May 22, 2019, 07:10:28 pm
elasticsearch shut down because it started to run out of disk space. How do I tune that? I've got a little over 300GB available for a family of 4 and a few guests a week.

Thanks!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 22, 2019, 07:58:03 pm
Hi @OPNsense4ever

You can use the following guide to determine for how many days you can have your reporting data.

https://guide.sunnyvalley.io/sensei/getting-started/getting-ready#disk-space

Then navigate to Sensei -> Configuration -> Reporting & Data

and set the maximum number of days to store reporting data.

When you set this number to a value smaller than the current one, Sensei will confirm with you if you want the surplus data to be deleted.

For this you need Elasticsearch to stay open, temporarily disable Health check to prevent Sensei from shutting it down again.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: OPNsense4ever on May 25, 2019, 12:34:38 am
Sweet! Thanks!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: JohnDoe17 on May 28, 2019, 05:28:35 pm
I'm new to Sensei, but I'm loving it so far!  Great work!

I do occasionally get a "crash report" notification though.

Here is the sequence of events:

0) Sensei was not installed.
1) Upgraded OPNsense from 18.7.10_4 to 19.1.8.
2) Installed Sensei 0.8.0.beta10.
3) Successfully completed the initial Sensei configure wizard.
4) Noticed a "crash report" when I went to the OPNsense Dashboard.

Unfortunately, I don't have the crash report in front of me at the moment, but I *did* submit it, so hopefully you'll get it from the OPNsense team eventually.  It was something about PHP crashing with bad data related to the "TCP Service Security" password.  I'll keep you posted if I see it again.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 28, 2019, 05:30:41 pm
Hi @JohnDoe17,

Thanks, great that you found Sensei useful for you.

One question: did you install Sensei 0.7 or the new 0.8 version?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: JohnDoe17 on May 28, 2019, 05:35:02 pm
Quote
2) Installed Sensei 0.8.0.beta10.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 28, 2019, 05:36:56 pm
Thanks JohnDoe17, I missed that.

Having a look at it if we're missing something. In the meantime, if you encounter it again, feel free to email the screenshot to sensei - at - sunnyvalley.io.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: JohnDoe17 on May 28, 2019, 06:11:31 pm
I got the crash to happen again.

Note that "Rainbow#Bicycle" is the password I was using for the test.  Does Sensei handle the "#" symbol in a password?

Code: [Select]
[28-May-2019 11:08:17 America/Chicago] PHP Fatal error:  Uncaught Error: Class 'OPNsense\Sensei\Exception' not found in /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Telnet.class.php:111
Stack trace:
#0 /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Telnet.class.php(75): OPNsense\Sensei\Telnet->connect()
#1 /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php(151): OPNsense\Sensei\Telnet->__construct('127.0.0.1', 4346, 1, '', 1)
#2 /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php(134): OPNsense\Sensei\Sensei->runTelnetCommands('127.0.0.1', 4346, 'Rainbow#Bicycle', Array, Array)
#3 /usr/local/opnsense/mvc/app/controllers/OPNsense/Sensei/Api/EngineController.php(89): OPNsense\Sensei\Sensei->runCLI(Array)
#4 [internal function]: OPNsense\Sensei\Api\EngineController->cliAction()
#5 [internal function]: Phalcon\Dispatcher->callActionMethod(Object(OPNsense\Sensei\Api\EngineController), 'cliAction', Array)
#6 [internal function]: Phalcon\Dispatcher->dispatch()
#7 /usr/local/opnsense/www/api.php(26): Phalcon\Mvc\Application->handle()
#8 {main in /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Telnet.class.php on line 111
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 29, 2019, 02:10:25 am
Dear Sensei users,

Sensei 0.8.0 Release Candidate 1 is out. This marks the first step into releasing 0.8 and towards 1.0. There will be no 0.9 :)

Change log is as follows:

We're running 0.7 to 0.8 upgrade tests. As soon as they show that we're good to go, 0.7 users will be reported of the new 0.8 update.

Enjoy :)

Sensei team
Title: Re: Sensei on OPNsense - Application based filtering
Post by: patcsy88 on May 29, 2019, 01:42:31 pm
Just reinstalled OPNsense and the RC1 on APU2C4 with 2GB Swap - so far so good!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 29, 2019, 01:48:31 pm
@patcsy88, thanks for sharing your experience. Glad to hear that.

@JohnDoe17, can you have a look and see if 0.8.0.rc1 is solving your issue?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on May 29, 2019, 03:36:27 pm
@mb: Any news concerning CARP? As soon as I start sensei on CARP master, I have split communication. Cannot ping between CARP members and both nodes are master, dhcp service is communication-interrupted.

Sensei just on backup node seems to works, but except for proxy there is no traffic passing.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 29, 2019, 03:43:13 pm
Hi @hbc,

Since running the netmap bridge application produces the same result, we suspect this to be a netmap issue. I've been trying to get Chelsio adapter to see if we can re-produce this.

In the meantime, any chances you can try the same setup with a different adapter -- preferably em or igb?

Title: Re: Sensei on OPNsense - Application based filtering
Post by: hbc on May 29, 2019, 03:53:08 pm
Not in our CARP HA cluster. We have 12 chelsio ports, so sensei needs to run with it.
Title: Sensei on OPNsense - Application based filtering
Post by: donatom3 on May 29, 2019, 05:33:35 pm
Dear Sensei users,

Sensei 0.8.0 Release Candidate 1 is out. This marks the first step into releasing 0.8 and towards 1.0. There will be no 0.9 :)

Change log is as follows:
  • Per-process health monitoring. Sensei engine now checks heartbeats from its packet processors and taking the corrective action in case of trouble.

We're running 0.7 to 0.8 upgrade tests. As soon as they show that we're good to go, 0.7 users will be reported of the new 0.8 update.

Enjoy :)

Sensei team

@mb Just checking if that is the fix we were talking about to the issue I was seeing with Sensei/netmap crashing causing all traffic to stop until I rebooted the whole firewall.

The last times it happened restarting Sensei from the GUI did not let traffic resume. I had to restart the whole firewall with the auto start of the packet engine turned off.

I did the upgrade to rc1 yesterday so I'll let you know if I still see the issue.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: JohnDoe17 on May 29, 2019, 06:18:58 pm
Hello @mb.

Yes, I can confirm the fix in rc1 did resolve the error I saw with the Sensei CLI API and OPNsense Crash Reporter.

Thank you!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 29, 2019, 07:31:30 pm
Great to hear that @JohnDoe17, thanks for letting us know.

@donatom3 hi,

Yes, it's also netmap related but a different issue. After many trials, I was able to reproduce your situation. Doing a ifconfig down/up seem to resolve the problem.

After Sensei 1.0, we'll have another dive at netmap. It's a great tool, but certainly needs some industry help to get to a super stable state.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: patcsy88 on May 30, 2019, 02:49:48 am
Just reinstalled OPNsense and the RC1 on APU2C4 with 2GB Swap - so far so good!

So Sensei detected high Swap usage over the last 10+ hours and shut itself down. On prompt, I restarted ES. I have now also disabled the Health Check and on the Configuration page started Sensei Packet Engine and the overlay on the page says it is waiting for the service to startup. After 10 or so minutes, nothing happens on the page but vmstat in a shell suggest it is back up. Refreshing the OPNsense page and then going to the Configuration page again shows Sensei is up and running. Not sure if it is the OPNsense framework or Sensei page that is not polling for refresh of content/data...

Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on May 30, 2019, 03:40:04 am
Great to hear that @JohnDoe17, thanks for letting us know.

@donatom3 hi,

Yes, it's also netmap related but a different issue. After many trials, I was able to reproduce your situation. Doing a ifconfig down/up seem to resolve the problem.

After Sensei 1.0, we'll have another dive at netmap. It's a great tool, but certainly needs some industry help to get to a super stable state.

@MB

I believe I just had one of the crashes again but looks like it reconnected on it's own. I noticed it while browsing my apple tv that streaming stopped working and my harmony showed it was offline then was online a few seconds later. This was in the main log file

Code: [Select]
2019-05-29T18:28:37 ERROR: Watchdog: Worker [0] failed to send heartbeat for 6 seconds
2019-05-29T18:28:37 ERROR: Watchdog: Killing Worker [0]
2019-05-29T18:28:37 CRITICAL: Sending TERM signal to worker pid 98083
2019-05-29T18:28:38 CRITICAL: WaitWorkers: processing dead child: pid: 98083
2019-05-29T18:28:38 CRITICAL: WaitWorkers: Child worker0, [pid: 98083] terminated with signal: 11
2019-05-29T18:28:38 CRITICAL: WaitWorkers: Child worker0, [new pid: 60913] re-spawned

And here is the matching time stamp from the worker log.

Code: [Select]
2019-05-29T18:28:38 INFO: Packet Processor [60913] started working
2019-05-29T18:28:38 INFO: Packet Processor [60913] sleeping a while since we're respawned
2019-05-29T18:28:50 INFO: Worker [pid:60913] Pinning to CPU #1
2019-05-29T18:28:50 INFO: Worker [60913] started working


If this was your fix it did it's job very fast. I wouldn't have noticed it unless I was doing some realtime traffic
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 30, 2019, 04:22:00 pm
...overlay on the page says it is waiting for the service to startup. After 10 or so minutes, nothing happens on the page but vmstat in a shell suggest it is back up. Refreshing the OPNsense page and then going to the Configuration page again shows Sensei is up and running. Not sure if it is the OPNsense framework or Sensei page that is not polling for refresh of content/data...

@patcsy88, we have been reported a similar case. Now, it looks like, if the system is under load and not responsive enough, Sensei UI might be waiting for the response for a long time.

Thanks for your input, this would be helpful in diagnosing the root cause.

One question: I guess you have like 4 GB of memory. For how many devices are you running Sensei for?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 30, 2019, 04:29:11 pm
If this was your fix it did it's job very fast. I wouldn't have noticed it unless I was doing some realtime traffic

Hi @donatom3, yes, chances are high that it might be fixing yours.

We implemented the heartbeat mechanism for any cases where packet engine might hang for more than 5 seconds.

If the main process senses that the packet processor process is not feeling well enough, it simply restarts the process.

This is to keep network availability high in case anything goes wrong.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: patcsy88 on May 30, 2019, 04:32:17 pm
One question: I guess you have like 4 GB of memory. For how many devices are you running Sensei for?

@MB only 4 devices with normal web browsing
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 30, 2019, 04:38:27 pm
@MB only 4 devices with normal web browsing

@patcsy88, what does the following tell?

Code: [Select]
cat /usr/local/libexec/elasticsearch/config/jvm.options  | grep "^\-Xm"
ps awxu | grep elastic | grep -v grep
ps awxu | grep eastpect | grep -v grep
Title: Re: Sensei on OPNsense - Application based filtering
Post by: patcsy88 on May 30, 2019, 04:44:24 pm

@patcsy88, what does the following tell?

Code: [Select]
cat /usr/local/libexec/elasticsearch/config/jvm.options  | grep "^\-Xm"
-Xms2g
-Xmx2g

ps awxu | grep elastic | grep -v grep
elasticsearch  4875   2.2 46.6 3878304 1927928  -  I    08:22     74:00.13 /usr/local/openjdk8/bin/java -Xms2g -Xmx2g -XX:+UseConcM

ps awxu | grep eastpect | grep -v grep
root           7417   0.5  4.5 3094852  185100  -  S<   08:35      8:29.81 eastpect: Eastpect Instance 0 (eastpect)
root          66470   0.0  0.0 1270428       0  -  IW<  -          0:00.00 eastpect: Eastpect Streamer Instance (eastpect)
root          80093   0.0  2.2 1270428   92760  -  S<   08:35      0:04.70 /usr/local/sensei//bin/eastpect -D
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 30, 2019, 04:50:50 pm
Code: [Select]
cat /usr/local/libexec/elasticsearch/config/jvm.options  | grep "^\-Xm"
-Xms2g
-Xmx2g

There it is. Edit this file, change these line to read:

Code: [Select]
-Xms512m
-Xmx512m

and stop/start elasticsearch service. You should be good to go.

For fresh installs we adjust this setting. Any chances you had a prior Sensei installation in this device?

Title: Re: Sensei on OPNsense - Application based filtering
Post by: patcsy88 on May 30, 2019, 04:57:50 pm


For fresh installs we adjust this setting. Any chances you had a prior Sensei installation in this device?

No it was a fresh install!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on May 31, 2019, 12:57:38 am
@patcsy88, got it. We'll have a check for that whenever sensei is update/installed.

How is the system doing after you adjusted Elastic memory?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: alelnr on May 31, 2019, 11:01:39 am
Hi All,
in our environment OPNsense 19.1.8 + Sensei 0.7, sensei cloud reputation is completely blocking OPNsense unbound DNS service. To allow unbound dns answer to queries on sensei protected interfaces, i had to disable cloud reputation service.
Thank you
Title: Re: Sensei on OPNsense - Application based filtering
Post by: patcsy88 on May 31, 2019, 10:17:05 pm
No it was a fresh install!

Running "ps awxu | grep elastic | grep -v grep" shows the following output:-

elasticsearch 18938  30.9 61.1 3897508 2528480  -  I    04:09       6:49.91 /usr/local/openjdk8/bin/java -Xms2g -Xmx2g -XX:+UseConc

"cat /usr/local/libexec/elasticsearch/config/jvm.options | grep 512" gives me

-Xms512m
-Xmx512m

I restarted ElasticSearch via the UI.

Is there a default setting it is picking up instead of from usr/local/libexec/elasticsearch/config/jvm.options?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 01, 2019, 03:13:01 am
Hi @alelnr, service should be restored as of today. This was due to a BGP configuration problem . Sorry for the inconvenience.

@patcsy88, that should be the file elasticsearch is getting the settings from. Let's try to reproduce the issue here. I'll update you.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: spetrillo on June 01, 2019, 10:17:31 pm
Is Sensei available from the plugins section or do we need to do a CLI install? I would very much like to try it out.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 01, 2019, 10:29:57 pm
Is Sensei available from the plugins section or do we need to do a CLI install? I would very much like to try it out.

Hi @spetrillo,

Thanks for your interest in Sensei. You'll need to install it from OPNsense CLI.

Please see here:

https://guide.sunnyvalley.io/sensei/getting-started/prepare-your-firewall
https://guide.sunnyvalley.io/sensei/getting-started/setup
Title: Re: Sensei on OPNsense - Application based filtering
Post by: spetrillo on June 01, 2019, 10:41:17 pm
Thanks @mb.

What does Sensei replace?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 02, 2019, 04:25:58 pm
Hi @spetrillo,

OPNsense is already a great firewall. Nothing to replace indeed.

Sensei is augmenting the firewall with commercial grade next generation features like:


And yet many to come...

It integrates in such a way that it makes it possible for you to continue to use all of the existing OPNsense functionality.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: spetrillo on June 02, 2019, 04:37:27 pm
@mb does Sensei augment what Suricata brings to the table or are they aimed at totally different things. It seems to me there is overlap and I am trying to understand if I should use one or the other or both.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: ruffy91 on June 02, 2019, 06:34:50 pm
They do different things but they overlap a bit.

Both do Deep Packet Inspection but with other targets.
Suricata is only an engine, you have to select the rules yourself to reach your target.
You can use abuse.ch SSL Blacklist to block known bad Certificates or ET Pro Trojan Rules to block and detect network traffic from trojans and many more. It's there to defend against known exploits, vulnerabilities and threats mostly. You can enhance it yourself by adding the right rules.

Sensei classifies Traffic into application + web categories and allows you to specify what to block.
For example block File-Upload/Sharing sites to enforce the policy that employees have to use your in-house file sharing system etc. which would be very hard to do using suricata.
As addition they provide a blacklist of sites they see spreading malware.

So I see it like this: Block known threats using suricata and use Sensei for defense-in-depth by disabling apps you do not need or do not want in your network.

Also sensei has usable reporting, suricata just shows alerts, sensei shows relations and also what is happening in your network even if it's not an alert.

Gesendet von meinem MI 9 mit Tapatalk

Title: Re: Sensei on OPNsense - Application based filtering
Post by: spetrillo on June 02, 2019, 07:05:56 pm
I would agree on what Suricata shows. I am actually trying to find some kind of front end that visualizes the Suricata data. Working with Elastic Search right to see where it can get me.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 04, 2019, 01:02:34 am
Is there a default setting it is picking up instead of from usr/local/libexec/elasticsearch/config/jvm.options?

Hi @patcsy88, it turns out that the correct jvm.options path should be:

Code: [Select]
/usr/local/lib/elasticsearch/config/jvm.options
Fix is also included in 0.8.0.rc2. Many thanks for bringing this into our attention.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: JohnDoe17 on June 04, 2019, 06:08:42 pm
@mb

Looks like this issue wasn't completely resolved afterall...

Code: [Select]
User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
FreeBSD 11.2-RELEASE-p10-HBSD  5e5adf26fc3(stable/19.1) amd64
OPNsense 19.1.8 dff8692b8
Plugins os-arp-scan-1.1 os-ftp-proxy-1.0_1 os-sensei-0.8.0.rc1 os-sensei-updater-0.8.0_21 os-vmware-1.5
Time Tue, 04 Jun 2019 11:05:35 -0500
OpenSSL 1.0.2r  26 Feb 2019
PHP 7.2.18
PHP Errors:
[04-Jun-2019 11:02:51 America/Chicago] Exception: Cannot connect to 127.0.0.1 on port 4343 in /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Telnet.class.php:111
Stack trace:
#0 /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Telnet.class.php(75): OPNsense\Sensei\Telnet->connect()
#1 /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php(155): OPNsense\Sensei\Telnet->__construct('127.0.0.1', 4343, 1, '', 0.5)
#2 /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php(138): OPNsense\Sensei\Sensei->runTelnetCommands('127.0.0.1', 4343, 'ballyhoo#Recons...', Array, Array)
#3 /usr/local/opnsense/mvc/app/controllers/OPNsense/Sensei/Api/EngineController.php(93): OPNsense\Sensei\Sensei->runCLI(Array, 'ballyhoo#Recons...')
#4 [internal function]: OPNsense\Sensei\Api\EngineController->cliAction()
#5 [internal function]: Phalcon\Dispatcher->callActionMethod(Object(OPNsense\Sensei\Api\EngineController), 'cliAction', Array)
#6 [internal function]: Phalcon\Dispatcher->dispatch()
#7 /usr/local/opnsense/www/api.php(26): Phalcon\Mvc\Application->handle()
#8 {main}
[04-Jun-2019 11:03:24 America/Chicago] Exception: Cannot connect to 127.0.0.1 on port 4343 in /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Telnet.class.php:111
Stack trace:
#0 /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Telnet.class.php(75): OPNsense\Sensei\Telnet->connect()
#1 /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php(155): OPNsense\Sensei\Telnet->__construct('127.0.0.1', 4343, 1, '', 0.5)
#2 /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php(138): OPNsense\Sensei\Sensei->runTelnetCommands('127.0.0.1', 4343, 'ballyhoo#Recons...', Array, Array)
#3 /usr/local/opnsense/mvc/app/controllers/OPNsense/Sensei/Api/EngineController.php(93): OPNsense\Sensei\Sensei->runCLI(Array, 'ballyhoo#Recons...')
#4 [internal function]: OPNsense\Sensei\Api\EngineController->cliAction()
#5 [internal function]: Phalcon\Dispatcher->callActionMethod(Object(OPNsense\Sensei\Api\EngineController), 'cliAction', Array)
#6 [internal function]: Phalcon\Dispatcher->dispatch()
#7 /usr/local/opnsense/www/api.php(26): Phalcon\Mvc\Application->handle()
#8 {main}
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 05, 2019, 07:14:02 am
@JohnDoe17, got it, thanks for the update.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on June 05, 2019, 04:24:07 pm
Hello Murat,

one question. The problem with the VLAN Interfaces should be fixed since two versions what i saw.
I'm on 0.8.0.rc1 and still have the same problem as in version 0.8.0.beta4.

Problem was described here in this topic -> https://forum.opnsense.org/index.php?topic=9521.msg55463#msg55463
Should this case also be fixed with the current version ?

Thank you!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 05, 2019, 05:58:39 pm
Hi @BeNe,

Yes, you should be able to protect your VLAN interfaces now. You have two options:

1. If you add the VLAN parent interface to the protected interfaces list, then you should be all set. Sensei processes all VLANs as well as the untagged packets for that interface.

2. If you want to add vlan child interfaces one by one, you should also be able to do that provided that you do not add the parent interface at the same time. (due to a netmap issue). We also have a check in the UI for that.

I've heard from people running both of the options fine, though option number #1 should be more preferable performance-wise. Since in that mode we're using the netmap mode natively for a variety of interfaces (em, igb etc).
Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on June 05, 2019, 09:41:22 pm
@mb Thank you for your answer.

If i add the VLAN parent interface to the protected interfaces list, all VLAN child are unable to connect to the OPNsense anymore. I can see entries in the Firewall Live-Log, that all packets are denied.
If i stop the Sensei Packet Engine everything works fine again and there are no more denied packets.

Is there something i can debug ?
Thanks
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on June 05, 2019, 10:01:30 pm
@mb Thank you for your answer.

If i add the VLAN parent interface to the protected interfaces list, all VLAN child are unable to connect to the OPNsense anymore. I can see entries in the Firewall Live-Log, that all packets are denied.
If i stop the Sensei Packet Engine everything works fine again and there are no more denied packets.

Is there something i can debug ?
Thanks
Bene you're only adding the parent interface right?

I had this problem before when adding both parents and vlan.

Sent from my Pixel 3 XL using Tapatalk

Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on June 05, 2019, 10:11:44 pm
Yes, ONLY the parent interface. One interface at all is added.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 05, 2019, 10:19:12 pm
Hi @BeNe,

A few questions:

1. I'm assuming you're on the latest 0.8.0.rc1, correct?
2. What happens if you enter bypass mode (Sensei -> Status -> Enter Bypass Mode) is it still the same?
3. Which ethernet adapter are you using? Intel, Broadcom or any other?

Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on June 05, 2019, 10:52:18 pm
1. I'm assuming you're on the latest 0.8.0.rc1, correct? -> Yes
2. What happens if you enter bypass mode (Sensei -> Status -> Enter Bypass Mode) is it still the same? -> Still the same
3. Which ethernet adapter are you using? Intel, Broadcom or any other? ->Intel

OPNsense is running inside a KVM (Virtual Maschine on a Proxmox Host).
The WAN Interface is a Intel Card with PCI Passthrough directly to the VM
The LAN is virtual Network Interface

(https://i.ibb.co/tcnX7Jy/block.png) (https://ibb.co/tcnX7Jy) (https://i.ibb.co/n1gwh6f/bypassed.png) (https://ibb.co/n1gwh6f) (https://i.ibb.co/yqvRm94/lan.png) (https://ibb.co/yqvRm94) (https://i.ibb.co/G7GGVJn/interfaces.png) (https://ibb.co/G7GGVJn)

There is the traffic blocked on the "LAN" interface from 172.16.50.0/24 that is normaly on VLAN_50.
On the LAN is 172.16.17.0/24. Of course is this traffic source blocked on that interface. Did i missed something that i need to adjust ?


Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 05, 2019, 11:07:55 pm
Hi @Bene,

I think there is something else in your configuration that needs attention. I'll reach out to you. Let's have a look together.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: BeNe on June 07, 2019, 01:23:11 pm
Hi Murat,

thanks for your help! I changed my interface from "em" to "igb" as you said.
Now it works.

So i can confirm a problem with "em" interfaces. In my case, i let the "igb" interface  ;)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 07, 2019, 05:46:38 pm
Hi @BeNe,

Thank you very much for your update. Now it's clear for me.

When an interface is opened in netmap mode, ARP packets destined for vlan child interfaces do not make its way to their destinations.

This seems to be fixed in FreeBSD 11.2-stable.

We'll sponsor another round of netmap work which is specifically focused on fixing known problems.

For now a bit of advise who are using Sensei or Suricata (IPS mode):

1. Last thing I'd want would be to endorse a brand/model, however for us, igb(4) based adapters seemed to be the ones which gave the best results in terms of reliability / performance (with regard to netmap support).

2. If you're using igb(4) and experiencing high interrupt utilization, you can set:

    a) hw.igb.rx_process_limit: -1 (default is 100)
    b) machdep.hyperthreading_allowed: 0

We've seen these settings help improve the performance for igb(4) based systems.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 11, 2019, 11:09:18 pm
Dear Sensei users,

Sensei 0.8.0 Release Candidate 2 is out. This marks the final step into releasing 0.8 and towards 1.0

This version is also available for an update for 0.7 users.

Change log is as follows:

Enjoy :)

Sensei team
Title: Re: Sensei on OPNsense - Application based filtering
Post by: adel_xf on June 14, 2019, 01:18:30 pm
Hello,

I tried to go with Sensei, when selecting the network interfaces I have no interface proposing networks.

My OPNSense configuration:

OPNsense 19.1.9-amd64
FreeBSD 11.2-RELEASE-p10-HBSD
OpenSSL 1.0.2s May 28, 2019

OPNSense is a VM Proxmox
2 virtio network cards
100 GB disk
8 GB of RAM

I tried both versions of Sensei (0.7, 0.8 ).
Thank you for your attention.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: adel_xf on June 14, 2019, 01:37:37 pm
I tested the following command that seems to work your opinions?

Code: [Select]
opnsense-update -fbkr 19.1.4-netmap
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 14, 2019, 06:25:41 pm
I tested the following command that seems to work your opinions?

Code: [Select]
opnsense-update -fbkr 19.1.4-netmap

Hi @adel_xf,

Many thanks for giving Sensei a try. OPNsense created 19.1.4-netmap kernel to integrate the latest improvements and bug fixes including the Sunny Valley sponsored virtio/vmx work.

It should be ok to use that. However make sure you're not missing anything important with the newer stock kernels

After Sensei 1.0, we'll do another round of netmap work to complete upstream netmap import process.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on June 15, 2019, 11:51:56 am
Hi MB, I am facing few issue after updating the sensei package.

1. Do not see deployment size above 25 (Using routed mode)

2. Disabled the health check in previous version and now if i enable it then do not see the save options. Disabled / grayed out.

3. Email reports not working: After update it generated the report once and it was working i.e. showing the result but after that one report didn't receive any new email.
If i reenter the mail server details and click test then it is working and sends  a notification email. but do not receive the report email generated at night.
Also why it happens i.e. if i test email and save it. Then refresh the page and retest it, it just give me an error:
Your mail configuration is invalid!
Response: (535, '5.7.8 Authentication rejected')
Meaning we can only test it once and then save details and leave it that way. It works and emails works, but why receive error when try to test again until we reenter the password before clicking the test.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 15, 2019, 10:45:04 pm
Hi @manjeet,

Thanks for the report.

Looks like #2 and #3 are buggies. We fixed them today. Should be arriving with 0.8 release next week.

#1, if your RAM is 4GB, this is the expected behavior, since we were reported of swap utilization with deployments of around 70-80 users and 4GB RAM.

So we thought that it would be safer to restrict deployment size to 25 users or less if the device has 4GB of memory.

If it's not the case for you, then it's probably a browser issue. Let's have a look together.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on June 17, 2019, 06:41:22 am
Hello MB,

As per your email and post, here are the details you asked:
1. Did you udpate from 0.7 or from an earlier 0.8 beta/rc?
---> Updated from 0.7
2. How much memory do you have?
---> 8GB
3. Which browser are you using? Anything changes if you switch to Google Chrome?
---> Chromium
4. Does your email account password include any special characters e.g. "&" ?
---> It does contains special characters
5. What happens if you invoke the report manually ? command is as follows:
---> Command ( /usr/local/sbin/configctl sensei mail-reports) gave me OK and received the email report

Update: Ever since i reconfigured the email reporting from Saturday (IST), i am receiving the report email. I think i must be the update which somehow messed something.
b> My System is Intel Core i5-7400 CPU @ 3.00GHz with 8 GB RAM and 8 GB SWAP.
c> I use Chromium. But tested it on Google Chrome and firefox and deployment size is still the same.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: thg0432 on June 17, 2019, 03:19:47 pm
Hi @mb,

Can you tell us if/when users/groups will be implemented within Sensei?

Title: Re: Sensei on OPNsense - Application based filtering
Post by: ruffy91 on June 17, 2019, 08:34:51 pm
For comparison I get the following throughput with/without sensei on a pcengines APU3A4:
The interface is just the LAN interface which is a igb NIC without VLAN or LAGG.

Without Sensei 250/50 Mbps
With Sensei 140/40 Mbps

I enabled some security features of sensei and I blocked the malware Web category.

I do not use any other features which do have an impact on throughput like IDS or traffic shaping.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Antaris on June 17, 2019, 11:41:06 pm
@ruffy91, it's good that enterprise addon even works on APU3A4s CPU(and on top of that - it's free). If you want fluent Sensei, remember few things: full blown Xeon or desktop i5-7 CPU, 8 ram, SSD. For energy efficient platforms will always be heavy performance loss.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 18, 2019, 04:00:08 am
@thg0432, yes, currently working on it. We'll provide more info on the timing and details early next month.

@manjeet, glad that your problem with the e-mail report is resolved.  it looks like re-configuring the e-mail server settings proved to be a workaround.

However, for the root cause, if anyone out there who has upgraded from 0.7 and experiencing the e-mail reporting problem, we'd like to dig together.

Regarding deployment size, it looks like that sometimes physical memory size is reported less than exact 8GB (e.g. 7.8GB). So we've adjusted the minimum threshold a bit to accommodate that case.

We'll ship 0.8 release tomorrow morning PST. Hopefully it will resolve your situation.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: marcri on June 18, 2019, 09:40:32 am
Hi,
some questions about sensei:
- is it possible to use  an existing elasticsearch instance on a dedicated server?
- if it's possible, can I use one elastic-server for two opnsense instances (failover-setup)?
- where can i get information about using sensei on a corporate network? Prices?
Best
Marc



Title: Re: Sensei on OPNsense - Application based filtering
Post by: aimdev on June 18, 2019, 11:03:08 am
Issues I encountered after installing Sensei included web interface locking up, and unable to access opnsesne via ssh. I could still interact with the console. After this occurred i had to uninstall the plugin.
Hopefully the information that the plugin sent back to sunnyvalley will provide insight into my web/ssh issues.

I have the same issue, no access to ssh (an operational requirement) however by enabling bypass mode I can access ssh.

I am running the latest beta version, downloaded today.

Fine Tuning the Remote access option (tediously disable all except the Secure Shell option) gets access back.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 18, 2019, 11:05:42 pm
Fine Tuning the Remote access option (tediously disable all except the Secure Shell option) gets access back.

@aimdev, many thanks for the feedback. I guess the confusing thing is we also have a "ssh" application under "General TCPIP" category. We're fixing this with the upcoming 1.0.

@marcri,

For the main database, you cannot use an external database at the moment. Though premium subscription is offering an option to stream reporting data to an "additional" elastic search database via either syslog or native elasticsearch REST API. 

From time to time we get this request. I guess we should start planning on having the database on an external system. When we do that, it should be trivial to have one elastic instance (either clustered or not) serving many Sensei deployments.

Imagine you're an MSP serving multiple clients or you are a corporate having multiple OPNsense deployments. With such a setup, you should be able to have an aggregate big picture view of whole assets in a centralized system. This way, you could also benefit from Kibana and other 3rd party reporting tools.

Today we're releasing 0.8. Next month, we'll ship 1.0, integrated with OPNsense; and with the details of Premium subscription. Stay tuned :)




Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 19, 2019, 02:38:53 am
Dear Sensei users,

After six months of ongoing effort & field testing, it's our pleasure to announce that Sensei 0.8 is finally released.

For some of you who were using 0.7, this version brings quite a loaded set of features:
https://www.sunnyvalley.io/post/sensei-0-8-is-released

We will be releasing Sensei 1.0 next month, in July 2019, which will also cease the BETA program and the software will be publicly available for all users.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on June 19, 2019, 02:33:45 pm
Wow! This is great! One of the bests and most wanted missing feautures added to our belowed opnsense firewall. Sensei is one of a kind software for sure! Keep up the good work! :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 20, 2019, 02:30:08 am
@Archanfel80,

Many thanks for your feedback. With its open, flexible, extendable architecture; and its great community of users, we love working with OPNsense.

We will do our best to keep adding more value.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on June 20, 2019, 06:21:47 am
HI MB, everything works fine as mentioned after the update.

Now i have 1 issue and 1 feature request (If its not already there)

Issue: I am not able to update sensei package from command line when using the autoupdate of opnsense i.e. option 12. Same thing happened when i upgraded from 0.7 and now same for yesterday's update. I can only update sensei package from sensei dashboard in web gui.

Feature: Is there any way to for an single or multiple websites / app / category to only put in alert mode. For example if i want my network users to allow access to certain websites but also want to know who access the website or protocol and when AND for specific blocked contents i.e. when someone tried to access it and rather then looking for access logs or block logs just simply have a different tab for alerts only to check easy and fast. I know we can filter it on reports but it will be easy to have an alert tab for both allowed and blocked for that specific alert mode. AND Can we also send alerts via email ?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Archanfel80 on June 20, 2019, 09:27:21 am
Just a quick report about an issue what i see.
If you installed sensei from the cli first while in the beta and updated since then for some reason the search data not deleted and consumed the disc space after the final 0.8 upgrade. I cant delete the date from the webui it just says simply 'error'.
I cant figured out why but removed the sensei completely, deleted the '/usr/local/sensei' folder and reinstall sensei from the plugins. Now everything works and the disc usage reduced dramatically. So if you're like me, so installed sensei while in the beta probably the best to backup the config remove sensei, delete the sensei directory, reinstall sensei and restore the config which is restore your custom sensei settings.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: the-mk on June 20, 2019, 11:25:11 am
will do a reinstall of sensei 0.8 too
looked at the /usr/local/sensei directory - mine was about 44 gigabytes - most of it in /usr/local/sensei/log/archive
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 21, 2019, 12:14:57 am
@Archanfel80, @the-mk,

With regard to archived logs, you can use the following commands to get rid of very old logs:

find /usr/local/sensei/log/active -type f -mtime +15d  | xargs rm -f {}\;
find /usr/local/sensei/log/archive -type f -mtime +15d  | xargs rm -f {}\;


Sensei health check system should have had this handled. Looks like a commit which did not end up in the release. Will integrate for 1.0.

For the elasticsearch data, along the way to 0.8, we changed the naming scheme for the indexes. This should be the reason why some indexes were not purged.

We'll also handle that with 1.0. For now, the workaround would be resetting reporting data (Sensei -> Configuration -> Reporting & Data) (be aware: this will delete all reporting history).


Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 21, 2019, 03:54:26 am
@manjeet,

Currently, we're locking the os-sensei package. This is why OPNsense autoupdate do not update Sensei package. This was done for the period of integration to the OPNsense and for a more controlled software delivery. Lock will be removed shortly and Sensei will get updated along with other OPNsense packages.

Your feature request sounds cool; though we'll need to think a bit more on the correct implementation and also try to see how many other users would also be interested in this feature.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: bulmaro on June 28, 2019, 04:28:12 pm
Sensei has detected swap was usage high (21 -- 13831872% usage) and has shut down Sensei services in order to prevent a network outage.

Any suggestions for my case?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 28, 2019, 05:40:31 pm
Hi @bulmaro,

The reason is most likely Elasticsearch consuming all memory and OS begins swapping. When the OS does swapping overall system performance is significantly degraded and this in turn affects Sensei doing its job.

To avoid a connectivity problem, we shut down Sensei with a warning like this (numbers seem weird, need to look at that)

How many devices do you have behind sensei and what is your hardware configuration?

https://help.sunnyvalley.io/hc/en-us/articles/360025047373-Hardware-Requirements

This will give you an overview of the recommended HW configuration according to the size of your deployment.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: bulmaro on June 28, 2019, 09:37:11 pm
I have two servers
physical equipment with 30 connected clients, equipment characteristics:
CPU 3-2105 CPU @ 3.10GHz (4 cores)
RAM memory: 8GB

Azure server, 3 clients connected
CPU E5-2673 v4 @ 2.30GHz (2 cores)
4GB RAM

it's exactly the same message for both
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 28, 2019, 09:43:46 pm
@bulmaro,

Thanks for the swift reply. These configurations look perfectly ok for the deployment size. Let me reach out to you; and we can have a look together.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on June 29, 2019, 01:28:22 am
These configurations look perfectly ok for the deployment size. Let me reach out to you; and we can have a look together.

Dear Sensei users,

Out of @bulmaro's case, I think it's important to give a heads-up on this:

The hardware recommendation we provide is calculated based on the fact that the system runs OPNsense with Sensei. We did not take other services which might be already running on the firewall (IDS, Proxy etc.) into consideration.

We highly recommend that you also oversee the requirement of those services and do your own sizing according to that.

In @bulmaro's specific case, 1/2 of the memory was already consumed by the squid service. And the system was swapping even Sensei and Elasticsearch were not active.

@bulmaro, many thanks for your help to diagnose the issue.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on July 01, 2019, 06:42:14 am
@MB

Not sure if this has ever been brought up. It's something I've seen for a while.

On any of the live session explorers or drill down of traffic if I do a whois for the record that is the domain name it always only resolves the top level domain. For example US.lgtvsdp.com does a whois for domain COM thus always giving me the same result for any .com address.

Shouldn't it be doing the whois query on the second level domain?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on July 01, 2019, 07:44:22 am
Hi @MB,

Few days back we had power issue and after that "Elasticsearch" is not working. I have tried start the service many times, rebooted and tried but didn't work. "Sensei Packet Engine" is working.

I have tried "Perform health check for indices" and it kind of stuck and does not do anything. "You can erase reporting data" option is grayed out. I also tried to run these command from terminal and got the error:
1. /usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
2. /usr/local/sensei/scripts/installers/elasticsearch/create_indices.py
ERROR: ***ERROR: Connection could not be established with elasticsearch server.**

Also tried reset the package but it didn't fix the issue. Haven't delete / uninstall and reinstall the package yet. kindly help.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 01, 2019, 11:13:51 pm
@donatom3, checking that one.

@manjeet, "Reset reporting" will be enabled even if Elasticsearch is not running. Fixing for 1.0.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: thg0432 on July 02, 2019, 02:52:03 pm
@mb

I was wondering if there's a setting for rotating the logs that are in /usr/local/sensei/log/archive ?  or is that something that needs to be cleaned out manually? 
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 03, 2019, 07:24:47 am
@donatom3,

You're right. Currently we run the whois query the for the whole FQDN. We should be doing for the domain part only. Fix is implemented today and shipping with 1.0.

@thg0432,

Engine logs older than two weeks are to be automatically deleted. 0.8 had a glitch doing the actual delete. Fix is implemented for 1.0.

In the meantime, you can get rid of them by running this command:

find /usr/local/sensei/log/archive -type f -mtime +15d  | xargs rm -f {}\;
Title: Re: Sensei on OPNsense - Application based filtering
Post by: zyon on July 03, 2019, 10:36:11 pm
Just installed Sensei and just awesome.
All i need in one application :)

For my information sensei work with squid ? if yes it's possible to use it like a proxy server ? ( for mobile for example )

Thanks for your hard work :)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 04, 2019, 04:15:42 am
hi @zyon,

Thanks for your feedback. Glad that you found Sensei useful for you. All welcome.

Sensei plugs kind of transparent to the system. So it does not change the way other services like Squid are operating.

I think I did not completely get your question.

Do you want to learn if Sensei can act like a proxy, for instance, for caching?

Title: Re: Sensei on OPNsense - Application based filtering
Post by: manjeet on July 04, 2019, 06:20:02 am
Hello @MB, Is there any way to bypass a user from sensei filter
OR
More accurately for my case, bypass anyone which goes from a particular gateway.

Actually, i have 2 ISPs which are in load balancing mode on opnsense, i want anyone connected to gateway 2 to just bypass any filters or blocking or logging.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 09, 2019, 02:30:29 am
Hi @manjeet,

Actually, i have 2 ISPs which are in load balancing mode on opnsense, i want anyone connected to gateway 2 to just bypass any filters or blocking or logging.

I believe - in your case - the outbound route selection is done randomly and not through a policy decision based on source IP address, am I correct?

If that is so,  and it's not something related to the source IP/network address, I'm afraid there is no way we can correlate the user with the outbound ISP. This is because we jump into the scene way too early, without routing/NAT'ing logic comes into the scene.

If it's source IP related, it's possible, and along with user/group based filtering, this is one of the features of the premium edition:

https://help.sunnyvalley.io/hc/en-us/articles/360025173953-Sensei-Editions
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on July 10, 2019, 01:34:29 am
@mb

This probably isn't Sensei since it affects Suricata to. But since I upgraded to 19.7 RC1 suricata won't start because it can't find my interface and Sensei says no interface selected in the status page. I can change to any of my interfaces and they all say the same.

Here is a portion of the worker thread when I started this morning. From it it looks like everything points to netmap being the issue. I started a thread in the 19.7 release candidate forums about it. Just a warning to anyone relying on Sensei.

Code: [Select]
2019-07-09T07:38:49 INFO: Packet Processor [39794] started working
2019-07-09T07:38:49 INFO: Worker [pid:39794] Pinning to CPU #1
2019-07-09T07:38:49 INFO: Worker [39794] started working
2019-07-09T07:38:49 INFO: License file /usr/local/sensei//etc//license.data not located (No such file or directory) assuming FREEMIUM
2019-07-09T07:38:49 INFO: Created Syn Filter Context Table [mask: 16383]
2019-07-09T07:38:49 INFO: Created a new Worker Instance pid: 39794
2019-07-09T07:38:49 INFO: Requested Single Threaded Stack
2019-07-09T07:38:49 INFO: Inline operation mode selected! Bridging br1 (netmap@igb1 <-> netmap@igb1^)
2019-07-09T07:38:50 INFO: Created Enrichment Service @127.0.0.1:4343
2019-07-09T07:38:50 WARNING: loadUserCache: file /usr/local/sensei//userdefined/db/Usercache//userauth_cache.db is not a regular file
2019-07-09T07:38:50 INFO: Number of Queues for interface: igb1: 2
2019-07-09T07:38:50 INFO: LAN: igb1[igb1] Queue: 0, #Queues: 2, Packet Device: Netmap
2019-07-09T07:38:50 INFO: WAN: igb1^[igb1], Queue: 0, #Queues: 1, Packet Device: Netmap-Host-Bridge
2019-07-09T07:38:50 INFO: Initializing for BRIDGE Mode
2019-07-09T07:38:50 CRITICAL: Failed to create LAN interface (igb1:0(igb1:0): 6(Device not configured)
2019-07-09T07:38:50 ERROR: Failed Initializing Interfaces, bailing out
2019-07-09T07:38:51 INFO: Packet Processor [19965] started working
2019-07-09T07:38:51 INFO: Packet Processor [19965] sleeping a while since we're respawned
2019-07-09T07:39:03 INFO: Worker [pid:19965] Pinning to CPU #1
2019-07-09T07:39:03 INFO: Worker [19965] started working
2019-07-09T07:39:03 INFO: License file /usr/local/sensei//etc//license.data not located (No such file or directory) assuming FREEMIUM
2019-07-09T07:39:03 INFO: Created Syn Filter Context Table [mask: 16383]
2019-07-09T07:39:03 INFO: Created a new Worker Instance pid: 19965
2019-07-09T07:39:03 INFO: Requested Single Threaded Stack
2019-07-09T07:39:03 INFO: Inline operation mode selected! Bridging br1 (netmap@igb1 <-> netmap@igb1^)
2019-07-09T07:39:04 INFO: Created Enrichment Service @127.0.0.1:4343
2019-07-09T07:39:04 WARNING: loadUserCache: file /usr/local/sensei//userdefined/db/Usercache//userauth_cache.db is not a regular file
2019-07-09T07:39:04 INFO: Number of Queues for interface: igb1: 2
2019-07-09T07:39:04 INFO: LAN: igb1[igb1] Queue: 0, #Queues: 2, Packet Device: Netmap
2019-07-09T07:39:04 INFO: WAN: igb1^[igb1], Queue: 0, #Queues: 1, Packet Device: Netmap-Host-Bridge
2019-07-09T07:39:04 INFO: Initializing for BRIDGE Mode
2019-07-09T07:39:04 CRITICAL: Failed to create LAN interface (igb1:0(igb1:0): 6(Device not configured)
2019-07-09T07:39:04 ERROR: Failed Initializing Interfaces, bailing out
2019-07-09T07:39:05 INFO: Packet Processor [18480] started working
2019-07-09T07:39:05 INFO: Packet Processor [18480] sleeping a while since we're respawned
2019-07-09T07:39:17 INFO: Worker [pid:18480] Pinning to CPU #1
2019-07-09T07:39:17 INFO: Worker [18480] started working
2019-07-09T07:39:17 INFO: License file /usr/local/sensei//etc//license.data not located (No such file or directory) assuming FREEMIUM
2019-07-09T07:39:17 INFO: Created Syn Filter Context Table [mask: 16383]
2019-07-09T07:39:17 INFO: Created a new Worker Instance pid: 18480
2019-07-09T07:39:17 INFO: Requested Single Threaded Stack
2019-07-09T07:39:17 INFO: Inline operation mode selected! Bridging br1 (netmap@igb1 <-> netmap@igb1^)
2019-07-09T07:39:18 INFO: Created Enrichment Service @127.0.0.1:4343
2019-07-09T07:39:18 WARNING: loadUserCache: file /usr/local/sensei//userdefined/db/Usercache//userauth_cache.db is not a regular file
2019-07-09T07:39:18 INFO: Number of Queues for interface: igb1: 2
2019-07-09T07:39:18 INFO: LAN: igb1[igb1] Queue: 0, #Queues: 2, Packet Device: Netmap
2019-07-09T07:39:18 INFO: WAN: igb1^[igb1], Queue: 0, #Queues: 1, Packet Device: Netmap-Host-Bridge
2019-07-09T07:39:18 INFO: Initializing for BRIDGE Mode
2019-07-09T07:39:18 CRITICAL: Failed to create LAN interface (igb1:0(igb1:0): 6(Device not configured)
2019-07-09T07:39:18 ERROR: Failed Initializing Interfaces, bailing out
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 10, 2019, 02:19:07 am
Hi @donatom3,

Many thanks for the heads-up.

Reading https://forum.opnsense.org/index.php?topic=13436.msg61861#new (https://forum.opnsense.org/index.php?topic=13436.msg61861#new), I'm guessing this is related to global netmap buffer size. Looks like something changed with the new netmap.

Can you try setting hw.igb.rxd and hw.igb.txd to 1024 and see if that helps.

This is the setting which is working for us for 19.7.r1

If this works, then we'll need to calculate & adjust dev.netmap.buf_num to accommodate 4096 rx/tx descriptors.


Title: Re: Sensei on OPNsense - Application based filtering
Post by: marcri on July 10, 2019, 08:25:06 am
Hello,
is it possible to block or release single URLs from the detailed connection view independent of the category? For example, I want to block Cloudstorage completely, but allow exactly one Nextcloud URL.
Best,
Marc
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on July 10, 2019, 10:48:28 am
Hi @donatom3,

Many thanks for the heads-up.

Reading https://forum.opnsense.org/index.php?topic=13436.msg61861#new (https://forum.opnsense.org/index.php?topic=13436.msg61861#new), I'm guessing this is related to global netmap buffer size. Looks like something changed with the new netmap.

Can you try setting hw.igb.rxd and hw.igb.txd to 1024 and see if that helps.

This is the setting which is working for us for 19.7.r1

If this works, then we'll need to calculate & adjust dev.netmap.buf_num to accommodate 4096 rx/tx descriptors.

MB,

Looks like Franco saw my post and sees that a merge for the ring size didn't make it to the 19.7 netmap kernel.

https://forum.opnsense.org/index.php?topic=13436.msg61879#msg61879

He says he'll have it fixed by release.

For now I've dropped my hw.igb.rxd and txd to 1024 rebooted and it's working. Just need to remember to switch back once they fix it.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 10, 2019, 06:52:06 pm

He says he'll have it fixed by release.

For now I've dropped my hw.igb.rxd and txd to 1024 rebooted and it's working. Just need to remember to switch back once they fix it.

@donatom3, perfect. Thanks for your help. This would cause some headache.

Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 11, 2019, 01:27:59 am
is it possible to block or release single URLs from the detailed connection view independent of the category? For example, I want to block Cloudstorage completely, but allow exactly one Nextcloud URL.

Hey Marc,

Yes, it is possible. In Web Controls menu, put the whitelisted URL in a user defined custom category. And mark the category as allowed.

Than you should be good to go.

More info:

https://help.sunnyvalley.io/hc/en-us/articles/360025100393-Web-Control

Look for User Defined Categories.

Title: Sensei on OPNsense - Spelling errors
Post by: aimdev on July 12, 2019, 12:54:22 pm
Configuration, select Bridge mode.

Please select the interface paris from below boxes to create your protected L2 pridge

change paris to pairs
change pridge to bridge
Title: Enhancements?
Post by: aimdev on July 12, 2019, 12:56:23 pm
1. allow dates to be entered in european/other  format dd/mm/yyyy or dd/mon/yyyy
2. allow in user web controls Custom Web Category: google (for example) blocking by domain, ie
doubleclick.net as opposed to

stats.g.doubleclick.net
stats.i.doubleclick.net
Title: Re: Enhancements?
Post by: mb on July 12, 2019, 08:25:52 pm
change paris to pairs
change pridge to bridge

1. allow dates to be entered in european/other  format dd/mm/yyyy or dd/mon/yyyy
2. allow in user web controls Custom Web Category: google (for example) blocking by domain, ie
doubleclick.net as opposed to

stats.g.doubleclick.net
stats.i.doubleclick.net

Hi @aimdev,

Thanks for the corrections. They had been fixed for 1.0.

You should be fine putting domain.com into a user defined category and it should also match subdomain.domain.com.

Didn't it work for you?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: aimdev on July 12, 2019, 08:31:29 pm
I didn't try it as the UI seemed to intimate a site (www.google.com)   not a domain, (google.com)
Can you confirm that entering google.com will work, or does it need wildcard character/regex?
Tks
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 13, 2019, 08:43:57 pm
Hi @aimdev,

Yep, it should work that way. Just put google.com there and it'll match all subdomains.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 17, 2019, 04:29:45 am
Anyone experiencing any issues with VMware deployments?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: donatom3 on July 18, 2019, 04:14:19 am
@mb

So after the upgrade to 19.7 release I was able to change my tunables back to 4096 for rx and tx.

Here is the issue. And I've seen this on a few upgrades with no changes but firmware or sensei upgrades.

After the unit reboots after the upgrade I can reach the firewall until Sensei's engine starts. At that point it drops all traffic on my protected interfaces. I've been keeping an unprotected interface that I can easily swap to for these times. All I have to do to fix this is to disable "Enable engine heartbeat monitoring". Once I do packets start flowing again and I can re enable it without issue. I'll pull the worker logs and send them to you if that helps.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: opnip on July 18, 2019, 01:58:09 pm
After upgrade OPNsense to 19.7, Sensei shows this error on "Configuration" -> "Cloud Threat Intel"

Quote
Error at /usr/local/opnsense/mvc/app/controllers/OPNsense/Sensei/Api/EngineController.php:125 - Trying to get property 'availables' of non-object (errno=8)
Title: Re: Sensei on OPNsense - Application based filtering
Post by: malac on July 18, 2019, 05:23:53 pm
quote author=opnip link=topic=9521.msg62264#msg62264 date=1563451089]
After upgrade OPNsense to 19.7, Sensei shows this error on "Configuration" -> "Cloud Threat Intel"

Quote
Error at /usr/local/opnsense/mvc/app/controllers/OPNsense/Sensei/Api/EngineController.php:125 - Trying to get property 'availables' of non-object (errno=8)
[/quote]

same error on my setup
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 18, 2019, 07:05:00 pm
@opnip @malac, thanks for the pointer. Having a look at it.

@donatom3, please go ahead and e-mail the logs to me. Does that happen in every reboot, or was it after the 19.7 upgrade reboot?

Title: Re: Sensei on OPNsense - Application based filtering
Post by: Space on July 18, 2019, 09:05:12 pm
Hi MB,

where can I configure the retention time for the worker logs? Shouldn't they be compressed somehow?
On my system the worker logs takes about 13GB ...

Thanks and best regards,

    Space
Title: Re: Sensei on OPNsense - Application based filtering
Post by: mb on July 20, 2019, 03:05:49 am
@opnip, @malac, @space,

A hotfix release 0.8.1 is available fixing these problems. It also fixes a compatibility issue with OPNsense 19.7.

You can update your installation through Sensei -> Status -> Check for updates. An update should have been reported already.

@space, can you check whether old logs are pruned?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Space on July 20, 2019, 12:04:03 pm
@mb Thanks for the quick response ... I have updated to 0.8.1. When are the logfiles usually pruned? At some specific time?
Title: Re: Sensei on OPNsense - Application based filtering
Post by: malac on July 20, 2019, 12:18:09 pm
@opnip, @malac, @space,

A hotfix release 0.8.1 is available fixing these problems. It also fixes a compatibility issue with OPNsense 19.7.

You can update your installation through Sensei -> Status -> Check for updates. An update should have been reported already.

@space, can you check whether old logs are pruned?

great!!
does it also fix:
Scheduled reports could not be generated. Probably elasticsearch service is not running or not working properly. Please check elasticsearch service manually.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: Space on July 20, 2019, 01:25:31 pm
@mb Thanks for the quick response ... I have updated to 0.8.1. When are the logfiles usually pruned? At some specific time?

Yes, it's fixed now ... I just checked and it only kept the last 14 days ... now it's using only 2GB ...

Thanks!
Title: Re: Sensei on OPNsense - Application based filtering
Post by: biomatrix on July 21, 2019, 03:49:58 am
I just registered to post this (as opposed to on github)
the 0.8.1 hotfix fixed the first error I was having - now I get this error :

(http://i.imgur.com/z4S9ymY.png) (https://imgur.com/z4S9ymY)

my settings are :

(http://i.imgur.com/kkh7oWv.png) (https://imgur.com/kkh7oWv)


I have restarted the device - I have reset the config - I have uninstalled and reinstalled 0.8.1.

let me know if there is any other steps or information I need to proceed.

EDIT : had the #'s of the versions wrong.
Title: Re: Sensei on OPNsense - Application based filtering
Post by: malac on July 21, 2019, 12:52:16 pm
@opnip, @malac, @space,

A hotfix release 0.8.1 is available fixing these problems. It also fixes a compatibility issue with OPNsense 19.7.

You can update your installation through Sensei -> Status -> Check for updates. An update should have been reported already.

@space, can you check whether old logs are pruned?


great!!
does it also fix:
Scheduled reports could not be generated. Probably elasticsearch service is not running or not working properly. Please check elasticsearch service manually.

i still get this error after upgrading to 0.8.1 (occuring since upgrade to 19.7)