Sensei on OPNsense - Application based filtering

[mb]

Hi @Alphakilo,

Many thanks for the input.

Currently it runs on the firewall. This was an important decision to make when we first started working on the plugin. All of the first users' feedback was to have it coupled with the firewall. Because the deployments were typical of a SOHO, SME, and they were not able to operate a separate deployment just for reporting.

So instead of starting with a distributed design, we started with this one, suggesting early users to increase the amount of memory they had. They were already using modern CPUs, so CPU was not a problem.

For a reference, with the current architecture, the largest deployment that we are reported is  700+ concurrent users and 500 Mbps/50 Mbps max, 300 Mbps sustained WAN throughput. HW: Dual-Core i5-2400 @3.10 GHz (4 threads) with 10GB RAM - OPNsense + Sensei. No IPS, No AV, No Caching. Use case is firewalling + application control + web security.

Looking forward, it looks like we'll offer this option. Since we see that more and more people want to see Sensei deployed in more large scale environments, with thousands of users.

For the time being, our focus is to have the software make super stable & make it cover the essential network security requirements of SOHO / SME users.

[mb]

Hi there,

Sensei 0.6.1 is released. This is a minor reliability release fixing a few issued reported for 0.6 release.

• A check added to Interface Configuration menu, preventing Sensei from being assigned to an interface which is in use by Suricata. User interface now shows a Warning popup recommending a workaround to assign WAN interface to Suricata and Sensei to LAN interface. 
• In Proxmox/QEMU/KVM deployments, Sensei UI Configurator filters out virtio based ethernet interfaces. This check is to prevent an underlying problem with netmap-virtio which results in traffic forwarding to stop.
  Web Filtering: Unknown sites - sites which Sensei categorization database did not have any information for - were not filtered. A patch has been applied to fix this problem.
• Web Filtering: Undecided sites - sites which Sensei categorization database did not have the final decision for - were not filtered. A patch has been applied to fix this problem.
• If filtering was enabled for some applications, you were not able to apply the new configuration. A patch has been applied to fix this problem.

More on how to update to 0.6.1: https://www.sunnyvalley.io/blog/sensei-0-6-1

[bulmaro]

Hi friends, thanks for the very interesting project work,
I'm testing version 0.6.1, my interface is vlan but I do not see Packets IN and Packets OUT, any settings I missed?

[mb]

Hi @bulmaro,

@svn is working on your bug report. Hope to update you about this soon.

[bulmaro]

thanks for your attention

[hyralak]

I tested Sensei for a couple weeks. In that time I observed some unexpected behavior. First i need to say that I have had zero issues with opnsense in the year that i have been running it, rock solid. I am running it at home, my internet speed is 300/80. The hardware is a Dell Optiplex 8gb ram Intel(R) Core(TM) i5-3475S CPU @ 2.90GHz. Memory usage never exceeded 35% with sensei running and cpu usage was minimal. 
Issues I encountered after installing Sensei included web interface locking up, and unable to access opnsesne via ssh. I could still interact with the console. After this occurred i had to uninstall the plugin.

Also, I run a pi-hole for DNS poisoning which logged Sensei as the top domain. I was seeing 25,000-35,000 connection attempts to updates.sunnyvalley.io. I turned off auto updates but it continued to hammer away at updates.sunnyvalley.io. The screenshot below is from the last 24 hours. I uninstalled Sensei about 13 hours ago.



I liked the visibility and functionality that Sensei offered, but the instability was not acceptable. Perhaps my hardware is not adequate for the plugin?
Hopefully the information that the plugin sent back to sunnyvalley will provide insight into my web/ssh issues.
Keep up the great work and thanks for letting me try out the plugin. Perhaps I will try again at a later date.

[mb]

Hi @hyralak,

Many thanks for taking the time and reporting your issue. If you find value in Sensei, than it's our job to make it super stable.

Your Hardware configuration is just fine. CPU/memory utilization seems to be low & as expected.

Do you remember which Sensei version you installed first? Because the symptoms you're seeing, we had an issue which might be causing them, and was fixed at 0.6.1 release. I'm suspecting an upgrade issue.

Updates.sunnyvalley.io is being used by two purposes:

1. If you enabled Automated health-checks, it collects these info and sends them to the updates server, which we run a monitoring service with alerting capability (It's actually nagios). This way we instantly know that some Sensei instance has a problem, and try to diagnose it. Information that's sent:
    a) Check whether the packet engine is currently running
    b) Check whether the packet engine crashed and created any core files
    c) Check whether the Sensei engine has any issues with packet forwarding
    d) Check whether Elastic Search is running & healthy
    e) Check whether Sensei is utilizing any SWAP memory
    f) Check disk free space has at least 20% free.
    g) Check if Sensei is using excessive cpu/memory
    h) Check if Elastic Search is using excessive cpu/memory
    i) Check if overall load average is within safe limits
    j) Check if overall cpu/memory consumption is within safe limits
    k) Check if Sensei is put onto bypass mode because of a problem.

System health checks are done once a minute. Instead of collecting the information and sending in batch mode, health script connects to the server for every one of the checks. So this makes 11 connections for a minute. This is why you see some many connections. Yep, this is inefficient & we have an open JIRA issue to address this.

2. Software update checks. If you enable update checks, they are done once an hour.

Though the number seems to be double the number we should be seeing. Our guess is that there is a runaway cron job from previous versions.

I'd love to explore more, I'll be writing to you via a private message. I'd like to find the root cause relating to this. Than fix is the easy part :)

[hyralak]

It appears that I installed sensei_installer_opnsense_0.6.1-release.sh

[rhyse]

Hi

I am seeing an issue where the "Sensei Packet Engine" keeps stopping, clicking start makes it come back to life.

Enviro: VMware 6.7, 10GB RAM, 2 x vCPU's (host CPU 2 x  E5-2670) , disk space 2.2 gb used out of 18Gb , Sensie deployment size Small (I have just enabled " Enable Generation of Support Data:" ), Sensei version 0.6.1-release (installed from this version)

This is a test infra, so doesn't have much traffic going through it

Any ideas ?

Thanks

[mb]

Hi @rhyse,

We did not have much users on VMware. Let's debug it together & make Sensei run there. I'm contacting you.

[mb]

Dear Sensei users,

@rhyse helping us debug his issue, we've spotted a bug with Netflow output formatter. If you're using Sensei with Netflow, better to disable it for now.

For the resolution, we'll issue a fix. Hopefully as 0.6.2.

Many thanks @rhyse !.

[Csykes27]

I am having an issue of when I Enable Cloud Reputation & Web Categorization all web traffic stops. all services are running and stay running from what I can tell.

[svn]

@Csykes27 thanks for reporting. We've heard this issue for the first time actually. Let's debug what is causing this together.

I shall be contacting you soon to resolve the issue.

[jjanzz]

During the initial installation, a dependency throws a 404 error:

Code Select Expand

pkg: https://updates.sunnyvalley.io/repo/libXtst-1.2.3.txz: Not Found
FAILED : Unable to install required packages. Please see install.log

[svn]

Hi @jjanzz,

We're currently working with OPNsense for the integration. As a part of this process, OPNsense started providing some of our dependencies. We'll update our installer and package to be compatible with the latest changes on OPNsense package repository, soon.

It seems that some of the dependencies not satisfied (namely, some configuration files of elasticsearch, and some java dependencies). We'll fix this urgently.

Right now, you can register for download and we'll send you a download link as soon as we fix the dependency issue.