Messages

Hi @franco, thank you very much. I hope this will be of some help to the project.

We're still testing the code in HEAD. After we're confident, it's going to be MFC'd to 11-STABLE. I'll be pinging you once we're done with that. 

I've been informed that we actually have the unmodified file (.default) with the package. Engine reads a "processed" version of that file, which -indeed- do not need to be included with the package. We're removing it. I guess we're done then.

Hi @mimugmail,

Our pleasure. All welcome :) Super excited to see the changes land in 19.1.

Hi @nasq,

Any chances your LAN interface is virtio?

https://guide.sunnyvalley.io/sensei/support/faq#no-ethernet-interface-is-being-shown-in-the-interface-configuration

As quick workaround, select Intel E1000 as the adapter type.

As the final solution we're sponsoring a development which will ship the latest upstream netmap code into FreeBSD.

This will also fix lots of issues that you might be encountering with Suricata as well.

https://svnweb.freebsd.org/base?view=revision&revision=340436

It's us. Commit is done to HEAD, will be MFC'd to 11-STABLE in the following timeframe.

Hi @shrdlu,

You're in the correct place :) We're receiving feedback & comments and help requests here. You can also shoot a ticket if you think you've found an issue with the software:

https://gitlab.com/svn-community/opnsense-sensei-plugin/issues

The thing with Node.csv is not an issue. Web UI updates the contents of this file with the best servers available. I guess this creates a mismatch with the OPNsense File Integrity Checker. We'll handle that.

With regard to processing order: Sensei receives packets while they traverse from Network Adapter to the FreeBSD networking stack; which means it receives them before Squid and even before L3/L4 Filtering.

You're all welcome, and thanks for sharing your experience.

Dear Sensei users,

0.7.0-beta1 update is out for those who are on 0.6.x releases:

https://www.sunnyvalley.io/blog/0-7-beta1-update-available-for-0-6-x-users

0.7 Beta1 comes with the following functionality:

1. New Report - Blocked Connections Sessions Explorer and drill-down reporting
2. Reports enhancement: Daily executive reports. Selected reports delivered via a daily
    e-mail.
3. Customizable Landing Page for Blocked connections
4. Reports data retiring: disk space consumed by Elastic Search (Reports) is now
    configurable
5. Release Changelog is now displayed during Sensei updates
6. Shortcut to add Block/Allow rules based on fields (IP Address, Application, App
    Category etc.) via Session Explorer Reports. 
7. 350+ new applications identified.
8. Documentation: Sensei Users' Manual
9. Sensei speaks your language now, we added i18n support to match your OPNsense
    UI language. English & German are the two for now, more coming soon.
10. More performance & stability improvements


If you've downloaded & installed Sensei later than October 15, you should already be using 0.7.0 beta1. This is an update package for older versions.

Hi @wordsmith,

Many thanks for taking the time and provide this valuable feedback. Now we become aware of a communication problem.

To clarify things:

• Either Community or Enterprise Edition, Sensei core packet engine is not open source. Speaking of today,  there are no plans to make it open source. We're like Oracle running on Linux.
• Community Edition (or may be we should call it freemium edition) is and will be free of charge for the open source community. - Free as in "free beer"  ;)

As you've correctly pointed out, if there is any misunderstanding, it's unintentional. Your comments shed a lot of light as to what needs to be adjusted in the messaging. We'll be working on that.

Taking this chance, I'd like to give a little bit of background why we started with "open source firewalls".

As Sensei team, we believe that we've created a powerful packet processing technology. We believe that better packet visibility means better decision making. Better decision making means better success rates in detecting malign traffic.

Sensei is the first of two products that we're going to create for a large market.

We hope to make Sensei available for any network security equipment / product which needs application classification & web security features. L3-L4 firewalls, UTMs all fall into this category.

The thing we started with open source firewall space is that, it was a request by an MSP who was deploying open source firewalls onto customers and providing support services. Very happy with their current firewalls, they needed several features that we could provide. We quickly did an integration and voila! The resulting solution (OPNsense + Sensei) was found to be better than many of the current players in the UTM market.

This sparked a light for us. Why not deliver the product as a plugin instead of yet another full-blown firewall appliance? It'd be cost effective for us and we would than be able to relay this cost advantage for the benefit of our prospective users.

In this regard, open source firewalls is a delivery channel for us, though it's not the complete target market. Via this initial channel, we learn very much from our users and improve Sensei. You can't believe how much Sensei improved from the day we announced first beta up until this day. Then of course, we are looking for market visibility. It's great to see people loving the solution and spreading the word.

A free of charge Sensei edition (maybe we should call this freemium edition) is a way of our giving back to the OPNsense community.

Having founded a local open source community (enderunix.org) and published some open source tools, I truly understand, appreciate and respect your stance.

Though we cannot make Sensei fully open source, I think the best we can do right now is to communicate what Sensei is and what it is not in a straight and open way. This way people would know what they will have and what they won't; and will make an informed decision about using / not using it.

It's somewhat hard to figure out a way to communicate people that the current product is for "open source firewalls" without using the words "open source". Because marketing wise, we would like to be as precise as possible so that people would know what it is for.

However I also see that it's creating confusion. We'll spend more time on this. I'd also like to consult you if you wouldn't mind.

Again, many thanks for bringing this up to our attention.

Hello all,

As part of 0.7 release effort, we've launched Sensei Users' Manual & Documentation.

Please find it here:

https://guide.sunnyvalley.io/sensei/

@jjanz and community,

Elasticsearch5 was added to OPNsense packages as part of the 18.7.5 update. There was a problem in the FreeBSD elasticsearch package builds which was inherited by the OPNsense build system.

Because elasticsearch was problematic, Sensei installations were failing.

Today we fixed the problem. In the meantime, OPNsense will be removing the package from its repository in the upcoming release.

Starting 18.7.6, elasticsearch will be provided by Sunny Valley Package repository.

Long story short: We're resuming Sensei downloads. You can now download and install new Sensei version, which is 0.7.0-beta1 as of now.

Dear Sensei users,

@rhyse helping us debug his issue, we've spotted a bug with Netflow output formatter. If you're using Sensei with Netflow, better to disable it for now.

For the resolution, we'll issue a fix. Hopefully as 0.6.2.

Many thanks @rhyse !.

Hi @rhyse,

We did not have much users on VMware. Let's debug it together & make Sensei run there. I'm contacting you.

Hi @hyralak,

Many thanks for taking the time and reporting your issue. If you find value in Sensei, than it's our job to make it super stable.

Your Hardware configuration is just fine. CPU/memory utilization seems to be low & as expected.

Do you remember which Sensei version you installed first? Because the symptoms you're seeing, we had an issue which might be causing them, and was fixed at 0.6.1 release. I'm suspecting an upgrade issue.

Updates.sunnyvalley.io is being used by two purposes:

1. If you enabled Automated health-checks, it collects these info and sends them to the updates server, which we run a monitoring service with alerting capability (It's actually nagios). This way we instantly know that some Sensei instance has a problem, and try to diagnose it. Information that's sent:
    a) Check whether the packet engine is currently running
    b) Check whether the packet engine crashed and created any core files
    c) Check whether the Sensei engine has any issues with packet forwarding
    d) Check whether Elastic Search is running & healthy
    e) Check whether Sensei is utilizing any SWAP memory
    f) Check disk free space has at least 20% free.
    g) Check if Sensei is using excessive cpu/memory
    h) Check if Elastic Search is using excessive cpu/memory
    i) Check if overall load average is within safe limits
    j) Check if overall cpu/memory consumption is within safe limits
    k) Check if Sensei is put onto bypass mode because of a problem.

System health checks are done once a minute. Instead of collecting the information and sending in batch mode, health script connects to the server for every one of the checks. So this makes 11 connections for a minute. This is why you see some many connections. Yep, this is inefficient & we have an open JIRA issue to address this.

2. Software update checks. If you enable update checks, they are done once an hour.

Though the number seems to be double the number we should be seeing. Our guess is that there is a runaway cron job from previous versions.

I'd love to explore more, I'll be writing to you via a private message. I'd like to find the root cause relating to this. Than fix is the easy part :)

Hi @bulmaro,

@svn is working on your bug report. Hope to update you about this soon.

Hi there,

Sensei 0.6.1 is released. This is a minor reliability release fixing a few issued reported for 0.6 release.

• A check added to Interface Configuration menu, preventing Sensei from being assigned to an interface which is in use by Suricata. User interface now shows a Warning popup recommending a workaround to assign WAN interface to Suricata and Sensei to LAN interface. 
• In Proxmox/QEMU/KVM deployments, Sensei UI Configurator filters out virtio based ethernet interfaces. This check is to prevent an underlying problem with netmap-virtio which results in traffic forwarding to stop.
  Web Filtering: Unknown sites - sites which Sensei categorization database did not have any information for - were not filtered. A patch has been applied to fix this problem.
• Web Filtering: Undecided sites - sites which Sensei categorization database did not have the final decision for - were not filtered. A patch has been applied to fix this problem.
• If filtering was enabled for some applications, you were not able to apply the new configuration. A patch has been applied to fix this problem.

More on how to update to 0.6.1: https://www.sunnyvalley.io/blog/sensei-0-6-1

Hi @Alphakilo,

Many thanks for the input.

Currently it runs on the firewall. This was an important decision to make when we first started working on the plugin. All of the first users' feedback was to have it coupled with the firewall. Because the deployments were typical of a SOHO, SME, and they were not able to operate a separate deployment just for reporting.

So instead of starting with a distributed design, we started with this one, suggesting early users to increase the amount of memory they had. They were already using modern CPUs, so CPU was not a problem.

For a reference, with the current architecture, the largest deployment that we are reported is  700+ concurrent users and 500 Mbps/50 Mbps max, 300 Mbps sustained WAN throughput. HW: Dual-Core i5-2400 @3.10 GHz (4 threads) with 10GB RAM - OPNsense + Sensei. No IPS, No AV, No Caching. Use case is firewalling + application control + web security.

Looking forward, it looks like we'll offer this option. Since we see that more and more people want to see Sensei deployed in more large scale environments, with thousands of users.

For the time being, our focus is to have the software make super stable & make it cover the essential network security requirements of SOHO / SME users.

Hi @samsonmcnulty,

Great to hear that it worked at your second try :) Yes, the check in the installer was for 8GB minimum RAM. I guess it was something else which went wrong.