Sensei on OPNsense - Application based filtering

[mb]

Dear Sensei users,

We know you’re looking forward to seeing 0.7.0 release. We also do indeed.

Yet, we decided to ship another release candidate before the actual release because some updates to the code base might have more impact than we originally planned. These code updates are preliminary work related to an effort to minimize external library dependencies and compiling Sensei engine as a Position Independent Executable (PIE).

Minimizing external library dependencies will allow Sensei to be able to run on embedded platforms which run on very low resources.

PIE is a nice feature which will be default for OPNsense@HardenedBSD and will provide mitigation capabilities against exploit attempts to the packet engine. (Note: PIE is not enabled yet)

So there we have 0.7.0-rc3 publicly available for you to test. This is the Changelog from rc2 to rc3:

New features (from 0.7.0-rc2 to 0.7.0-rc3).
* More lightweight core packet engine
* Option to delete all reporting data
* Mobile web browsers compatibility. You’ll be able to view Sensei reports through a mobile device.
* Prevented scheduled jobs from submitting unnecessary emails.
* HW requirements check has been made available for the UI initial configuration wizard.
* Some stability improvements. 

0.7.0-rc3 has been under testing for about a week now, but if you’re running Sensei on a more production like environment, you might want to wait till we ship 0.7.0 final release, which should arrive in a week if we do not see any issues with 0.7.0-rc3.

To update to 0.7.0-rc3, login to OPNsense UI, navigate to Sensei -> Status and click Check for Updates. You should see an update reported. Click Update to proceed with the update. Sensei updater should take care of the rest.

Best
Sensei team

[cgwork]

Great News mb,

In my personal opinion RC (Release Candidate) are like the actual gold image, as it progress and other clients testing it will become better with the final release.

[manjeet]

Hello MB, I can see the option in "Table of local / remote assets" to select different top users. Can you also add another option to sort it ascending or descending so that we can check the top user in top list rather then going to the entire list to find one.

[Antaris]

Another idea about "Session details": give the user ability to restrict begin and end date and time fields to reduce search results to concrete time period.

[mb]

@cgwork, @manjeet, @Antaris,

Many thanks for the suggestions. Feature requests have been added to 0.8 workload. We'll do a more general re-visit to table reports. Please feel free to reach out for more ideas.

[the-mk]

a question from a maybe future sensei user:
since this elastic search module needs a lot of diskspace and sure does a lot of writing - is there a possibility to divide the installation into an "OS"-disk (binaries; usually on a SSD) and a "data"-disk (storage intensive data, lots of writes; usually on a HDD)?

[nikkon]

Hi all,

After upgrading to version 0.7.0-rc3 none of my dashboards or reports are loading anymore

That's an error example:
{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "resource.type": "index_or_alias",
        "resource.id": "conn_all",
        "index_uuid": "_na_",
        "index": "conn_all"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index",
    "resource.type": "index_or_alias",
    "resource.id": "conn_all",
    "index_uuid": "_na_",
    "index": "conn_all"
  },
  "status": 404
}

Any clue?

[manjeet]

Thanks @MB for considering this.

I have an another thing to ask. I am not if that is 100% possible or is it already implemented because i did not find it in any details.

In report we can see the source address, destination address or host, app category and protocol it is used. It gives us huge information about who has download / uploaded to where and how much data, also time stamp of session etc. But i do not see any ways to check what exactly the user has downloaded. For e.g one of my user used 5GB data in one day which is used by google services and it gives us the list of when and where, but no info about what exactly which for now we have to ask the user. This could be useful because if user is downloading / uploading something not allowed to server / account which they are allowed to access then they probably will deny it.

Also can you add option to export reports (excel or pdf) including custom / filtered reports so that we can provide report to management whenever needed rather then filling mail box with auto reporting.

[maekar]

Hi,

Is there anything special to do with VLAN?

We have interfaces tagged and untagged. When I activated Sensei and configured just a few web categories to test, everything worked well with the untagged interface but all VLAN networks lost connectivity, devices in all VLAN not even get IP address by DHCP. And the problem persisted even when I deselected those interfaces to get managed by Sensei, I had to stop it and uninstall it to get VLAN networks working again.

Thanks!

[mb]

Hi @the-mk,

Thank you very much for the suggestion: We get this request quite many times. People who’d like to see this functionality seem to be either running on the low end - the device is very weak and lack the resources to run reporting on the device itself, or they run on the high end - throughput & number of users are quite high (>1K users) and it makes  sense to put reporting on a separate device.

In addressing this requirement, we’ll offer an option - in the initial configuration wizard - asking the user whether s/he wants the reporting on the device itself, or on a remote server.

[mb]

Hi @nikkon,

Looks like alias indexes are messed up. By any chance, did you do any "reset to factory defaults" ?

We'd like to dig deeper. Can you share your /var/log/elasticsearch/elasticsearch-2018-12-19.log through sensei - at - sunnyvalley.io ?

For a workaround, you can run these two commands to reset the indexes: (beware: this will erase your reporting history)

/usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
/usr/local/sensei/scripts/installers/elasticsearch/create_indices.py

Let us know if this does not fix the problem.

[mb]

@manjeet, you’re all welcome.

If the connection is clear-text (e.g. HTTP), you can see the individual downloaded files from Web Reports: Web - Table of URIs. For the TLS encrypted sessions (e.g. HTTPS), this will be possible with the all ports TLS Inspection feature - though it’s going to be available for Premium Subscriptions.

For the Table reports, development & tests have been completed, and it’s ready to ship with 0.7.0 release.
I’ve sent you a link today to try it and see if there are any more issues.

Reports - PDF export - its’ on the short term roadmap. Probably it will ship with 0.8.

[mb]

Hi @maekar,

Thanks for reporting this. Yes, we’re aware of this problem. Unfortunately part of the solution required some development on the Operating System itself (FreeBSD netmap implementation).

Good news is that hopefully it’ll be fixed with OPNsense 19.1. On the FreeBSD side, we’ve sponsored a development which fixes this and some other issues with the netmap implementation on FreeBSD (https://svnweb.freebsd.org/base?view=revision&revision=340436)

We’ve been testing the 11.2-STABLE MFC code for some time and it looks good to be finally integrated with OPNsense.

We’re working very closely with the OPNsense team on this. I’ll be posting an ETA after we sync with @franco.

[nikkon]

@mb thanks for replying
I did execute the 2 scripts.

please check the log below:

cat /var/log/elasticsearch/elasticsearch-2018-12-
elasticsearch-2018-12-16.log  elasticsearch-2018-12-20.log
root@Skynet:~ # cat /var/log/elasticsearch/elasticsearch-2018-12-20.log
[2018-12-20T01:05:36,849][INFO ][o.e.n.Node               ] [yCObJMR] stopping ...
[2018-12-20T01:05:36,889][INFO ][o.e.n.Node               ] [yCObJMR] stopped
[2018-12-20T01:05:36,889][INFO ][o.e.n.Node               ] [yCObJMR] closing ...
[2018-12-20T01:05:36,911][INFO ][o.e.n.Node               ] [yCObJMR] closed
[2018-12-20T01:07:19,550][INFO ][o.e.n.Node               ] [] initializing ...
[2018-12-20T01:07:19,707][INFO ][o.e.e.NodeEnvironment    ] [yCObJMR] using [1] data paths, mounts [[/var (tmpfs)]], net usable_space [1.9gb], net total_space [2.4gb], spins? [unknown], types [tmpfs]
[2018-12-20T01:07:19,707][INFO ][o.e.e.NodeEnvironment    ] [yCObJMR] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-12-20T01:07:19,711][INFO ][o.e.n.Node               ] node name [yCObJMR] derived from node ID [yCObJMRsQcSMKeQy7KNhyA]; set [node.name] to override
[2018-12-20T01:07:19,711][INFO ][o.e.n.Node               ] version[5.6.8], pid[32322], build[688ecce/2018-02-16T16:46:30.010Z], OS[FreeBSD/11.1-RELEASE-p17/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_172/25.172-b11]
[2018-12-20T01:07:19,711][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/local/lib/elasticsearch]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [aggs-matrix-stats]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [ingest-common]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-expression]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-groovy]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-mustache]
[2018-12-20T01:07:21,817][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [lang-painless]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [parent-join]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [percolator]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [reindex]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [transport-netty3]
[2018-12-20T01:07:21,818][INFO ][o.e.p.PluginsService     ] [yCObJMR] loaded module [transport-netty4]
[2018-12-20T01:07:21,819][INFO ][o.e.p.PluginsService     ] [yCObJMR] no plugins loaded
[2018-12-20T01:07:25,240][INFO ][o.e.d.DiscoveryModule    ] [yCObJMR] using discovery type [zen]
[2018-12-20T01:07:26,419][INFO ][o.e.n.Node               ] initialized
[2018-12-20T01:07:26,420][INFO ][o.e.n.Node               ] [yCObJMR] starting ...
[2018-12-20T01:07:26,927][INFO ][o.e.t.TransportService   ] [yCObJMR] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-12-20T01:07:30,078][INFO ][o.e.c.s.ClusterService   ] [yCObJMR] new_master {yCObJMR}{yCObJMRsQcSMKeQy7KNhyA}{QHCtod64RcOkM74GkkvW-g}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-12-20T01:07:30,120][INFO ][o.e.h.n.Netty4HttpServerTransport] [yCObJMR] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-12-20T01:07:30,121][INFO ][o.e.n.Node               ] [yCObJMR] started
[2018-12-20T01:07:30,140][INFO ][o.e.g.GatewayService     ] [yCObJMR] recovered

• indices into cluster_state

in Gui i got this:
Error at /usr/local/opnsense/mvc/app/models/OPNsense/Sensei/Sensei.php:74 - fsockopen(): unable to connect to 127.0.0.1:4343 (Operation timed out) (errno=2)

[mb]

Hi @Nikkon,

Is this the log after you executed the delete/create scripts, or the one with the errors?

Looks like the former? Did the scripts resolve the problem?