Sensei on OPNsense - Application based filtering

[mb]

@mb: is there an update about this issue?
I ran into this issue on my physical test hardware with only two ethernet ports, where one is running several VLANs. After reading your post I removed the parent adapter from the protected interfaces and it is working again. But that was on Sensei 0.8.0.beta8...
Thanks!

Hi @the-mk,

Yes, we have updates on this. Sensei is is now able to process VLAN trunk interfaces.

So, if you're using VLANs -, the latest advise is:

• Stay with the stock kernel which comes default with the OPNsense release, we need more work in new kernel with regard to netmap
• You can now protect untagged (trunk) vlan interfaces. Sensei will process both tagged and untagged frames at the same time. This is the advised & performant method.
  
• Or, you can still choose to protect vlan child interfaces or vlan parent interfaces. The important thing to be careful here is do not have them at the same time, or you'll hit a bug present in current netmap code

[mb]

Just installed 0.8.0.beta8 and did an audit of the packages and found that an outdated library is being used that is vulnerable.  Any way I can manually update this w/o breaking anything or will it be fixed in the stable release?

libXdmcp-1.1.2_2 is vulnerable:
libXdmcp -- insufficient entropy generating session keys
CVE: CVE-2017-2625
WWW: https://vuxml.freebsd.org/freebsd/1b6a10e9-4b7b-11e9-9e89-54e1ad3d6335.html

Hi rb_newbie, many thanks for pointing this out. This is a dependency package required by Elasticsearch/java. We'll go ahead and update it.

[donatom3]

@mb: is there an update about this issue?
I ran into this issue on my physical test hardware with only two ethernet ports, where one is running several VLANs. After reading your post I removed the parent adapter from the protected interfaces and it is working again. But that was on Sensei 0.8.0.beta8...
Thanks!

Hi @the-mk,

Yes, we have updates on this. Sensei is is now able to process VLAN trunk interfaces.

So, if you're using VLANs -, the latest advise is:

• Stay with the stock kernel which comes default with the OPNsense release, we need more work in new kernel with regard to netmap
• You can now protect untagged (trunk) vlan interfaces. Sensei will process both tagged and untagged frames at the same time. This is the advised & performant method.
  
• Or, you can still choose to protect vlan child interfaces or vlan parent interfaces. The important thing to be careful here is do not have them at the same time, or you'll hit a bug present in current netmap code

MB,

Are you saying if I move my 2 vlans off their own interface back to my main trunk I should stop seeing that netmap crash that was causing sensei to stop all traffic?

Sent from my Pixel 3 XL using Tapatalk

[mb]

Hi Ruud,

With beta7 i was able to add OPT1(vlan interface) to the protected interfaces. I can still do this with beta 8. What did actually change with beta8? I think I'm overlooking something.

Correct. The difference is; beta7 did not actually process tagged frames, they were just forwarded; whereas beta8 does process both tagged and untagged frames.

It would be nice to add different Sensei rules for different Vlans. I was hoping this was one of the things that changed in beta8.

We're addressing this with Policy based filtering (Interface, VLAN, Subnet based policies) which will appear in Premium subscription.

I can confirm faster DNS lookups now with cloud threat intel enabled!

Many thanks for this update. 0.8.0.beta9 should be slightly better.

[mb]

is anyone using the scheduled reports in reports&data section of the configuration (Sensei 0.7)?
Is it just me or is the "click to download and view detailed reports" link not working within the email (getting a blank html file with 0 bytes)?
It does not work for me in MS Outlook (Office 365) and Thunderbird.
If I access that mails through the webmail of my GMX (my mail provider) I can see that there's a html file attached. I can see the attachment in the app on my iPhone, but that's not my favorite "view that report" device.

Hi @the-mk,

Gmail web/iPhone looking good. It looks like a problem embedding the report for Office365/Thunderbird,

Having a look at it. Many thanks for reporting.

[the-mk]

@mb - thanks!
tested adding the trunk interface only to the protected interfaces - and it processes all VLANs that are on that trunk interface - that's ok for me!
looking forward to beta9! I guess we get a notification here in the forums as soon as it is available?
scheduled reports - the embedded report problem also exists in 0.8 beta8...

[mb]

Hi @tk-mk,

Glad to hear that vlans are working for you. beta9 is reporting vlans & interfaces. Final tests are run for it & should arrive late today (PST) or tomorrow.

Got it. Not able to make the fix for beta9, hopefully with the next beta.

[shijo]

Hi there,

Is there any possible way to block  Ultrasurf client proxy by using Sensei. Ultrasurf sets up a local proxy on the user's computer, and then configures Internet Explorer's proxy settings to run all Internet requests through that local proxy. The default port is 9666. Since the traffic between Ultrasurf and IE is entirely on the localhost, it never goes to the network and can't be blocked by a firewall. Ultrasurf then sets up an encrypted connection with a remote server in its network of proxy servers. The connection to the remote proxy server is made over port 443. Hopefully someone out there can help me with this.

Thanks in advance !  :)  :)

[mb]

Hi @shijo,

Thank you very much for trying out Sensei.

The pre-requisite for filtering an application is the identification of that application in the first place. Once its traffic is correctly identified, filtering is the easiest part.

It looks like we're not able to identify this traffic as Ultrasurf Proxy.

We've had requests for Ultrasurf and its identification is on the roadmap.

In the meantime, if you'd like to give that a pace, you can share pcap of a "test" ultrasurf session, that would be really helpful.

Then it'd be faster for us to write the signature for identifying the application.

And once it's identified, filtering is automatically in place.

[mb]

Dear Sensei users,

0.8.0.beta9 is out now. Below are the updates against 0.8.0.beta8:

Support for Large Settings (More than 1000 users)

Thanks to the newly introduced L2 Transparent Bridge Mode, you can run Sensei for thousands of users.

In this mode, where Sensei literally bridges two of your ethernet interfaces, we can scale to the number of Rx/Tx ethernet queues, thus making maximum use of the multiple CPU cores in the system.

This also helps you to keep your existing firewall and still enjoy the functionality offered by OPNsense & Sensei as an additional layer of defense.

Practically, what this means is that, if you deploy Sensei on an 8-core server with a --say 64GB of memory, you can serve 8000 users behind this configuration.

Please be noted that we'll need a small integration with OPNsense to be able to fully provide this functionality. We'll keep you posted.

Support for 4GB RAM

In an effort to be able to provide Sensei for people who have less than 8GB memory, and as per Archanfel80's suggestion, we've enabled Sensei to run for deployments with 4B of RAM.

Please note that if you have 4GB memory, maximum number of users will be 100.

Improved application signatures

• Browsec VPN
• Microsoft Updates
• Office Updates
• Fixed a bug in Web based applications classification module which -in some cases- might lead to a crash.

Cloud

New Cloud Query Infrastructure

Filtering

Fixed a bug where auto-whitelisting a host does not immediately take effect / requiring a restart of engine.

Integrations

• Improved CLI access API
• First bits of Active Directory Integration

Better Reporting

• New report: Ethernet interface reports. You can now see which ethernet interfaces carry the most bandwidth and drill down to per-interface detailed reports.
• New report: VLAN reports. You can filter out a VLAN and drill down as deep as session details.
• New report: User reports. When the OPNsense captive integration is finished, you’ll be able to view user-based reports.
• All live session reports now have VLAN, Interface, Username columns.
• All live session reports now have auto-refresh / refresh interval options
• Fixed a bug where charts were refreshed randomly causing excessive page loads
• Fixed a bug where setting Elasticsearch not to start at boot causing reporting to cease.
• Introduced an option to be able to reset all Elasticsearch Indexes.
• Introduced Elasticsearch Index Health Checker, where you can check and do a fix-up on an index basis
• Elasticsearch shards are now single. Not requiring a replica. All indexes can be seen green now.
• Fixed a bug in Elasticsearch data retiring module, which -in some cases- would result in more disk space consumption

How to update?

For 0.8 users, in the OPNsense Web UI, you should have already seen Sensei reporting 0.8.0.beta9 update. Just click on "Update" and Sensei will take care of the rest.

For 0.7 users, please wait for an announcement for 0.8.0.rc1; when it's out, you should also see 0.8 update in the OPNsense UI. We'll announce it from here and our twitter page.

Hope you enjoy this one.

--
Sensei team

[shijo]

Hi @mb,

Thank you very much for the reply. As you suggested I'm attaching the pcap file for your reference.

Thanks in advance !

[mb]

Hi @shijo,

That's awesome. Thank you. This'll help a lot.

[Archanfel80]

Im glad i can help :)

Dear Sensei users,

0.8.0.beta9 is out now. Below are the updates against 0.8.0.beta8:

Support for Large Settings (More than 1000 users)

Thanks to the newly introduced L2 Transparent Bridge Mode, you can run Sensei for thousands of users.

In this mode, where Sensei literally bridges two of your ethernet interfaces, we can scale to the number of Rx/Tx ethernet queues, thus making maximum use of the multiple CPU cores in the system.

This also helps you to keep your existing firewall and still enjoy the functionality offered by OPNsense & Sensei as an additional layer of defense.

Practically, what this means is that, if you deploy Sensei on an 8-core server with a --say 64GB of memory, you can serve 8000 users behind this configuration.

Please be noted that we'll need a small integration with OPNsense to be able to fully provide this functionality. We'll keep you posted.

Support for 4GB RAM

In an effort to be able to provide Sensei for people who have less than 8GB memory, and as per Archanfel80's suggestion, we've enabled Sensei to run for deployments with 4B of RAM.

Please note that if you have 4GB memory, maximum number of users will be 100.

Improved application signatures

• Browsec VPN
• Microsoft Updates
• Office Updates
• Fixed a bug in Web based applications classification module which -in some cases- might lead to a crash.

Cloud

New Cloud Query Infrastructure

Filtering

Fixed a bug where auto-whitelisting a host does not immediately take effect / requiring a restart of engine.

Integrations

• Improved CLI access API
• First bits of Active Directory Integration

Better Reporting

• New report: Ethernet interface reports. You can now see which ethernet interfaces carry the most bandwidth and drill down to per-interface detailed reports.
• New report: VLAN reports. You can filter out a VLAN and drill down as deep as session details.
• New report: User reports. When the OPNsense captive integration is finished, you’ll be able to view user-based reports.
• All live session reports now have VLAN, Interface, Username columns.
• All live session reports now have auto-refresh / refresh interval options
• Fixed a bug where charts were refreshed randomly causing excessive page loads
• Fixed a bug where setting Elasticsearch not to start at boot causing reporting to cease.
• Introduced an option to be able to reset all Elasticsearch Indexes.
• Introduced Elasticsearch Index Health Checker, where you can check and do a fix-up on an index basis
• Elasticsearch shards are now single. Not requiring a replica. All indexes can be seen green now.
• Fixed a bug in Elasticsearch data retiring module, which -in some cases- would result in more disk space consumption

How to update?

For 0.8 users, in the OPNsense Web UI, you should have already seen Sensei reporting 0.8.0.beta9 update. Just click on "Update" and Sensei will take care of the rest.

For 0.7 users, please wait for an announcement for 0.8.0.rc1; when it's out, you should also see 0.8 update in the OPNsense UI. We'll announce it from here and our twitter page.

Hope you enjoy this one.

--
Sensei team

[malac]

Hi updated from beta8 to 9, everythings looks fine so far.
Also local DNS an Cloud Threat Intel is working, GREAT!

Only: I cannot set deployment size, drop down is empty....but thats it

[hbc]

Im glad i can help :)

How does it help to just quote the complete previous text without any sensful addition?  ::)