Sensei on OPNsense - Application based filtering

[mb]

Hi @bulmaro,

The reason is most likely Elasticsearch consuming all memory and OS begins swapping. When the OS does swapping overall system performance is significantly degraded and this in turn affects Sensei doing its job.

To avoid a connectivity problem, we shut down Sensei with a warning like this (numbers seem weird, need to look at that)

How many devices do you have behind sensei and what is your hardware configuration?

https://help.sunnyvalley.io/hc/en-us/articles/360025047373-Hardware-Requirements

This will give you an overview of the recommended HW configuration according to the size of your deployment.

[bulmaro]

I have two servers
physical equipment with 30 connected clients, equipment characteristics:
CPU 3-2105 CPU @ 3.10GHz (4 cores)
RAM memory: 8GB

Azure server, 3 clients connected
CPU E5-2673 v4 @ 2.30GHz (2 cores)
4GB RAM

it's exactly the same message for both

[mb]

@bulmaro,

Thanks for the swift reply. These configurations look perfectly ok for the deployment size. Let me reach out to you; and we can have a look together.

[mb]

These configurations look perfectly ok for the deployment size. Let me reach out to you; and we can have a look together.

Dear Sensei users,

Out of @bulmaro's case, I think it's important to give a heads-up on this:

The hardware recommendation we provide is calculated based on the fact that the system runs OPNsense with Sensei. We did not take other services which might be already running on the firewall (IDS, Proxy etc.) into consideration.

We highly recommend that you also oversee the requirement of those services and do your own sizing according to that.

In @bulmaro's specific case, 1/2 of the memory was already consumed by the squid service. And the system was swapping even Sensei and Elasticsearch were not active.

@bulmaro, many thanks for your help to diagnose the issue.

[donatom3]

@MB

Not sure if this has ever been brought up. It's something I've seen for a while.

On any of the live session explorers or drill down of traffic if I do a whois for the record that is the domain name it always only resolves the top level domain. For example US.lgtvsdp.com does a whois for domain COM thus always giving me the same result for any .com address.

Shouldn't it be doing the whois query on the second level domain?

[manjeet]

Hi @MB,

Few days back we had power issue and after that "Elasticsearch" is not working. I have tried start the service many times, rebooted and tried but didn't work. "Sensei Packet Engine" is working.

I have tried "Perform health check for indices" and it kind of stuck and does not do anything. "You can erase reporting data" option is grayed out. I also tried to run these command from terminal and got the error:
1. /usr/local/sensei/scripts/installers/elasticsearch/delete_all.py
2. /usr/local/sensei/scripts/installers/elasticsearch/create_indices.py
ERROR: ***ERROR: Connection could not be established with elasticsearch server.**

Also tried reset the package but it didn't fix the issue. Haven't delete / uninstall and reinstall the package yet. kindly help.

[mb]

@donatom3, checking that one.

@manjeet, "Reset reporting" will be enabled even if Elasticsearch is not running. Fixing for 1.0.

[thg0432]

@mb

I was wondering if there's a setting for rotating the logs that are in /usr/local/sensei/log/archive ?  or is that something that needs to be cleaned out manually? 

[mb]

@donatom3,

You're right. Currently we run the whois query the for the whole FQDN. We should be doing for the domain part only. Fix is implemented today and shipping with 1.0.

@thg0432,

Engine logs older than two weeks are to be automatically deleted. 0.8 had a glitch doing the actual delete. Fix is implemented for 1.0.

In the meantime, you can get rid of them by running this command:

find /usr/local/sensei/log/archive -type f -mtime +15d  | xargs rm -f {}\;

[zyon]

Just installed Sensei and just awesome.
All i need in one application :)

For my information sensei work with squid ? if yes it's possible to use it like a proxy server ? ( for mobile for example )

Thanks for your hard work :)

[mb]

hi @zyon,

Thanks for your feedback. Glad that you found Sensei useful for you. All welcome.

Sensei plugs kind of transparent to the system. So it does not change the way other services like Squid are operating.

I think I did not completely get your question.

Do you want to learn if Sensei can act like a proxy, for instance, for caching?

[manjeet]

Hello @MB, Is there any way to bypass a user from sensei filter
OR
More accurately for my case, bypass anyone which goes from a particular gateway.

Actually, i have 2 ISPs which are in load balancing mode on opnsense, i want anyone connected to gateway 2 to just bypass any filters or blocking or logging.

[mb]

Hi @manjeet,

Actually, i have 2 ISPs which are in load balancing mode on opnsense, i want anyone connected to gateway 2 to just bypass any filters or blocking or logging.

I believe - in your case - the outbound route selection is done randomly and not through a policy decision based on source IP address, am I correct?

If that is so,  and it's not something related to the source IP/network address, I'm afraid there is no way we can correlate the user with the outbound ISP. This is because we jump into the scene way too early, without routing/NAT'ing logic comes into the scene.

If it's source IP related, it's possible, and along with user/group based filtering, this is one of the features of the premium edition:

https://help.sunnyvalley.io/hc/en-us/articles/360025173953-Sensei-Editions

[donatom3]

@mb

This probably isn't Sensei since it affects Suricata to. But since I upgraded to 19.7 RC1 suricata won't start because it can't find my interface and Sensei says no interface selected in the status page. I can change to any of my interfaces and they all say the same.

Here is a portion of the worker thread when I started this morning. From it it looks like everything points to netmap being the issue. I started a thread in the 19.7 release candidate forums about it. Just a warning to anyone relying on Sensei.

Code Select Expand

2019-07-09T07:38:49 INFO: Packet Processor [39794] started working
2019-07-09T07:38:49 INFO: Worker [pid:39794] Pinning to CPU #1
2019-07-09T07:38:49 INFO: Worker [39794] started working
2019-07-09T07:38:49 INFO: License file /usr/local/sensei//etc//license.data not located (No such file or directory) assuming FREEMIUM
2019-07-09T07:38:49 INFO: Created Syn Filter Context Table [mask: 16383]
2019-07-09T07:38:49 INFO: Created a new Worker Instance pid: 39794
2019-07-09T07:38:49 INFO: Requested Single Threaded Stack
2019-07-09T07:38:49 INFO: Inline operation mode selected! Bridging br1 (netmap@igb1 <-> netmap@igb1^)
2019-07-09T07:38:50 INFO: Created Enrichment Service @127.0.0.1:4343
2019-07-09T07:38:50 WARNING: loadUserCache: file /usr/local/sensei//userdefined/db/Usercache//userauth_cache.db is not a regular file
2019-07-09T07:38:50 INFO: Number of Queues for interface: igb1: 2
2019-07-09T07:38:50 INFO: LAN: igb1[igb1] Queue: 0, #Queues: 2, Packet Device: Netmap
2019-07-09T07:38:50 INFO: WAN: igb1^[igb1], Queue: 0, #Queues: 1, Packet Device: Netmap-Host-Bridge
2019-07-09T07:38:50 INFO: Initializing for BRIDGE Mode
2019-07-09T07:38:50 CRITICAL: Failed to create LAN interface (igb1:0(igb1:0): 6(Device not configured)
2019-07-09T07:38:50 ERROR: Failed Initializing Interfaces, bailing out
2019-07-09T07:38:51 INFO: Packet Processor [19965] started working
2019-07-09T07:38:51 INFO: Packet Processor [19965] sleeping a while since we're respawned
2019-07-09T07:39:03 INFO: Worker [pid:19965] Pinning to CPU #1
2019-07-09T07:39:03 INFO: Worker [19965] started working
2019-07-09T07:39:03 INFO: License file /usr/local/sensei//etc//license.data not located (No such file or directory) assuming FREEMIUM
2019-07-09T07:39:03 INFO: Created Syn Filter Context Table [mask: 16383]
2019-07-09T07:39:03 INFO: Created a new Worker Instance pid: 19965
2019-07-09T07:39:03 INFO: Requested Single Threaded Stack
2019-07-09T07:39:03 INFO: Inline operation mode selected! Bridging br1 (netmap@igb1 <-> netmap@igb1^)
2019-07-09T07:39:04 INFO: Created Enrichment Service @127.0.0.1:4343
2019-07-09T07:39:04 WARNING: loadUserCache: file /usr/local/sensei//userdefined/db/Usercache//userauth_cache.db is not a regular file
2019-07-09T07:39:04 INFO: Number of Queues for interface: igb1: 2
2019-07-09T07:39:04 INFO: LAN: igb1[igb1] Queue: 0, #Queues: 2, Packet Device: Netmap
2019-07-09T07:39:04 INFO: WAN: igb1^[igb1], Queue: 0, #Queues: 1, Packet Device: Netmap-Host-Bridge
2019-07-09T07:39:04 INFO: Initializing for BRIDGE Mode
2019-07-09T07:39:04 CRITICAL: Failed to create LAN interface (igb1:0(igb1:0): 6(Device not configured)
2019-07-09T07:39:04 ERROR: Failed Initializing Interfaces, bailing out
2019-07-09T07:39:05 INFO: Packet Processor [18480] started working
2019-07-09T07:39:05 INFO: Packet Processor [18480] sleeping a while since we're respawned
2019-07-09T07:39:17 INFO: Worker [pid:18480] Pinning to CPU #1
2019-07-09T07:39:17 INFO: Worker [18480] started working
2019-07-09T07:39:17 INFO: License file /usr/local/sensei//etc//license.data not located (No such file or directory) assuming FREEMIUM
2019-07-09T07:39:17 INFO: Created Syn Filter Context Table [mask: 16383]
2019-07-09T07:39:17 INFO: Created a new Worker Instance pid: 18480
2019-07-09T07:39:17 INFO: Requested Single Threaded Stack
2019-07-09T07:39:17 INFO: Inline operation mode selected! Bridging br1 (netmap@igb1 <-> netmap@igb1^)
2019-07-09T07:39:18 INFO: Created Enrichment Service @127.0.0.1:4343
2019-07-09T07:39:18 WARNING: loadUserCache: file /usr/local/sensei//userdefined/db/Usercache//userauth_cache.db is not a regular file
2019-07-09T07:39:18 INFO: Number of Queues for interface: igb1: 2
2019-07-09T07:39:18 INFO: LAN: igb1[igb1] Queue: 0, #Queues: 2, Packet Device: Netmap
2019-07-09T07:39:18 INFO: WAN: igb1^[igb1], Queue: 0, #Queues: 1, Packet Device: Netmap-Host-Bridge
2019-07-09T07:39:18 INFO: Initializing for BRIDGE Mode
2019-07-09T07:39:18 CRITICAL: Failed to create LAN interface (igb1:0(igb1:0): 6(Device not configured)
2019-07-09T07:39:18 ERROR: Failed Initializing Interfaces, bailing out


[mb]

Hi @donatom3,

Many thanks for the heads-up.

Reading https://forum.opnsense.org/index.php?topic=13436.msg61861#new, I'm guessing this is related to global netmap buffer size. Looks like something changed with the new netmap.

Can you try setting hw.igb.rxd and hw.igb.txd to 1024 and see if that helps.

This is the setting which is working for us for 19.7.r1

If this works, then we'll need to calculate & adjust dev.netmap.buf_num to accommodate 4096 rx/tx descriptors.