Sensei on OPNsense - Application based filtering

[Antaris]

Hi faisal,

Than it must be the cpu score. There is a 300.000 minimum cpu score requirement for Elasticsearch.

Here's  a quick hack:

1. Remove /usr/local/sensei/etc/.configdone

Code: [Select]

rm /usr/local/sensei/etc/.configdone
3. Edit /usr/local/opnsense/scripts/OPNsense/Sensei/check_hardware.sh file and locate these lines:

Code: [Select]

if [ $CPU_SCORE -le 300000 ]; then
       CPU_PROPER="false"
else
       CPU_PROPER="true"
fi

Change 300000 to a lower value, like 200000. 

4. Do a browser refresh on the OPNsense UI, and click on any sensei menu. It'll re-run the config wizard. Now it should select Elasticsearch.

Now I'm thinking: for cpu scores between 200K and 300K and if there is enough memory (>=8GB) I think we should let the user decide on the database backend.

This solution no longer works on fresh install today. And i can't find from where to choose Elastic engine...

[mb]

Hi @Antaris,

This looks good and should've worked. But with 1.5 database selection will be optional if the device has enough memory but weak cpu (e.g. 200.000<>300.000 cpu score).

We hope to release 1.5 late this month.

By the way, I think this was your request, you can now request re-classification for a web site through Sunny Valley website

https://www.sunnyvalley.io/site-classification/

[Antaris]

Hi @Antaris,

This looks good and should've worked. But with 1.5 database selection will be optional if the device has enough memory but weak cpu (e.g. 200.000<>300.000 cpu score).

We hope to release 1.5 late this month.

By the way, I think this was your request, you can now request re-classification for a web site through Sunny Valley website

https://www.sunnyvalley.io/site-classification/

Hi @mb,

Looking forward to 1.5 and thx for the classification option.

[deibit]

Hi,

I have just installed OPNsense after many (more than i recall) years using PfSense. One of the first things I installed is Sensei.. I liked it so much that I even paid for the home license

I run OPNsense in a SC813 Supermicro Rack, with a X10SLM+LN4F MoBo, 32GB DDR-1600 ECC RAM, a Xeon E3-1230L V3 CPU and a 2TB Seagate Ironwolf HDD (all baremetal, not virtualized).

I use a single 1.000/50 Mbit/s WAN, not many simultaneous webusers (4 or 5) but heavy traffic on the 4 VPN tunnels (fileserver)

The CPU gives 267.081 Single CPU Ubench index. So it's underpowered according to Sensei/Elasticsearch standards.

I guess I wouldn't have any problem in upgrading the CPU, but why is the single core performance so important? ElasticSearch s supposed to take advange of multiple cores isn't it? (CPU has 4 cores and 8 threads)

Should I be worried? Where can I look if my CPU is struggling? Everything "feels" good so far...

[Antaris]

Hi @deibit,

I think every Intel Core with AES and 4 or more cores @3GHz or more is OK for OPNsense with Sensei and Elastic. In the incoming 1.5 you will have the option to choose backend database manually and this misunderstanding will be solved. May be it's good to upgrade cpu to non-L Haswell or Broadwell cpu for better VPN throughput. Also you have to know that for some strange reason Ubench rates Haswell Xeons way lower than non-xeon i5 on same clocks...

[Quetschwalze]

@mb sensei is still running very smooth for me
Any news/eta on those botnet and DNS tunneling features already shown in the policy?

Gesendet von meinem MI 9 mit Tapatalk

[deibit]

Hi @deibit,

I think every Intel Core with AES and 4 or more cores @3GHz or more is OK for OPNsense with Sensei and Elastic. In the incoming 1.5 you will have the option to choose backend database manually and this misunderstanding will be solved. May be it's good to upgrade cpu to non-L Haswell or Broadwell cpu for better VPN throughput. Also you have to know that for some strange reason Ubench rates Haswell Xeons way lower than non-xeon i5 on same clocks...

I "upgraded" to a E3-1268L that I had here "laying around". Now the ubench score is in the 342.000 range, I don't think it makes a big difference but my karma is again in balance due to sensei not complaining about my router being lower end

I still wonder why the single core performance is so important for elasticsearch though...

[bEeReE]

@mb sensei is still running very smooth for me
Any news/eta on those botnet and DNS tunneling features already shown in the policy?

Would be also great if we can get a little bit more background how the sensei cloud works. Is it feeded with external sources (e.g. phishtank, malware companies etc.) so that we can avoid duplicated filtering on dns, proxy or ICAP level?
Furthermore, I am wondering how often decisions about security categorization like "undecided safe sites" are made. I believe there are some blocked domains hanging in such status for more than 1 week. Shouldn't this be faster?

[scrensen]

I've searched the forums as well as Google but didn't find the answer to my issue. Which is that during Sensei installation I get the following message: "Oops, it looks like LAN interface is also in use by Suricata"

But I do not have Suricata running, Intrusion detection is completely disabled. I did have it running few weeks back, but disabled it a week ago. So to be sure I rebooted my router/fw before installing Sensei, but still same message.

I did a quick search via the command line for Suricate config files to check for interface config, but didn't find anything useful.

Anyone that might be able to help me out here?

Thanks in advance!

[scrensen]

during Sensei installation I get the following message: "Oops, it looks like LAN interface is also in use by Suricata"

But I do not have Suricata running

I still decided to check the Captive Portal settings and I removed LAN from the interfaces and applied. And that actually solved the issue. Even though it was not active....

[mb]

@Quetschwalze, glad that it's working good.

We're investing heavily on application, threat intelligence and security databases. With 1.5 version, you'll start seeing weekly / daily database updates. Up until now databases were being updated with every release.

With regard to Botnet filtering / DNS Tunnel filtering, we expect to land them in Q3 this year.

@bEeReE, we are adding more information to our documentation about the cloud architecture. We'll make it available early April.

[mb]

I still wonder why the single core performance is so important for elasticsearch though...

Hi @deibit, glad that Sensei is happy with you hardware now.

As for Elasticsearch, you are right, Elasticsearch is multi-threaded and will benefit from multi-core cpus.

For the packet engine itself, for each interface, it has a worker process and workers are currently running single threaded; so each worker process pin itself to a single cpu core. This is why single-core cpu score is also important.

When the kernel in OPNsense becomes RSS-enabled, packet engine will also be able to make use of multiple cpu cores, basically enabling it to process multi-gigabit workloads. 

[mb]

But I do not have Suricata running, Intrusion detection is completely disabled. I did have it running few weeks back, but disabled it a week ago. So to be sure I rebooted my router/fw before installing Sensei, but still same message.

Hi @scrensen, even if it's disabled, if you have the interface configured for Suricata, Sensei will still warn you. Because what usually happens is people enable Suricata in some future time forgetting that it's in use by Sensei.

We wanted to be in the safe side; and decided to warn users about this even Suricata is disabled.

[scrensen]

Ok, makes sense.

So I got Sensei to work, but then I found my wifi access point (Unifi AP-HD) stopped working right after.

And after some troubleshooting I saw that it could not reach the controller anymore (running on server in same LAN). Since Sensei was the only and most recent change in my network I disabled it and within seconds the UAP-HD came online again an I was able to adopt it again in the controller.

So it seems Sensei is blocking my Unifi AP to reach the controller (http://ip-of-controller:8080/inform) somehow. I'm using the hostname.domain of my controller, perhaps it's something to do with DNS being blocked or so?

[Quetschwalze]

Ok, makes sense.

So I got Sensei to work, but then I found my wifi access point (Unifi AP-HD) stopped working right after.

And after some troubleshooting I saw that it could not reach the controller anymore (running on server in same LAN). Since Sensei was the only and most recent change in my network I disabled it and within seconds the UAP-HD came online again an I was able to adopt it again in the controller.

So it seems Sensei is blocking my Unifi AP to reach the controller (http://ip-of-controller:8080/inform) somehow. I'm using the hostname.domain of my controller, perhaps it's something to do with DNS being blocked or so?

Pretty sure that's the cause. You probably need to whitelist that URL. Verify that sensei is the cause via Reports-Blocked-Live blocked sessions
This will show everything being blocked right now. If you see your unifi URL there just whitelist it.

Gesendet von meinem MI 9 mit Tapatalk