OPNsense
  • Home
  • Help
  • Search
  • Login
  • Register

  • OPNsense Forum »
  • English Forums »
  • Zenarmor (Sensei) »
  • Sensei on OPNsense - Application based filtering
« previous next »
  • Print
Pages: 1 ... 54 55 [56] 57 58 ... 79

Author Topic: Sensei on OPNsense - Application based filtering  (Read 393188 times)

Antaris

  • Full Member
  • ***
  • Posts: 146
  • Karma: 13
    • View Profile
    • EVOTEK ltd.
Re: Sensei on OPNsense - Application based filtering
« Reply #825 on: March 13, 2020, 04:25:38 pm »
Quote from: mb on January 27, 2020, 05:10:10 pm
Hi faisal,

Than it must be the cpu score. There is a 300.000 minimum cpu score requirement for Elasticsearch.

Here's  a quick hack:

1. Remove /usr/local/sensei/etc/.configdone
Code: [Select]
rm /usr/local/sensei/etc/.configdone
3. Edit /usr/local/opnsense/scripts/OPNsense/Sensei/check_hardware.sh file and locate these lines:

Code: [Select]
if [ $CPU_SCORE -le 300000 ]; then
       CPU_PROPER="false"
else
       CPU_PROPER="true"
fi

Change 300000 to a lower value, like 200000. 

4. Do a browser refresh on the OPNsense UI, and click on any sensei menu. It'll re-run the config wizard. Now it should select Elasticsearch.

Now I'm thinking: for cpu scores between 200K and 300K and if there is enough memory (>=8GB) I think we should let the user decide on the database backend.

This solution no longer works on fresh install today. And i can't find from where to choose Elastic engine...
Logged
Proxmox enthusiast @home, bare metal @work.

mb

  • Hero Member
  • *****
  • Posts: 935
  • Karma: 98
    • View Profile
    • Sunny Valley Networks
Re: Sensei on OPNsense - Application based filtering
« Reply #826 on: March 13, 2020, 05:07:58 pm »
Hi @Antaris,

This looks good and should've worked. But with 1.5 database selection will be optional if the device has enough memory but weak cpu (e.g. 200.000<>300.000 cpu score).

We hope to release 1.5 late this month.

By the way, I think this was your request, you can now request re-classification for a web site through Sunny Valley website ;)

https://www.sunnyvalley.io/site-classification/

Logged

Antaris

  • Full Member
  • ***
  • Posts: 146
  • Karma: 13
    • View Profile
    • EVOTEK ltd.
Re: Sensei on OPNsense - Application based filtering
« Reply #827 on: March 14, 2020, 12:01:43 pm »
Quote from: mb on March 13, 2020, 05:07:58 pm
Hi @Antaris,

This looks good and should've worked. But with 1.5 database selection will be optional if the device has enough memory but weak cpu (e.g. 200.000<>300.000 cpu score).

We hope to release 1.5 late this month.

By the way, I think this was your request, you can now request re-classification for a web site through Sunny Valley website ;)

https://www.sunnyvalley.io/site-classification/
Hi @mb,

Looking forward to 1.5 and thx for the classification option. :)
Logged
Proxmox enthusiast @home, bare metal @work.

deibit

  • Newbie
  • *
  • Posts: 2
  • Karma: 0
    • View Profile
Re: Sensei on OPNsense - Application based filtering
« Reply #828 on: March 15, 2020, 12:29:07 pm »
Hi,

I have just installed OPNsense after many (more than i recall) years using PfSense. One of the first things I installed is Sensei.. I liked it so much that I even paid for the home license :)

I run OPNsense in a SC813 Supermicro Rack, with a X10SLM+LN4F MoBo, 32GB DDR-1600 ECC RAM, a Xeon E3-1230L V3 CPU and a 2TB Seagate Ironwolf HDD (all baremetal, not virtualized).

I use a single 1.000/50 Mbit/s WAN, not many simultaneous webusers (4 or 5) but heavy traffic on the 4 VPN tunnels (fileserver)

The CPU gives 267.081 Single CPU Ubench index. So it's underpowered according to Sensei/Elasticsearch standards.

I guess I wouldn't have any problem in upgrading the CPU, but why is the single core performance so important? ElasticSearch s supposed to take advange of multiple cores isn't it? (CPU has 4 cores and 8 threads)

Should I be worried? Where can I look if my CPU is struggling? Everything "feels" good so far...



Logged

Antaris

  • Full Member
  • ***
  • Posts: 146
  • Karma: 13
    • View Profile
    • EVOTEK ltd.
Re: Sensei on OPNsense - Application based filtering
« Reply #829 on: March 16, 2020, 12:28:20 am »
Hi @deibit,

I think every Intel Core with AES and 4 or more cores @3GHz or more is OK for OPNsense with Sensei and Elastic. In the incoming 1.5 you will have the option to choose backend database manually and this misunderstanding will be solved. May be it's good to upgrade cpu to non-L Haswell or Broadwell cpu for better VPN throughput. Also you have to know that for some strange reason Ubench rates Haswell Xeons way lower than non-xeon i5 on same clocks...
Logged
Proxmox enthusiast @home, bare metal @work.

Quetschwalze

  • Newbie
  • *
  • Posts: 29
  • Karma: 3
    • View Profile
Re: Sensei on OPNsense - Application based filtering
« Reply #830 on: March 17, 2020, 11:57:08 pm »
@mb sensei is still running very smooth for me
Any news/eta on those botnet and DNS tunneling features already shown in the policy?

Gesendet von meinem MI 9 mit Tapatalk

Logged

deibit

  • Newbie
  • *
  • Posts: 2
  • Karma: 0
    • View Profile
Re: Sensei on OPNsense - Application based filtering
« Reply #831 on: March 18, 2020, 09:26:36 pm »
Quote from: Antaris on March 16, 2020, 12:28:20 am
Hi @deibit,

I think every Intel Core with AES and 4 or more cores @3GHz or more is OK for OPNsense with Sensei and Elastic. In the incoming 1.5 you will have the option to choose backend database manually and this misunderstanding will be solved. May be it's good to upgrade cpu to non-L Haswell or Broadwell cpu for better VPN throughput. Also you have to know that for some strange reason Ubench rates Haswell Xeons way lower than non-xeon i5 on same clocks...

I "upgraded" to a E3-1268L that I had here "laying around". Now the ubench score is in the 342.000 range, I don't think it makes a big difference but my karma is again in balance due to sensei not complaining about my router being lower end :)

I still wonder why the single core performance is so important for elasticsearch though...
Logged

bEeReE

  • Newbie
  • *
  • Posts: 17
  • Karma: 2
    • View Profile
Re: Sensei on OPNsense - Application based filtering
« Reply #832 on: March 20, 2020, 08:57:48 am »
Quote from: Quetschwalze on March 17, 2020, 11:57:08 pm
@mb sensei is still running very smooth for me
Any news/eta on those botnet and DNS tunneling features already shown in the policy?

Would be also great if we can get a little bit more background how the sensei cloud works. Is it feeded with external sources (e.g. phishtank, malware companies etc.) so that we can avoid duplicated filtering on dns, proxy or ICAP level?
Furthermore, I am wondering how often decisions about security categorization like "undecided safe sites" are made. I believe there are some blocked domains hanging in such status for more than 1 week. Shouldn't this be faster?
Logged

scrensen

  • Newbie
  • *
  • Posts: 23
  • Karma: 1
    • View Profile
Re: Sensei on OPNsense - Application based filtering
« Reply #833 on: March 20, 2020, 11:21:14 pm »
I've searched the forums as well as Google but didn't find the answer to my issue. Which is that during Sensei installation I get the following message: "Oops, it looks like LAN interface is also in use by Suricata"

But I do not have Suricata running, Intrusion detection is completely disabled. I did have it running few weeks back, but disabled it a week ago. So to be sure I rebooted my router/fw before installing Sensei, but still same message.

I did a quick search via the command line for Suricate config files to check for interface config, but didn't find anything useful.

Anyone that might be able to help me out here?

Thanks in advance!
Logged

scrensen

  • Newbie
  • *
  • Posts: 23
  • Karma: 1
    • View Profile
Re: Sensei on OPNsense - Application based filtering
« Reply #834 on: March 20, 2020, 11:29:08 pm »
Quote from: scrensen on March 20, 2020, 11:21:14 pm
during Sensei installation I get the following message: "Oops, it looks like LAN interface is also in use by Suricata"

But I do not have Suricata running

I still decided to check the Captive Portal settings and I removed LAN from the interfaces and applied. And that actually solved the issue. Even though it was not active....
Logged

mb

  • Hero Member
  • *****
  • Posts: 935
  • Karma: 98
    • View Profile
    • Sunny Valley Networks
Re: Sensei on OPNsense - Application based filtering
« Reply #835 on: March 21, 2020, 06:00:05 am »
@Quetschwalze, glad that it's working good.

We're investing heavily on application, threat intelligence and security databases. With 1.5 version, you'll start seeing weekly / daily database updates. Up until now databases were being updated with every release.

With regard to Botnet filtering / DNS Tunnel filtering, we expect to land them in Q3 this year.

@bEeReE, we are adding more information to our documentation about the cloud architecture. We'll make it available early April.
« Last Edit: March 21, 2020, 06:09:22 am by mb »
Logged

mb

  • Hero Member
  • *****
  • Posts: 935
  • Karma: 98
    • View Profile
    • Sunny Valley Networks
Re: Sensei on OPNsense - Application based filtering
« Reply #836 on: March 21, 2020, 06:07:01 am »
Quote from: deibit on March 18, 2020, 09:26:36 pm
I still wonder why the single core performance is so important for elasticsearch though...

Hi @deibit, glad that Sensei is happy with you hardware now.

As for Elasticsearch, you are right, Elasticsearch is multi-threaded and will benefit from multi-core cpus.

For the packet engine itself, for each interface, it has a worker process and workers are currently running single threaded; so each worker process pin itself to a single cpu core. This is why single-core cpu score is also important.

When the kernel in OPNsense becomes RSS-enabled, packet engine will also be able to make use of multiple cpu cores, basically enabling it to process multi-gigabit workloads. 
Logged

mb

  • Hero Member
  • *****
  • Posts: 935
  • Karma: 98
    • View Profile
    • Sunny Valley Networks
Re: Sensei on OPNsense - Application based filtering
« Reply #837 on: March 21, 2020, 06:11:49 am »
Quote from: scrensen on March 20, 2020, 11:21:14 pm
But I do not have Suricata running, Intrusion detection is completely disabled. I did have it running few weeks back, but disabled it a week ago. So to be sure I rebooted my router/fw before installing Sensei, but still same message.

Hi @scrensen, even if it's disabled, if you have the interface configured for Suricata, Sensei will still warn you. Because what usually happens is people enable Suricata in some future time forgetting that it's in use by Sensei.

We wanted to be in the safe side; and decided to warn users about this even Suricata is disabled.
Logged

scrensen

  • Newbie
  • *
  • Posts: 23
  • Karma: 1
    • View Profile
Re: Sensei on OPNsense - Application based filtering
« Reply #838 on: March 22, 2020, 02:27:18 pm »
Ok, makes sense.

So I got Sensei to work, but then I found my wifi access point (Unifi AP-HD) stopped working right after.

And after some troubleshooting I saw that it could not reach the controller anymore (running on server in same LAN). Since Sensei was the only and most recent change in my network I disabled it and within seconds the UAP-HD came online again an I was able to adopt it again in the controller.

So it seems Sensei is blocking my Unifi AP to reach the controller (http://ip-of-controller:8080/inform) somehow. I'm using the hostname.domain of my controller, perhaps it's something to do with DNS being blocked or so?
Logged

Quetschwalze

  • Newbie
  • *
  • Posts: 29
  • Karma: 3
    • View Profile
Re: Sensei on OPNsense - Application based filtering
« Reply #839 on: March 22, 2020, 04:26:52 pm »
Quote from: scrensen on March 22, 2020, 02:27:18 pm
Ok, makes sense.

So I got Sensei to work, but then I found my wifi access point (Unifi AP-HD) stopped working right after.

And after some troubleshooting I saw that it could not reach the controller anymore (running on server in same LAN). Since Sensei was the only and most recent change in my network I disabled it and within seconds the UAP-HD came online again an I was able to adopt it again in the controller.

So it seems Sensei is blocking my Unifi AP to reach the controller (http://ip-of-controller:8080/inform) somehow. I'm using the hostname.domain of my controller, perhaps it's something to do with DNS being blocked or so?
Pretty sure that's the cause. You probably need to whitelist that URL. Verify that sensei is the cause via Reports-Blocked-Live blocked sessions
This will show everything being blocked right now. If you see your unifi URL there just whitelist it.

Gesendet von meinem MI 9 mit Tapatalk

Logged

  • Print
Pages: 1 ... 54 55 [56] 57 58 ... 79
« previous next »
  • OPNsense Forum »
  • English Forums »
  • Zenarmor (Sensei) »
  • Sensei on OPNsense - Application based filtering
 

OPNsense is an OSS project © Deciso B.V. 2015 - 2023 All rights reserved
  • SMF 2.0.19 | SMF © 2021, Simple Machines
    Privacy Policy
    | XHTML | RSS | WAP2