Sensei on OPNsense - Application based filtering

[Jppp]

Hi,

You can reinstall database by following the below document. Can you share a bug report before DB reinstall to look into the elasticsearch issue?


https://www.sunnyvalley.io/docs/troubleshooting/reporting#how-do-i-reinstall-the-reporting-database

Send the bug-report via Sensei plugin, was happy to see that logs can be included really easily! (Added a link to my comment in the report)
 There are quite a lot of logs in /urs/local/sensei/active ***, do you want them all here too?

To make sure it wasn't a one off crash, I enabled Sensei without enabling the ES service. After ~3 minutes the network had a small crash of ~1 minute, came back up and the system crashed ~2 minutes later (although I can't find anything in the logs).
I shut the system off via the hardware button, waited for a few minutes and booted it up again. Ram usage after boot was ~4gb and reached 6gb when I stopped the Zenarmor engine. ES is still running and ram seems stable ~4gb.

I have removed the database manually, doing the wizard again now. Will update my comment afterwards.


Configuration
WAN: re0, Realtek RTL8111HSD-CG
LAN: lagg0(), 2-port LACP on intel i340-t2

OPNsense community-repo: mimugmail [update1]

ZenArmor
General:
 Mode: Routed with native netmap driver
 Interface: LAN
 DB: ES
 size: Small II (< 51 devices), sensei's doc [1] estimates a throughput of 500 Mbps for this setup with a min. of 4gb.
 
Cloud Threat intel:
 Enabled: yes

Updates & Health:
 Max. Swap Util: 60% *

Reporting & Data:
 Size of the Fast Temporary Memory Disk: 48% **
 Real-time DNS reverse queries for local IP: Disabled
 OPNsense Host aliases for DNS enrichment: Disabled
 Maximum number of days to store reporting data: 7 days


* SWAP is disabled on OPN, does this setting interfere with that? (I assumed the setting is being ignored)
** The default setting. This metric does not include the ES service itself right? (as in, the whole sensei service memory usage). My system uses 1.5gb avg, so ~2gb, add 4gb for fast temp mem disk and I've got only ~2gb left for Sensei?
*** main_, periodical_, seneigui, idpr*_,streamer_, worker_ and update_check.

1. https://www.sunnyvalley.io/docs/introduction/hardware-requirements#cpu--memory

UPDATE 1: Wizard: reporting & database.
During database selection I got the following notification

It looks like you also have mimugmail community repo enabled. Please be advised that this repo is also serving Elasticsearch and Mongodb packages with their dependencies. In this regard sunnyvalley and community repositoriees (spelling error in modal, if a Sensei dev is reading this) are not compatible when enabled at the same time.

If you would like to continue using both repositories, we advise to install Elasticsearch from the community repository and point zenarmor to this database as a "Remote Elasticsearch" database.

My dashboard shows that ES is still running, so I'm going to remove ZenArmor, add mimugmail-ES, install ZenArmor, external source for ES. Will update again.

Also, for my usecase e.g. low user count, relatively low usage, is ES that beneficial compared to Mongo? I'd like to also run OPNsense IDS (suricata) which doesn't really feel feasible right now.

UPDATE 2: ZenArmor ES & community plugins
An existing issue in the plugin repo, https://github.com/mimugmail/opn-repo/issues/116.
I'm already using AdGuardHome & speedtest from the repo, I'm going to offload it to another machine, remove the community repo and try again (bummer that they don't work together though, I was thinking of using some of his plugins)

[Jppp]

my bad, this is a known issue with netmap on lagg interfaces.
https://forum.opnsense.org/index.php?topic=24015.0

To be sure I

• removed lagg, moved LAN to igb0 (netmap issue)
• Moved WAN from re0 to igb1 (realtek driver history)

Everything seems to work as normal again