Sensei and external Elasticsearch

[ingvarr]

I have tried to reach with part of this to Sunnyvalley support, but I reckon that it gets broader and might benefit from a discussion.
Sensei does, indeed, look like a promising initiative. And the idea of offloading the DB out of the firewall is great — I'd love to see it more in OPNSense (partly there are pieces like remote logging here and there, but sparse).
Unfortunately, as of now, the support is very incomplete.
I am going to try and collect (perhaps, to be updated) the list of my own concerns. Any comments and opinions will be appreciated — both from SV and the community.
A disclaimer: Sunnyvalley has no obligations to me, for I did not buy any service (only considering), and the discrepancies listed hereby are not implied by any promises made by the company that I am aware of, thus this is just a conversation.

Permissions

Sensei requires a lot to run. That would make sense for an enterprise that can afford a dedicated ELK installation for the firewall, but for a humble home setup this is burdensome.
The documentation mentions indices (<HOSTUUID>_)?(conn|http|tls|alert|sip|dns)_<date>. Well, this is incorrect. First of all, it is dash, not underscore before date. Second, there are also aliases $1_all and $1_write, which makes sense, but would be nice to say a word in the manual.
Now, this is the role definition that seems to have worked for me:

    offensive_role_name:
      cluster: [ "monitor", "read_ilm", "manage_own_api_key" ]
      indices:
        - names:
      #      - "/(conn|http|tls|alert|dns|sip)(_all|_read|_write)(-[23][0-9][01][0-9][0-9][0-9])?/"
            - "/(conn|http|tls|alert|dns|sip).*/"
          privileges: [ "all" ]

Please note the commended more precise definition: I believe it should work, but I just got tired of guesswork.
Interestingly, when I alter the DB URL in the configuration, I get an error — and I do, even if I give sensei user super permissions. Still, the indices are getting created and rolled over, all right.
But of course, it would be better to use a dedicated service account.

Suggested actions:

• Update the manual to reflect actual naming and to mention aliases.
• Provide a role definition example with least privilege necessary.
• Consider switching to service accounts with tokens instead of username and password.

Index naming

By the same reason of reasonably expected coexistence with other datasets, it is a pain to have a diverse set of names pertaining to a single service without a common prefix.

Possible solutions:

• Provide a configuration option for both free and business versions to prepend something to index and alias names is necessary for the sake of prettiness and tidiness,
• Or just prefix sensei-<version>_

Index properties, replicas number above all

I have noticed that newly created indices have zero replicas.
Moreover, scripts/datastore/retire_elasticsearch.py says in elasticsearch_rollover(index_name):
       set_numberof_replicas = '{"index": {"number_of_replicas": 0}}'
Why? This hurts. I can't even touch the ELK without missing primary shards...

Possible solutions:

• Make replicas number configurable,
• Or make replicas number unset (so that the cluster settings apply),
• Or make index template configurable and/or allow to enable relying on the pre-created index template

[mb]

@ingvarr,

Thanks for taking the time to post this. I think there's a bunch of good hints for us to use here. We'll have a thorough look and get back to you.

[ingvarr]

@ingvarr,

Thanks for taking the time to post this. I think there's a bunch of good hints for us to use here. We'll have a thorough look and get back to you.

You are very welcome. And I am happy to elaborate upon any items and share my k8s manifests if relevant (running Elastic in single node microk8s).

BTW, I did forget one thing — recreating the indices if they died (e.g. Elastic was reset) — I did not find a better way than to reinstall Sensei and restore configuration (it might be possible to manually create stuff or to try and reuse something like that retire script, but I am too lazy), but then I saw it in the recent release notes (the version is not yet propagated to the repository).
Very nice.