ElasticSearch migration

[DenverTech]

I was eyeing the external ElasticSearch option and was curious about a few things there.
1) If we fully offload the ElasticSearch to something external, does that change anything with our reporting/graphs, etc? I wasn't clear on whether the "Kibana" portion of the directions was something required as part of the move to external.
2) If we offload to an external DB how do we move to the external database, or do we need to reinstall? I see the directions for attaching it as a stream target, or how to use external from the start, but wasn't sure how to switch after installing.
3) Lastly, what's the benefit of storing locally and also streaming to external? Wasn't sure what benefit that provided.

Thanks!

[mb]

Hi @DenverTech,

Very good questions. Please find my answers below:

If we fully offload the ElasticSearch to something external, does that change anything with our reporting/graphs, etc? I wasn't clear on whether the "Kibana" portion of the directions was something required as part of the move to external.

Nothing changes at all. You'll get the same reports as usual. With offloading to a remote, you get the additional benefit of being able to create your own graphs and dashboards. Kibana is not required; but allows you to do that.

If we offload to an external DB how do we move to the external database, or do we need to reinstall? I see the directions for attaching it as a stream target, or how to use external from the start, but wasn't sure how to switch after installing.

A simple re-install will do the job. Or, rm /usr/local/sensei/etc/.configdone file and re-do the initial configuration.

Lastly, what's the benefit of storing locally and also streaming to external? Wasn't sure what benefit that provided.

1. You have a backup of your reporting data
2. With a copy of the all reportign data, you can create lots of additional reports to your liking and business scenarios.

Some MSP partners of ours, offload their clients' reporting to a central location to be able to visualize and create a "consolidated" "big picture" view of the reports.

This way, they can utilize this to be able to better detect and respond to incidents as they occur, since the data is real-time.

PS: We'll put these onto our documentation.

[DenverTech]

Thank you very much! That answers everything for me. I appreciate the quick replies.

[DenverTech]

For anyone else trying this, a few minor bits of advice:
- If you already setup Sensei and are switching to external ElasticSearch, stream your reporting to it first, so you have a baseline. It makes for an instantaneous swap with no lost data.
- MB's suggestion of deleting the .configdone file is perfect. It keeps your full configuration but allows you to switch database setups.
- An external ElasticSearch server actually runs very well as a smallish VM. I've got a 2 CPU, 8gb memory vm running smoothly for 1000+ users. This thread (https://forum.opnsense.org/index.php?topic=19252.0) also has some advice for speeding up your ES box that improves responsiveness considerably.
- Speaking of ElasticSearch as a VM, bitnami has virtual appliances for this that are free. Takes a few seconds and you have a working ES guest. Their newer ones are a bit more of a pain to setup, so I actually recommend version 7.3.2 (you can find it on the VMware marketplace).
- Why do I recommend this? The load change on the firewall is HUGE. See below...
Internal ES utilization: ~10% CPU, 8gb memory, max bandwidth 650mbit
External ES utilization: ~2% CPU, 2gb memory, max bandwidth 950mbit

[koushun]

For further offloading the firewall, I am using Kibana as the go to for viewing reports: https://github.com/psychogun/zenarmor-kibana-dashboards

I am not getting indicies for the "Threats" section, though. Do not know just yet how I can visualize that data.