OPNsense Forum

English Forums => Zenarmor (Sensei) => Topic started by: serbans on August 08, 2021, 03:51:41 PM

Title: Sensei - External elastic search - socket/open file descriptors exhaustion
Post by: serbans on August 08, 2021, 03:51:41 PM
Hi everybody!

I have the following issue with Sensei 1.9.3 on an external elastic search database. The number of opened sockets from OPNsense to ES is increasing around 1 TCP socket per second, and the sockets do not seem to be closing on either side (they show connected on both OPNSense and ES) at the same rate as they are being created. This leads to open file descriptors exhaustion on ES side, after a period of time.

I opened a ticket with SunnyValley as well, but wondering if there is some mitigation on the ES side. I tried setting a lower TCP keepalive interval, but this is usually good for connections passing through a firewall in order to avoid state table timeouts, but I do not think it is the case here.

thanks a lot,
Serban
Title: Re: Sensei - External elastic search - socket/open file descriptors exhaustion
Post by: mb on August 08, 2021, 08:47:08 PM
@serbans, we have received the ticket. At first sight, this looks like a socket leak. We're digging deeper and will get back to you soon.
Title: Re: Sensei - External elastic search - socket/open file descriptors exhaustion
Post by: Phiolin on August 18, 2021, 11:31:15 AM
Might well be the same issue as this one I had earlier: https://forum.opnsense.org/index.php?topic=23786.0

I've moved to local Elasticsearch as there wasn't really any progress in finding out why it's hugging up that much memory over time, but I guess there is a chance the underlying issue is the same as for your TCP sockets.
Title: Re: Sensei - External elastic search - socket/open file descriptors exhaustion
Post by: serbans on October 09, 2021, 09:04:58 AM
A short update here:

Received a patch from Sensei about 3-5 days after the ticket was created on the system (thanks!). I have not applied it due to an OpSec issue - I was given an executable with extension .py to replace a python script (which is - in a way - a big no-no)

Was told that the changes will be reflected in the 1.10 version, that is supposed to move (partly) to a new language, hence the executable.

Decided to wait for the official release. Will update then again if the issue is solved.
Title: Re: Sensei - External elastic search - socket/open file descriptors exhaustion
Post by: koushun on October 13, 2021, 12:43:51 AM
Do you use Kibana for visualization of the different reports / Sensei dashboards? Do you have any, to share? :D

https://github.com/psychogun/zenarmor-kibana-dashboards

Title: Re: Sensei - External elastic search - socket/open file descriptors exhaustion
Post by: serbans on October 25, 2021, 04:22:43 PM
New version 1.10 apparently solves the issue of the file descriptors exhaustion.

thanks koushun for the dashboards, really interesting stuff !!

Thanks a lot !
Serban