OPNsense Forum

English Forums => Zenarmor (Sensei) => Topic started by: Seimus on May 13, 2024, 10:50:17 AM

Title: ZenArmor 1.17 memory consumption
Post by: Seimus on May 13, 2024, 10:50:17 AM
Hello,

Just my observation, but it hit my eyes. Since 1.17 release of Zenarmor there is a huge consumption of RAM happening and its increasing. The only way how to lower the consumption is to restart the ZenEngine, but this is only temperarely the memory consumption starts to grow again.


# top -atSzo res -s 3

last pid: 33864;  load averages:  0.62,  0.63,  0.57                                                                                                                                                  up 13+21:28:35  10:44:34
86 processes:  1 running, 84 sleeping, 1 waiting
CPU:  8.5% user,  0.0% nice,  8.3% system,  0.0% interrupt, 83.2% idle
Mem: 1696M Active, 3179M Inact, 2317M Laundry, 8014M Wired, 176K Buf, 491M Free
ARC: 526M Total, 233M MFU, 243M MRU, 6040K Anon, 7334K Header, 37M Other
     422M Compressed, 2114M Uncompressed, 5.01:1 Ratio
Swap: 8192M Total, 2557M Used, 5635M Free, 31% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
31143 root         13  20  -20    10G  9033M nanslp   2 716:03   5.88% eastpect: Eastpect Instance 1 (eastpect)
21223 elasticsea   63  52    0  7052M  1982M uwait    3 473:38   1.34% /usr/local/openjdk8/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+Always
30149 root         13  20  -20  4455M   173M nanslp   1 216:19   4.65% eastpect: Eastpect Instance 0 (eastpect)
60818 root          2  20  -20   396M   126M nanslp   2   9:22   0.13% /usr/local/zenarmor//bin/eastpect -D
33027 root          1  52  -20   396M   126M wait     0   0:00   0.00% eastpect: Eastpect Streamer Instance (eastpect)
50178 root          1  20    0   102M    51M nanslp   2 152:20   0.80% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_watcher.php interface routes alarm
1137 root          1  20    0    81M    47M accept   3   0:01   1.15% /usr/local/bin/php-cgi
69064 root          1  52    0    81M    46M accept   2   0:02   1.74% /usr/local/bin/php-cgi
20343 root          1  21    0    81M    46M accept   3   0:01   2.69% /usr/local/bin/php-cgi
47394 root          1  20    0    81M    46M accept   0   0:00   0.54% /usr/local/bin/php-cgi
   20 root          1  52    0    81M    46M accept   3   0:01   0.00% /usr/local/bin/php-cgi
14981 root          1  52    0    81M    46M accept   1   0:00   0.00% /usr/local/bin/php-cgi


To me it looks like this is just insane amount it takes. This behavior was not there before.

Regards,
S.
Title: Re: ZenArmor 1.17 memory consumption
Post by: IHK on May 13, 2024, 11:36:56 AM
Can you submit a log by following the instructions below to examine the problem in detail?

Can you share the logs and configuration by following the instructions in the below link?

https://www.zenarmor.com/docs/support/reporting-bug

Have a nice week
Title: Re: ZenArmor 1.17 memory consumption
Post by: Seimus on May 13, 2024, 11:40:23 AM
Yes I am planing to do that cause for me this is not a normal behavior and looks like a memory leak.

Topic was opened just for user awareness as well if somebody else has the same observation.

Regards,
S.
Title: Re: ZenArmor 1.17 memory consumption
Post by: almodovaris on May 13, 2024, 12:27:14 PM
Elasticsearch uses several GB of RAM. It's not Zenarmor itself which eats all that memory. You could switch to SQL or MongoDB.

In my case Eastpect uses 4.4 GB and Java uses 6 GB. That's on OPNsense.

On Debian Eastpect uses by an order of magnitude less memory than Java. Java uses 6.7 GB. The machine has almost 7 GB free memory (it has 16 GB in total). And it is still running two Docker apps and a Samba server. Oh, yes, it has zram swap, which, going by CPU usage, it's mostly not being actively used (top says it has 2.7 GB used memory out of 9.4 GB total memory). Kibana and Elasticsearch use a ridiculous amount of virtual memory (over 21 GB each).
Title: Re: ZenArmor 1.17 memory consumption
Post by: sy on May 13, 2024, 12:53:24 PM
Hi,

In this top command output, it seems Zenarmor Engines instance 1 uses so much Ram. Please share the report and let's check it.

Title: Re: ZenArmor 1.17 memory consumption
Post by: Seimus on May 13, 2024, 01:01:42 PM
Quote from: almodovaris on May 13, 2024, 12:27:14 PM
Elasticsearch uses several GB of RAM. It's not Zenarmor itself which eats all that memory. You could switch to SQL or MongoDB.

In my case Eastpect uses 4.4 GB and Java uses 6 GB. That's on OPNsense.

On Debian Eastpect uses by an order of magnitude less memory than Java. Java uses 6.7 GB. The machine has almost 7 GB free memory (it has 16 GB in total). And it is still running two Docker apps and a Samba server. Oh, yes, it has zram swap, which, going by CPU usage, it's mostly not being actively used (top says it has 2.7 GB used memory out of 9.4 GB total memory). Kibana and Elasticsearch use a ridiculous amount of virtual memory (over 21 GB each).

I've seen extreme memory usage since the 1.17 update. Before it was much much lower. Additionally to this Since 1.17 the usage constantly increases which was not seen on previous releases.

Quote from: sy on May 13, 2024, 12:53:24 PM
Hi,

In this top command output, it seems Zenarmor Engines instance 1 uses so much Ram. Please share the report and let's check it.



Ticket with Zen support team was already opened and its under inspection. I've sent the top outputs as well to the support team.

Regards,
S.
Title: Re: ZenArmor 1.17 memory consumption
Post by: almodovaris on May 13, 2024, 01:05:32 PM
Anyway, here are screen images from my two machines.
Title: Re: ZenArmor 1.17 memory consumption
Post by: Seimus on May 16, 2024, 10:04:19 AM
So,

I was able to find out what is causing the Memory consumption (as well SWAP consumption) after having an call with Zenarmor Devs.

It was synflood/syncache being full and not released.

Due to this I have actually a question >
Shouldn't the ZenArmor FW take precaution and prevent resources draining caused by a synflood?

Regards,
S.
Title: Re: ZenArmor 1.17 memory consumption
Post by: w9hdg on June 20, 2024, 07:38:22 PM
How did you end up dealing with this? I'm seeing the same behavior and I'm sure that I'm encountering the same problem.
Title: Re: ZenArmor 1.17 memory consumption
Post by: Seimus on June 20, 2024, 08:01:58 PM
I have opened ticket with them and gave them extra logs, they still investigate I am waiting for update.

But for now If you run NMAP, set it with -f parameter

Regards,
S.
Title: Re: ZenArmor 1.17 memory consumption
Post by: w9hdg on June 20, 2024, 09:54:47 PM
Sweet...forgive my ignorance...how do I do that?

I just dropped into terminal and tried nmap and got command not found so I'm guessing I'm not using nmap.
Title: Re: ZenArmor 1.17 memory consumption
Post by: Seimus on June 20, 2024, 10:32:20 PM
To be more precise,

The memory increase I have seen was caused due to main one thing.

Which is that if syncache is being eaten UP it causes an immediate back pressure on the system resources specifically on the RAM.

ZenArmor was informing that synflood happened because of syncache was eaten up. But this was just a warning and no protective action was done. The moment as mentioned syncache was eaten UP system resources started to be degraded which caused this. Additional to this syncache was not properly released.

In my case NMAP scanning once per 24H was causing this by scanning only 1 IP.

For you it maybe another trigger if yo don't use NMAP port scanner.

Also be sure you have in Zenarmor logging level set to INFO and disabled the CORE file generation.

Regards,
S.
Title: Re: ZenArmor 1.17 memory consumption
Post by: w9hdg on June 24, 2024, 09:08:00 PM
I am honestly at a loss. I really want to like Zenarmor and what it brings to the table but frankly this memory consumption issue has me about ready to uninstall it for a while.

I'm using the Elastisearch Data base (whatever the version 5 one is, but have also tried version 8). I have tried the mongoDB option. No matter what after about 2-3 days I am pretty much out of RAM and SWAP space.

I'm willing to help contribute logs, etc to whatever/whomever in order to fix this because it really is a good product, one I have considered opening my wallet for, but I just can't until this gets resolved.

Any other ideas of what I could try? Every time I change something, restart the Zen Engine, or reboot, my IPv6 prefix changes which means that DNS overrides need to be updated, firewall rules need to be updated, and my external DNS needs to be updated (yes I know I could probably automate some of this, but I haven't had time because I'm always fighting Zenarmor).

Thanks in advance,
~T
Title: Re: ZenArmor 1.17 memory consumption
Post by: Seimus on June 24, 2024, 09:12:14 PM
It would be good to actually 1st see what process is eating the memory and if its really related to zenarmor.

You need to run zenarmor and than to check in CLI via top command what is eating it. If its related to Zenarmor, than open the ma ticket thru the GUI and provide the logs.

Activate Zenarmor let it run and when you see the memory is being eaten out go to CLI via ssh and as root execute

top -ao res

This will tell you what is eating it. Additional to this check as well the Zenarmor logs in GUI if you see there something.

If the TOP process that eats the RAM is related to ZenArmor open them a ticket thru the GUI.

Regards,
S.
Title: Re: ZenArmor 1.17 memory consumption
Post by: w9hdg on June 24, 2024, 09:21:43 PM
I just restarted the Zen engine so it will be a few hours, but as I recall the top processes are usually the database (elastisearch) and eastpect. I'll copy the output of top into here the next time it really starts to get hungry (probably this time tomorrow).

I also restarted the database engine and it cleared the swap out too...seems like something isn't releasing memory properly.
Title: Re: ZenArmor 1.17 memory consumption
Post by: sy on June 25, 2024, 01:42:07 PM
Hi,

Can you share a report with the support team to check the logs and configuration by following the instructions in the below link?

https://zenarmor.com/docs/support/reporting-bug

Title: Re: ZenArmor 1.17 memory consumption
Post by: w9hdg on June 25, 2024, 03:27:36 PM
Here we go, as expected memory utilization is climbing this morning and it looks like Java based with elasticearch and eastpect are the to two which I believe are related to ZenArmor
Title: Re: ZenArmor 1.17 memory consumption
Post by: almodovaris on June 25, 2024, 03:32:29 PM
Yup,  Java based with elasticsearch looks like it wants 8 GB RAM. Even in my machines. Maybe not all the time.
Title: Re: ZenArmor 1.17 memory consumption
Post by: w9hdg on June 25, 2024, 03:33:28 PM
I've tried MongoDB with the same results.
Title: Re: ZenArmor 1.17 memory consumption
Post by: almodovaris on June 25, 2024, 03:37:24 PM
If you're using MongoDB you should disable elasticsearch and it does not use Java.
Title: Re: ZenArmor 1.17 memory consumption
Post by: w9hdg on June 25, 2024, 03:38:30 PM
Right now I'm using Elastisearch, I have tried MongoDB thinking that maybe it would be better. I could switch back and try again but my question is...how would I disable Java then?
Title: Re: ZenArmor 1.17 memory consumption
Post by: Seimus on June 25, 2024, 03:53:43 PM
Do you see any messages popping in the logs "Notification tab" of ZenArmor?

Anyway open them a ticket, tick the the box to sent them the logs as well.

Regards,
S.
Title: Re: ZenArmor 1.17 memory consumption
Post by: almodovaris on June 26, 2024, 08:22:49 AM
Unless there is a reason to execute Java, it does not run. If you disable and stop elasticsearch there's probably no reason to execute Java.