ZenArmor 1.17 memory consumption

[Seimus]

Hello,

Just my observation, but it hit my eyes. Since 1.17 release of Zenarmor there is a huge consumption of RAM happening and its increasing. The only way how to lower the consumption is to restart the ZenEngine, but this is only temperarely the memory consumption starts to grow again.

Code: [Select]

# top -atSzo res -s 3

last pid: 33864;  load averages:  0.62,  0.63,  0.57                                                                                                                                                  up 13+21:28:35  10:44:34
86 processes:  1 running, 84 sleeping, 1 waiting
CPU:  8.5% user,  0.0% nice,  8.3% system,  0.0% interrupt, 83.2% idle
Mem: 1696M Active, 3179M Inact, 2317M Laundry, 8014M Wired, 176K Buf, 491M Free
ARC: 526M Total, 233M MFU, 243M MRU, 6040K Anon, 7334K Header, 37M Other
     422M Compressed, 2114M Uncompressed, 5.01:1 Ratio
Swap: 8192M Total, 2557M Used, 5635M Free, 31% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
31143 root         13  20  -20    10G  9033M nanslp   2 716:03   5.88% eastpect: Eastpect Instance 1 (eastpect)
21223 elasticsea   63  52    0  7052M  1982M uwait    3 473:38   1.34% /usr/local/openjdk8/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+Always
30149 root         13  20  -20  4455M   173M nanslp   1 216:19   4.65% eastpect: Eastpect Instance 0 (eastpect)
60818 root          2  20  -20   396M   126M nanslp   2   9:22   0.13% /usr/local/zenarmor//bin/eastpect -D
33027 root          1  52  -20   396M   126M wait     0   0:00   0.00% eastpect: Eastpect Streamer Instance (eastpect)
50178 root          1  20    0   102M    51M nanslp   2 152:20   0.80% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_watcher.php interface routes alarm
 1137 root          1  20    0    81M    47M accept   3   0:01   1.15% /usr/local/bin/php-cgi
69064 root          1  52    0    81M    46M accept   2   0:02   1.74% /usr/local/bin/php-cgi
20343 root          1  21    0    81M    46M accept   3   0:01   2.69% /usr/local/bin/php-cgi
47394 root          1  20    0    81M    46M accept   0   0:00   0.54% /usr/local/bin/php-cgi
   20 root          1  52    0    81M    46M accept   3   0:01   0.00% /usr/local/bin/php-cgi
14981 root          1  52    0    81M    46M accept   1   0:00   0.00% /usr/local/bin/php-cgi

To me it looks like this is just insane amount it takes. This behavior was not there before.

Regards,
S.

[IHK]

Can you submit a log by following the instructions below to examine the problem in detail?

Can you share the logs and configuration by following the instructions in the below link?

https://www.zenarmor.com/docs/support/reporting-bug

Have a nice week

[Seimus]

Yes I am planing to do that cause for me this is not a normal behavior and looks like a memory leak.

Topic was opened just for user awareness as well if somebody else has the same observation.

Regards,
S.

[almodovaris]

Elasticsearch uses several GB of RAM. It's not Zenarmor itself which eats all that memory. You could switch to SQL or MongoDB.

In my case Eastpect uses 4.4 GB and Java uses 6 GB. That's on OPNsense.

On Debian Eastpect uses by an order of magnitude less memory than Java. Java uses 6.7 GB. The machine has almost 7 GB free memory (it has 16 GB in total). And it is still running two Docker apps and a Samba server. Oh, yes, it has zram swap, which, going by CPU usage, it's mostly not being actively used (top says it has 2.7 GB used memory out of 9.4 GB total memory). Kibana and Elasticsearch use a ridiculous amount of virtual memory (over 21 GB each).

[sy]

Hi,

In this top command output, it seems Zenarmor Engines instance 1 uses so much Ram. Please share the report and let's check it.

[Seimus]

Elasticsearch uses several GB of RAM. It's not Zenarmor itself which eats all that memory. You could switch to SQL or MongoDB.

In my case Eastpect uses 4.4 GB and Java uses 6 GB. That's on OPNsense.

On Debian Eastpect uses by an order of magnitude less memory than Java. Java uses 6.7 GB. The machine has almost 7 GB free memory (it has 16 GB in total). And it is still running two Docker apps and a Samba server. Oh, yes, it has zram swap, which, going by CPU usage, it's mostly not being actively used (top says it has 2.7 GB used memory out of 9.4 GB total memory). Kibana and Elasticsearch use a ridiculous amount of virtual memory (over 21 GB each).

I've seen extreme memory usage since the 1.17 update. Before it was much much lower. Additionally to this Since 1.17 the usage constantly increases which was not seen on previous releases.

Hi,

In this top command output, it seems Zenarmor Engines instance 1 uses so much Ram. Please share the report and let's check it.

Ticket with Zen support team was already opened and its under inspection. I've sent the top outputs as well to the support team.

Regards,
S.

[almodovaris]

Anyway, here are screen images from my two machines.

[Seimus]

So,

I was able to find out what is causing the Memory consumption (as well SWAP consumption) after having an call with Zenarmor Devs.

It was synflood/syncache being full and not released.

Due to this I have actually a question >
Shouldn't the ZenArmor FW take precaution and prevent resources draining caused by a synflood?

Regards,
S.

[w9hdg]

How did you end up dealing with this? I'm seeing the same behavior and I'm sure that I'm encountering the same problem.

[Seimus]

I have opened ticket with them and gave them extra logs, they still investigate I am waiting for update.

But for now If you run NMAP, set it with -f parameter

Regards,
S.

[w9hdg]

Sweet...forgive my ignorance...how do I do that?

I just dropped into terminal and tried nmap and got command not found so I'm guessing I'm not using nmap.

[Seimus]

To be more precise,

The memory increase I have seen was caused due to main one thing.

Which is that if syncache is being eaten UP it causes an immediate back pressure on the system resources specifically on the RAM.

ZenArmor was informing that synflood happened because of syncache was eaten up. But this was just a warning and no protective action was done. The moment as mentioned syncache was eaten UP system resources started to be degraded which caused this. Additional to this syncache was not properly released.

In my case NMAP scanning once per 24H was causing this by scanning only 1 IP.

For you it maybe another trigger if yo don't use NMAP port scanner.

Also be sure you have in Zenarmor logging level set to INFO and disabled the CORE file generation.

Regards,
S.

[w9hdg]

I am honestly at a loss. I really want to like Zenarmor and what it brings to the table but frankly this memory consumption issue has me about ready to uninstall it for a while.

I'm using the Elastisearch Data base (whatever the version 5 one is, but have also tried version . I have tried the mongoDB option. No matter what after about 2-3 days I am pretty much out of RAM and SWAP space.

I'm willing to help contribute logs, etc to whatever/whomever in order to fix this because it really is a good product, one I have considered opening my wallet for, but I just can't until this gets resolved.

Any other ideas of what I could try? Every time I change something, restart the Zen Engine, or reboot, my IPv6 prefix changes which means that DNS overrides need to be updated, firewall rules need to be updated, and my external DNS needs to be updated (yes I know I could probably automate some of this, but I haven't had time because I'm always fighting Zenarmor).

Thanks in advance,
~T

[Seimus]

It would be good to actually 1st see what process is eating the memory and if its really related to zenarmor.

You need to run zenarmor and than to check in CLI via top command what is eating it. If its related to Zenarmor, than open the ma ticket thru the GUI and provide the logs.

Activate Zenarmor let it run and when you see the memory is being eaten out go to CLI via ssh and as root execute

Code: [Select]

top -ao res
This will tell you what is eating it. Additional to this check as well the Zenarmor logs in GUI if you see there something.

If the TOP process that eats the RAM is related to ZenArmor open them a ticket thru the GUI.

Regards,
S.

[w9hdg]

I just restarted the Zen engine so it will be a few hours, but as I recall the top processes are usually the database (elastisearch) and eastpect. I'll copy the output of top into here the next time it really starts to get hungry (probably this time tomorrow).

I also restarted the database engine and it cleared the swap out too...seems like something isn't releasing memory properly.