23.7 CPU / RAM 100% crashing - Zenarmor?

[ThyOnlySandman]

Updated to 23.7 today.  System resource usage out of control and crashing opnsense within 20 minutes.
VMware VM - 12 CPU core / 12GB RAM / 8GB SWAP.
Around 20 min of uptime RAM + SWAP 100% + CPU 100%. 
Been running this VM long time throughout many Opnsense versions without issue.  Normally very low CPU and ~60-70% RAM use with both zenarmor + surcata + few others.

Despite turning off Zenarmor from starting engine + elasticsearch at boot I still see Zenarmor processes in a "ps axmfv" using excessive CPU - So thinking it's zenarmor...

Anyone seen this? 
Going to give Zenarmor full uninstall + reinstall momentarily.  If that doesn't fix then Veeam VM restore back to 23.1

[ThyOnlySandman]

  PID STAT      TIME  SL  RE PAGEIN    VSZ    RSS LIM TSIZ  %CPU %MEM COMMAND
  327 R      3:19.71 127  12      0 776148  10476   - 3384  28.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
  485 R      2:10.97 127 127      0 776148 707568   - 3384  28.0  5.6 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
  803 R      1:04.09 127 127      0 776148 709516   - 3384  30.0  5.7 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
 7462 R      3:58.24 127 127     29 776148 650952   - 3384  31.0  5.2 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
13624 R      8:20.46 127  12  17237 776180   9036   - 3384  30.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
16332 R      4:27.52 127  12   4300 776148   8908   - 3384  31.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
26000 S      1:44.36   1   1     92  81928  21872   - 3384   0.0  0.2 /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_watcher.php interface routes alarm
30046 R      6:22.34 127  12   9509 776148   8816   - 3384  28.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
31736 R      7:08.05 127  12   5007 776148   8820   - 3384  29.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
36510 R      5:44.67 127  12   2873 776180   9060   - 3384  29.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
41468 R     10:55.15 127  12  11959 776148  11228   - 3384  30.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
42713 D      7:46.38   0  12   5226 776148   8992   - 3384  31.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
46351 R     11:28.01 127  12  17286 776148  11120   - 3384  29.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
48021 R      0:32.50 127 127      0 776148 709544   - 3384  30.0  5.7 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
49063 S      5:03.43   0  12   2803 776180   8916   - 3384  28.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
51823 D     10:35.04   0  12  12930 776148  11152   - 3384  30.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
53271 R     10:19.38 127  12  20167 776148  11024   - 3384  29.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
53394 R      0:05.37 127   5      0 718804 672644   - 3384  41.0  5.4 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
59011 R      9:32.89 127  12   9386 776148   9032   - 3384  29.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
64584 S     10:05.75   0  12   5943 776180   9344   - 3384  32.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
66152 R      1:37.47 127 127      0 776148 709540   - 3384  30.0  5.7 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
77401 R     11:14.23 127  12   7703 776148  13228   - 3384  31.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
78895 R      9:09.37 127  12   1912 776148  11104   - 3384  29.0  0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
84200 R      2:46.46 127 127      0 776180 709652   - 3384  31.0  5.7 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
96845 S      0:00.36   0   1      0 102408  73352   - 3384   3.0  0.6 /usr/local/bin/php /usr/local/etc/rc.routing_configure alarm
98164 R      0:09.26 127  12      0 776180 679960   - 3384  41.0  5.4 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases

[ThyOnlySandman]

It’s Zenarmor 1.14.2 crashing OPNsense 23.7.1_3-amd64.

Uninstalled and system stable ~10% CPU + 32%(4GB) RAM for over 30 minutes.
Installed Zenarmor 1.14.2 and couldn’t even finish the initial Zenarmor config before the WebUI crashed and became inaccessible.  While installing the DB I saw RAM jump to 90% prior to webui crash.

Rebooted and same observed behavior prior to reinstall.
18 min uptime
CPU usage   100 %
Memory usage   98 % ( 12076/12250 MB )
SWAP usage   99 % ( 8191/8192 MB )

Same behavior happens with Zenarmor engine + DB off at boot.  So appears to be Zenarmor webUI / php issue.

[ThyOnlySandman]

Tried one more attempt uninstall via Zenarmor with delete all folders rather than uninstall via plugins.
Upon fully fresh install with zero config same issue.

The excessive processes show:
/usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases

The backend log has:
2023-08-11T22:50:52-07:00   Error   configd.py   [046e3ca0-9ed2-4322-9aa0-4a5740b456e5] returned exit status 255   
2023-08-11T22:47:43-07:00   Error   configd.py   [9b953469-e127-4741-8073-efa5f588d696] Script action stderr returned "b'2023.08.11 - 22:47:43 - INFO - [main] Starting ipdr retiring for ELASTICSEARCH...\n2023.08.11 - 22:47:43 - INFO - [conn] Rename alias to zenarmor_0000000000_c56fad43-5729-4727-858a-ff35e2cee6f6_conn \n2023.08.11 - 22:47:43 - INFO - [conn_all] Create New Ali'"   
2023-08-11T22:43:58-07:00   Error   configd.py   [93b402db-76cc-4938-ace9-8065fb688881] Script action stderr returned "b'umount: /dev/md43: statfs: No such file or directory\numount: /dev/md43: unknown file system'"   
2023-08-11T22:43:54-07:00   Error   configd.py   [45a13456-4081-4a2c-a34c-6fe3effd8f71] Script action failed with Command '/usr/local/zenarmor/scripts/installers/elasticsearch/create_indices.py '' ''' returned non-zero exit status 5. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 44, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/zenarmor/scripts/installers/elasticsearch/create_indices.py '' ''' returned non-zero exit status 5.   
2023-08-11T22:43:38-07:00   Error   configd.py   [402ca649-9846-4c30-a750-42756b0b3cdf] Script action failed with Command '/usr/local/zenarmor/scripts/datastore/rename_alias_elasticsearch.py 'zenarmor_0000000000_c56fad43-5729-4727-858a-ff35e2cee6f6_'' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 44, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/zenarmor/scripts/datastore/rename_alias_elasticsearch.py 'zenarmor_0000000000_c56fad43-5729-4727-858a-ff35e2cee6f6_'' returned non-zero exit status 1.


Restoring back to Opn 23.1 + 13.x Zen.  Revisit upgrade in a month.

[yeraycito]

Instead of checking the update in a month's time, I recommend you to do it in a couple of centuries and see if it works well by then, in the meantime use Adguard:

https://forum.opnsense.org/index.php?topic=22162.225

[lilsense]

Adguard is NO IPS. You have no clue what you are talking about yeraycito.

[ThyOnlySandman]

Instead of checking the update in a month's time, I recommend you to do it in a couple of centuries and see if it works well by then, in the meantime use Adguard:

https://forum.opnsense.org/index.php?topic=22162.225

I take it you don't like Zenarmor...
Over the years I've only had 1 other issue on another deployment before.  After month or two it would crash restarting protected LAN interface which would down network for 5 min but would recover.  Believe it was an update that later repaired.  Also believe it was shortly after a new Opnsense release.  Just like now, but they've completely re-done UI too.

Zenarmor does more than ad blocking.
My main complaint with Zenarmor is their business / license model.  I have a Fortigate 60F in transparent mode downstream this Opnsense I'm having issues with now.  The Fortigate is a real NGFW.  Zenarmor is not.  The yearly pricing plans for business edition simply don't compete against a Fortigate UTP subscription .  Having said that I still value the ads + app blocking + traffic analysis Zenarmor provides. 
Briefly reviewing their 1.14 new version bit bummed that custom port applications require license.  As does the https blocking page.  Active directory agent now requires Enterprise.
Summary.  Decent tool.  Bad license model for provided value.

[JonStuart]

Instead of checking the update in a month's time, I recommend you to do it in a couple of centuries and see if it works well by then, in the meantime use Adguard:

https://forum.opnsense.org/index.php?topic=22162.225

I take it you don't like Zenarmor...
Over the years I've only had 1 other issue on another deployment before.  After month or two it would crash restarting protected LAN interface which would down network for 5 min but would recover.  Believe it was an update that later repaired.  Also believe it was shortly after a new Opnsense release.  Just like now, but they've completely re-done UI too.

Zenarmor does more than ad blocking.
My main complaint with Zenarmor is their business / license model.  I have a Fortigate 60F in transparent mode downstream this Opnsense I'm having issues with now.  The Fortigate is a real NGFW.  Zenarmor is not.  The yearly pricing plans for business edition simply don't compete against a Fortigate UTP subscription .  Having said that I still value the ads + app blocking + traffic analysis Zenarmor provides. 
Briefly reviewing their 1.14 new version bit bummed that custom port applications require license.  As does the https blocking page.  Active directory agent now requires Enterprise.
Summary.  Decent tool.  Bad license model for provided value.

Have you tried to run a system audit and seeing what the results are? That looks like its related to php and/or python.

Goto System >> Firmware >> Status

Then choose "Health" from the "Run An Audit" button next to the  "Check for updates" button. You may also want to look at the "Upgrade" audit as well. Something isn't right though. An audit should help you find it.

I have the latest version of OPNsense and Zenarmor and it's working fine but on one older firewall (Been through many OPNsense updates) I had to manually fix some upgrade issues with OPNsense for a proper install of dependencies for Zenarmor. So you are probably looking for dependency issues in the audit.

[ThyOnlySandman]

Have you tried to run a system audit and seeing what the results are? That looks like its related to php and/or python.

Goto System >> Firmware >> Status

Then choose "Health" from the "Run An Audit" button next to the  "Check for updates" button. You may also want to look at the "Upgrade" audit as well. Something isn't right though. An audit should help you find it.

I have the latest version of OPNsense and Zenarmor and it's working fine but on one older firewall (Been through many OPNsense updates) I had to manually fix some upgrade issues with OPNsense for a proper install of dependencies for Zenarmor. So you are probably looking for dependency issues in the audit.

I did not try the audit.  Good idea though.  Agree php / python / elasticsearch issue.
I've since restored back to opn 23.1 + za 1.13.
I have an email into Sunny Valley regarding issue.  (Of course decided best day for upgrade was a Friday
I did get a backup of broken VM prior to delete/restore which I can later restore for more troubleshooting.  Or can clone production VM and re-do another 23.7 upgrade.
I just needed to get operational again and will follow-up testing more with secondary lab VM.

[JonStuart]

Instead of checking the update in a month's time, I recommend you to do it in a couple of centuries and see if it works well by then, in the meantime use Adguard:

https://forum.opnsense.org/index.php?topic=22162.225

Zenarmor is not just an ad block tool like adguard or pihole. It's a combination of that PLUS packet inspection which is the only REAL ad block. It combines two technologies together for a single purpose eliminating the need to have two separate programs (consuming more resources) to accomplish the task. This is especially useful when ransomware and malware try to reach out to control servers that don't use DNS because hackers have gotten smart. Instead they have hard coded IP's which completely bypass DNS. Even more....ad networks have also figured this out and are now hard-coding IP's into their ad code to bypass ad blockers such as adguard or pihole or even web browser plugins to stop ads. It is wreak-less to hand out advice about such things without understanding what each thing does and how it does it. For a simple ad block tool what you are suggesting might be fine.....but if anyone needs intrusion prevention (IPS) it is absolutely NOT ok. I should also note that the default rules for OPNsense Suracata are for basic protection ONLY. They are in no way useful for advanced protection nor are they the latest up to date rules. In fact they are NEVER newer than 30 days and most are 60 to 90 days old. If you want better rules you must pay for them.

In short I understand if you don't like this product and that is fine. Everyone has their preference....but please stop going from thread to thread bashing it. Just stop using it and let it be. People who are in here are trying to be productive and give valuable feedback.....you are not helping.

[depc80]

Have you tried to run a system audit and seeing what the results are? That looks like its related to php and/or python.

Goto System >> Firmware >> Status

Then choose "Health" from the "Run An Audit" button next to the  "Check for updates" button. You may also want to look at the "Upgrade" audit as well. Something isn't right though. An audit should help you find it.

I have the latest version of OPNsense and Zenarmor and it's working fine but on one older firewall (Been through many OPNsense updates) I had to manually fix some upgrade issues with OPNsense for a proper install of dependencies for Zenarmor. So you are probably looking for dependency issues in the audit.

I did not try the audit.  Good idea though.  Agree php / python / elasticsearch issue.
I've since restored back to opn 23.1 + za 1.13.
I have an email into Sunny Valley regarding issue.  (Of course decided best day for upgrade was a Friday
I did get a backup of broken VM prior to delete/restore which I can later restore for more troubleshooting.  Or can clone production VM and re-do another 23.7 upgrade.
I just needed to get operational again and will follow-up testing more with secondary lab VM.

I have the same issue with swap page at 100%. There was error about Sqlite.php need to change line 136 to 2048MB but I only use elastic search as database. I have to uninstall ZA for now and wait for the team responds. Love the new UI.

[ThyOnlySandman]

Have you tried to run a system audit and seeing what the results are? That looks like its related to php and/or python.

Goto System >> Firmware >> Status

Then choose "Health" from the "Run An Audit" button next to the  "Check for updates" button. You may also want to look at the "Upgrade" audit as well. Something isn't right though. An audit should help you find it.

I have the latest version of OPNsense and Zenarmor and it's working fine but on one older firewall (Been through many OPNsense updates) I had to manually fix some upgrade issues with OPNsense for a proper install of dependencies for Zenarmor. So you are probably looking for dependency issues in the audit.

Believe found culprit.  Sunny Valley responded requesting a remote session.
So I cloned the restored 23.1 to do a fresh upgrade again.

Upon upgrading to OPNsense 23.1.11_1-amd64 the log had:

Checking all packages: .......... done
py37-markupsafe has a missing dependency: python37
py37-markupsafe has a missing dependency: py37-setuptools
py37-markupsafe is missing a required shared library: libpython3.7m.so.1.0

>>> Missing package dependencies were detected.
>>> Found 2 issue(s) in the package database.

pkg-static: No packages available to install matching 'python37' have been found in the repositories
pkg-static: No packages available to install matching 'py37-setuptools' have been found in the repositories
>>> Summary of actions performed:

python37 dependency failed to be fixed
py37-setuptools dependency failed to be fixed

>>> There are still missing dependencies.
>>> Try fixing them manually.

Ran - pkg remove py37-markupsafe

Opnsense 23.1.11_1 + ZA 1.14.2 appeared stable. 
Then upgraded to OPNsense 23.7.1_3-amd64.  Also stable along with ZA 1.14.2.
And no more backend log errors in previous post.

So not a dependency issue exactly but rather a left over python 3.7 package that didn't get cleaned up with past upgrades.  It kinda scary one left over package can cause so much havoc.

Hopefully can help someone.  Prior to 23.7 I'd run health audit or double check only "py39-  " packages are installed.

EDIT
Nevermind .  Ran for 9+ hours without issue.  Then happened again...all resources consumed and crash.
And now all resources are consumed within 20 min uptime just like previous VM / upgrade.
Remove ZA and now 35 min uptime RAM use stable at ( 4063/12250 MB )
I give up without some dev direction...

spammed processes consuming resources = usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases

[fatbob01]

I have the same problem on a n100 mini pc. Funny enough, i have the exact same configuration on a esxi vm, no issues.  Had to uninstall zenarmor only on mini pc.  Both 8GB ram.  Let me know what zenarmor has to say.

Thanks!

[rudiservo]

Guys, for the sake of trying to figure what is going on, what type of nics are you guys running this.

I have the same problem on a n100 mini pc. Funny enough, i have the exact same configuration on a esxi vm, no issues.  Had to uninstall zenarmor only on mini pc.  Both 8GB ram.  Let me know what zenarmor has to say.

Thanks!

I am suspecting an issue with zenarmor talking with netmap, I have ZA only on one nic (realtek) and suricata on the wan.
The only time ZA does not go to 100% is if all my networks stop talking, not just the one it is attach to.

[fatbob01]

Guys, for the sake of trying to figure what is going on, what type of nics are you guys running this.

I have the same problem on a n100 mini pc. Funny enough, i have the exact same configuration on a esxi vm, no issues.  Had to uninstall zenarmor only on mini pc.  Both 8GB ram.  Let me know what zenarmor has to say.

Thanks!

I am suspecting an issue with zenarmor talking with netmap, I have ZA only on one nic (realtek) and suricata on the wan.
The only time ZA does not go to 100% is if all my networks stop talking, not just the one it is attach to.

Intel i226V for all ports, zenarmor configured on lan and IoT VLAN.  Suricata on wan