OPNsense Forum
English Forums => Zenarmor (Sensei) => Topic started by: ThyOnlySandman on August 12, 2023, 03:53:40 am
-
Updated to 23.7 today. System resource usage out of control and crashing opnsense within 20 minutes.
VMware VM - 12 CPU core / 12GB RAM / 8GB SWAP.
Around 20 min of uptime RAM + SWAP 100% + CPU 100%.
Been running this VM long time throughout many Opnsense versions without issue. Normally very low CPU and ~60-70% RAM use with both zenarmor + surcata + few others.
Despite turning off Zenarmor from starting engine + elasticsearch at boot I still see Zenarmor processes in a "ps axmfv" using excessive CPU - So thinking it's zenarmor...
Anyone seen this?
Going to give Zenarmor full uninstall + reinstall momentarily. If that doesn't fix then Veeam VM restore back to 23.1
-
PID STAT TIME SL RE PAGEIN VSZ RSS LIM TSIZ %CPU %MEM COMMAND
327 R 3:19.71 127 12 0 776148 10476 - 3384 28.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
485 R 2:10.97 127 127 0 776148 707568 - 3384 28.0 5.6 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
803 R 1:04.09 127 127 0 776148 709516 - 3384 30.0 5.7 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
7462 R 3:58.24 127 127 29 776148 650952 - 3384 31.0 5.2 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
13624 R 8:20.46 127 12 17237 776180 9036 - 3384 30.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
16332 R 4:27.52 127 12 4300 776148 8908 - 3384 31.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
26000 S 1:44.36 1 1 92 81928 21872 - 3384 0.0 0.2 /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_watcher.php interface routes alarm
30046 R 6:22.34 127 12 9509 776148 8816 - 3384 28.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
31736 R 7:08.05 127 12 5007 776148 8820 - 3384 29.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
36510 R 5:44.67 127 12 2873 776180 9060 - 3384 29.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
41468 R 10:55.15 127 12 11959 776148 11228 - 3384 30.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
42713 D 7:46.38 0 12 5226 776148 8992 - 3384 31.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
46351 R 11:28.01 127 12 17286 776148 11120 - 3384 29.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
48021 R 0:32.50 127 127 0 776148 709544 - 3384 30.0 5.7 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
49063 S 5:03.43 0 12 2803 776180 8916 - 3384 28.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
51823 D 10:35.04 0 12 12930 776148 11152 - 3384 30.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
53271 R 10:19.38 127 12 20167 776148 11024 - 3384 29.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
53394 R 0:05.37 127 5 0 718804 672644 - 3384 41.0 5.4 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
59011 R 9:32.89 127 12 9386 776148 9032 - 3384 29.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
64584 S 10:05.75 0 12 5943 776180 9344 - 3384 32.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
66152 R 1:37.47 127 127 0 776148 709540 - 3384 30.0 5.7 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
77401 R 11:14.23 127 12 7703 776148 13228 - 3384 31.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
78895 R 9:09.37 127 12 1912 776148 11104 - 3384 29.0 0.1 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
84200 R 2:46.46 127 127 0 776180 709652 - 3384 31.0 5.7 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
96845 S 0:00.36 0 1 0 102408 73352 - 3384 3.0 0.6 /usr/local/bin/php /usr/local/etc/rc.routing_configure alarm
98164 R 0:09.26 127 12 0 776180 679960 - 3384 41.0 5.4 /usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
-
It’s Zenarmor 1.14.2 crashing OPNsense 23.7.1_3-amd64.
Uninstalled and system stable ~10% CPU + 32%(4GB) RAM for over 30 minutes.
Installed Zenarmor 1.14.2 and couldn’t even finish the initial Zenarmor config before the WebUI crashed and became inaccessible. While installing the DB I saw RAM jump to 90% prior to webui crash.
Rebooted and same observed behavior prior to reinstall.
18 min uptime
CPU usage 100 %
Memory usage 98 % ( 12076/12250 MB )
SWAP usage 99 % ( 8191/8192 MB )
Same behavior happens with Zenarmor engine + DB off at boot. So appears to be Zenarmor webUI / php issue.
-
Tried one more attempt uninstall via Zenarmor with delete all folders rather than uninstall via plugins.
Upon fully fresh install with zero config same issue.
The excessive processes show:
/usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
The backend log has:
2023-08-11T22:50:52-07:00 Error configd.py [046e3ca0-9ed2-4322-9aa0-4a5740b456e5] returned exit status 255
2023-08-11T22:47:43-07:00 Error configd.py [9b953469-e127-4741-8073-efa5f588d696] Script action stderr returned "b'2023.08.11 - 22:47:43 - INFO - [main] Starting ipdr retiring for ELASTICSEARCH...\n2023.08.11 - 22:47:43 - INFO - [conn] Rename alias to zenarmor_0000000000_c56fad43-5729-4727-858a-ff35e2cee6f6_conn \n2023.08.11 - 22:47:43 - INFO - [conn_all] Create New Ali'"
2023-08-11T22:43:58-07:00 Error configd.py [93b402db-76cc-4938-ace9-8065fb688881] Script action stderr returned "b'umount: /dev/md43: statfs: No such file or directory\numount: /dev/md43: unknown file system'"
2023-08-11T22:43:54-07:00 Error configd.py [45a13456-4081-4a2c-a34c-6fe3effd8f71] Script action failed with Command '/usr/local/zenarmor/scripts/installers/elasticsearch/create_indices.py '' ''' returned non-zero exit status 5. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 44, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/zenarmor/scripts/installers/elasticsearch/create_indices.py '' ''' returned non-zero exit status 5.
2023-08-11T22:43:38-07:00 Error configd.py [402ca649-9846-4c30-a750-42756b0b3cdf] Script action failed with Command '/usr/local/zenarmor/scripts/datastore/rename_alias_elasticsearch.py 'zenarmor_0000000000_c56fad43-5729-4727-858a-ff35e2cee6f6_'' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 44, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/zenarmor/scripts/datastore/rename_alias_elasticsearch.py 'zenarmor_0000000000_c56fad43-5729-4727-858a-ff35e2cee6f6_'' returned non-zero exit status 1.
Restoring back to Opn 23.1 + 13.x Zen. Revisit upgrade in a month.
-
Instead of checking the update in a month's time, I recommend you to do it in a couple of centuries and see if it works well by then, in the meantime use Adguard:
https://forum.opnsense.org/index.php?topic=22162.225
-
Adguard is NO IPS. You have no clue what you are talking about yeraycito.
-
Instead of checking the update in a month's time, I recommend you to do it in a couple of centuries and see if it works well by then, in the meantime use Adguard:
https://forum.opnsense.org/index.php?topic=22162.225
I take it you don't like Zenarmor...
Over the years I've only had 1 other issue on another deployment before. After month or two it would crash restarting protected LAN interface which would down network for 5 min but would recover. Believe it was an update that later repaired. Also believe it was shortly after a new Opnsense release. Just like now, but they've completely re-done UI too.
Zenarmor does more than ad blocking.
My main complaint with Zenarmor is their business / license model. I have a Fortigate 60F in transparent mode downstream this Opnsense I'm having issues with now. The Fortigate is a real NGFW. Zenarmor is not. The yearly pricing plans for business edition simply don't compete against a Fortigate UTP subscription . Having said that I still value the ads + app blocking + traffic analysis Zenarmor provides.
Briefly reviewing their 1.14 new version bit bummed that custom port applications require license. As does the https blocking page. Active directory agent now requires Enterprise.
Summary. Decent tool. Bad license model for provided value.
-
Instead of checking the update in a month's time, I recommend you to do it in a couple of centuries and see if it works well by then, in the meantime use Adguard:
https://forum.opnsense.org/index.php?topic=22162.225
I take it you don't like Zenarmor...
Over the years I've only had 1 other issue on another deployment before. After month or two it would crash restarting protected LAN interface which would down network for 5 min but would recover. Believe it was an update that later repaired. Also believe it was shortly after a new Opnsense release. Just like now, but they've completely re-done UI too.
Zenarmor does more than ad blocking.
My main complaint with Zenarmor is their business / license model. I have a Fortigate 60F in transparent mode downstream this Opnsense I'm having issues with now. The Fortigate is a real NGFW. Zenarmor is not. The yearly pricing plans for business edition simply don't compete against a Fortigate UTP subscription . Having said that I still value the ads + app blocking + traffic analysis Zenarmor provides.
Briefly reviewing their 1.14 new version bit bummed that custom port applications require license. As does the https blocking page. Active directory agent now requires Enterprise.
Summary. Decent tool. Bad license model for provided value.
Have you tried to run a system audit and seeing what the results are? That looks like its related to php and/or python.
Goto System >> Firmware >> Status
Then choose "Health" from the "Run An Audit" button next to the "Check for updates" button. You may also want to look at the "Upgrade" audit as well. Something isn't right though. An audit should help you find it.
I have the latest version of OPNsense and Zenarmor and it's working fine but on one older firewall (Been through many OPNsense updates) I had to manually fix some upgrade issues with OPNsense for a proper install of dependencies for Zenarmor. So you are probably looking for dependency issues in the audit.
-
Have you tried to run a system audit and seeing what the results are? That looks like its related to php and/or python.
Goto System >> Firmware >> Status
Then choose "Health" from the "Run An Audit" button next to the "Check for updates" button. You may also want to look at the "Upgrade" audit as well. Something isn't right though. An audit should help you find it.
I have the latest version of OPNsense and Zenarmor and it's working fine but on one older firewall (Been through many OPNsense updates) I had to manually fix some upgrade issues with OPNsense for a proper install of dependencies for Zenarmor. So you are probably looking for dependency issues in the audit.
I did not try the audit. Good idea though. Agree php / python / elasticsearch issue.
I've since restored back to opn 23.1 + za 1.13.
I have an email into Sunny Valley regarding issue. (Of course decided best day for upgrade was a Friday :)
I did get a backup of broken VM prior to delete/restore which I can later restore for more troubleshooting. Or can clone production VM and re-do another 23.7 upgrade.
I just needed to get operational again and will follow-up testing more with secondary lab VM.
-
Instead of checking the update in a month's time, I recommend you to do it in a couple of centuries and see if it works well by then, in the meantime use Adguard:
https://forum.opnsense.org/index.php?topic=22162.225
Zenarmor is not just an ad block tool like adguard or pihole. It's a combination of that PLUS packet inspection which is the only REAL ad block. It combines two technologies together for a single purpose eliminating the need to have two separate programs (consuming more resources) to accomplish the task. This is especially useful when ransomware and malware try to reach out to control servers that don't use DNS because hackers have gotten smart. Instead they have hard coded IP's which completely bypass DNS. Even more....ad networks have also figured this out and are now hard-coding IP's into their ad code to bypass ad blockers such as adguard or pihole or even web browser plugins to stop ads. It is wreak-less to hand out advice about such things without understanding what each thing does and how it does it. For a simple ad block tool what you are suggesting might be fine.....but if anyone needs intrusion prevention (IPS) it is absolutely NOT ok. I should also note that the default rules for OPNsense Suracata are for basic protection ONLY. They are in no way useful for advanced protection nor are they the latest up to date rules. In fact they are NEVER newer than 30 days and most are 60 to 90 days old. If you want better rules you must pay for them.
In short I understand if you don't like this product and that is fine. Everyone has their preference....but please stop going from thread to thread bashing it. Just stop using it and let it be. People who are in here are trying to be productive and give valuable feedback.....you are not helping.
-
Have you tried to run a system audit and seeing what the results are? That looks like its related to php and/or python.
Goto System >> Firmware >> Status
Then choose "Health" from the "Run An Audit" button next to the "Check for updates" button. You may also want to look at the "Upgrade" audit as well. Something isn't right though. An audit should help you find it.
I have the latest version of OPNsense and Zenarmor and it's working fine but on one older firewall (Been through many OPNsense updates) I had to manually fix some upgrade issues with OPNsense for a proper install of dependencies for Zenarmor. So you are probably looking for dependency issues in the audit.
I did not try the audit. Good idea though. Agree php / python / elasticsearch issue.
I've since restored back to opn 23.1 + za 1.13.
I have an email into Sunny Valley regarding issue. (Of course decided best day for upgrade was a Friday :)
I did get a backup of broken VM prior to delete/restore which I can later restore for more troubleshooting. Or can clone production VM and re-do another 23.7 upgrade.
I just needed to get operational again and will follow-up testing more with secondary lab VM.
I have the same issue with swap page at 100%. There was error about Sqlite.php need to change line 136 to 2048MB but I only use elastic search as database. I have to uninstall ZA for now and wait for the team responds. Love the new UI.
-
Have you tried to run a system audit and seeing what the results are? That looks like its related to php and/or python.
Goto System >> Firmware >> Status
Then choose "Health" from the "Run An Audit" button next to the "Check for updates" button. You may also want to look at the "Upgrade" audit as well. Something isn't right though. An audit should help you find it.
I have the latest version of OPNsense and Zenarmor and it's working fine but on one older firewall (Been through many OPNsense updates) I had to manually fix some upgrade issues with OPNsense for a proper install of dependencies for Zenarmor. So you are probably looking for dependency issues in the audit.
Believe found culprit. Sunny Valley responded requesting a remote session.
So I cloned the restored 23.1 to do a fresh upgrade again.
Upon upgrading to OPNsense 23.1.11_1-amd64 the log had:
Checking all packages: .......... done
py37-markupsafe has a missing dependency: python37
py37-markupsafe has a missing dependency: py37-setuptools
py37-markupsafe is missing a required shared library: libpython3.7m.so.1.0
>>> Missing package dependencies were detected.
>>> Found 2 issue(s) in the package database.
pkg-static: No packages available to install matching 'python37' have been found in the repositories
pkg-static: No packages available to install matching 'py37-setuptools' have been found in the repositories
>>> Summary of actions performed:
python37 dependency failed to be fixed
py37-setuptools dependency failed to be fixed
>>> There are still missing dependencies.
>>> Try fixing them manually.
Ran - pkg remove py37-markupsafe
Opnsense 23.1.11_1 + ZA 1.14.2 appeared stable.
Then upgraded to OPNsense 23.7.1_3-amd64. Also stable along with ZA 1.14.2.
And no more backend log errors in previous post.
So not a dependency issue exactly but rather a left over python 3.7 package that didn't get cleaned up with past upgrades. It kinda scary one left over package can cause so much havoc.
Hopefully can help someone. Prior to 23.7 I'd run health audit or double check only "py39- " packages are installed.
EDIT
Nevermind . Ran for 9+ hours without issue. Then happened again...all resources consumed and crash.
And now all resources are consumed within 20 min uptime just like previous VM / upgrade.
Remove ZA and now 35 min uptime RAM use stable at ( 4063/12250 MB )
I give up without some dev direction...
spammed processes consuming resources = usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
-
I have the same problem on a n100 mini pc. Funny enough, i have the exact same configuration on a esxi vm, no issues. Had to uninstall zenarmor only on mini pc. Both 8GB ram. Let me know what zenarmor has to say.
Thanks!
-
Guys, for the sake of trying to figure what is going on, what type of nics are you guys running this.
I have the same problem on a n100 mini pc. Funny enough, i have the exact same configuration on a esxi vm, no issues. Had to uninstall zenarmor only on mini pc. Both 8GB ram. Let me know what zenarmor has to say.
Thanks!
I am suspecting an issue with zenarmor talking with netmap, I have ZA only on one nic (realtek) and suricata on the wan.
The only time ZA does not go to 100% is if all my networks stop talking, not just the one it is attach to.
-
Guys, for the sake of trying to figure what is going on, what type of nics are you guys running this.
I have the same problem on a n100 mini pc. Funny enough, i have the exact same configuration on a esxi vm, no issues. Had to uninstall zenarmor only on mini pc. Both 8GB ram. Let me know what zenarmor has to say.
Thanks!
I am suspecting an issue with zenarmor talking with netmap, I have ZA only on one nic (realtek) and suricata on the wan.
The only time ZA does not go to 100% is if all my networks stop talking, not just the one it is attach to.
Intel i226V for all ports, zenarmor configured on lan and IoT VLAN. Suricata on wan
-
Guys, for the sake of trying to figure what is going on, what type of nics are you guys running this.
I am suspecting an issue with zenarmor talking with netmap, I have ZA only on one nic (realtek) and suricata on the wan.
The only time ZA does not go to 100% is if all my networks stop talking, not just the one it is attach to.
ESXI VMXNET3 NICs. Native netmap. Zenarmor LAN. Suricata WAN.
However it does not appear related to traffic inspect. Issue happens with both Zenarmor + Elasticsearch off. Even happens on new lab VM without completing the post install setup of Zenarmor. (Prior to pick DB type, Choose protected interface, etc.)
There will be multiple processes of - usr/local/bin/php /usr/local/opnsense/mvc/app/library/OPNsense/Zenarmor/CLI.php aliases
I ran handful of tests disabling all addons except unbound to see if I could identify a conflict with another addon. Just ZA being installed. Not even running.
I have the same problem on a n100 mini pc. Funny enough, i have the exact same configuration on a esxi vm, no issues. Had to uninstall zenarmor only on mini pc. Both 8GB ram. Let me know what zenarmor has to say.
Thanks!
Interesting and strange. I shared more details with Sunny Valley but have not gotten a response.
-
I was able to fix it, I had to reset to factory defaults.
It's in the uninstall tab.
Nonetheless there is another issue, even if you haverouted native netmap, ZA will use emulated netmap.
It's in the Opnsense general logs (debug)
-
I switch to mongodb as database instead of Elasticsearch since mimugmail repo also uses elasticsearch. Would it be the cause of out of swap page?
-
Make the tunable dev.netmap.buf_num , set it to 70000 and reboot. YMMV.
-
I have the same problem on a n100 mini pc. Funny enough, i have the exact same configuration on a esxi vm, no issues. Had to uninstall zenarmor only on mini pc. Both 8GB ram. Let me know what zenarmor has to say.
Thanks!
I received a response from Sunny Valley today.
"We have a fix for your issue in 1.14.3"
installed 1.14.3 (Existing ZA config still). Installed ElasticSearch + set native Netmap Protect LAN + OPENVPN INTs.
Didn't reboot, 7 hours uptime since install. So far so good.
12 vCPU ~5-25%
Memory usage = 52 % ( 6384/12250 MB )
SWAP usage = 8 % ( 700/8192 MB )