OPNsense Forum
English Forums => Zenarmor (Sensei) => Topic started by: Giant850 on January 27, 2022, 08:14:14 pm
-
Just upgraded to Opnsense 22.1, then upgraded Zenarmor. Now Elasticsearch won't start. Packet engine and cloud agent are running. Ideas?
-
Hi @Giant850,
Can you run the following two commands and paste the results here:
pkg info | grep elasticsearch | cut -d" " -f1 | xargs pkg info
service elasticsearch status
-
I had the same.
Under Zenarmor - status -> Engine Version - I hit check updates and also App & Rules DB Version - I hit check updates.
after that I got an green notification on top from zenarmor about "noticing OS change" or something and did some reinstall of elastic search, after the notification the system came back online.
-
@RamSense,
Yes, this is how it's supposed to happen.
After you switch to 22.1, zenarmor should detect the ABI change and re-install database with the correct ABI.
-
Morning,
with my update to 22.1 the elasticsearch does not start.
"Error occured during elasticsearch start process"
installed and reinstalled
elasticsearch5 5.6.16_8 72.9MiB SunnyValley APACHE20
os-sensei 1.10.1 120MiB SunnyValley BSD2CLAUSE
os-sunnyvalley 1.2_1 652B OPNsense BSD2CLAUSE
any tip for me?
-
hm.. as i had a backup i decided to reinstall zenarmor and elasticsearch.
no luck with that. DB does not start and system behaves very unstable.
-
Same issue here. Cannot start the Elasticsearch database.
I did upgrade to 22.1 as well as the Zenarmor packages.
-
twice reinstalled..
1st attempt with restore backup - elastic stopped
2ns with default config and just one interface - went OK
Started activating options, exclusion etc.
System went unstable and after reboot elasitc stopped.
besides is get this error as soon as i install sensei
[28-Jan-2022 10:38:50 Europe/Amsterdam] PHP Warning: A non-numeric value encountered in /usr/local/sbin/carp_service_status on line 51
-
Same issue here. OPNsense reinstalled Sensei after upgrading to 22.1 but Elasticsearch won’t start.
How do I force the reinstall?
-
Altight, I reinstalled elasticsearch manually from CLI. Now it works, for some reason it was still on FreeBSD 12 ABI. Manual reinstall fixed and it is now 13 and everything is working fine
-
Can you paste the CLI command code here?
So i can give it a try.
i installed three times through GUI.. failed all the time.
thanks Armin
-
First, I did:
pkg deinstall elasticsearch5
After that:
pkg install elasticsearch5
You can use pkg info command to check the exact name of the package if the above does not work.
I am not sure if this was too "brute force" approach. I compared the installed packages with other identical systems I have running OK on 22.1 and the packages seem identical.
I upgraded multiple "identical" machines from 21.7 to 22.1 with same procedure. No idea what went differently here...
-
Thanks hfvk
that did the trick.
pkg remove elasticsearh5
pkg install elasticsearch5
then enable and restore config&rules
-
Hi all,
The repo is updated and it should be worked as expected.
-
Thanks Sy,
but my issue came back right after reboot.
Elasticsearch does not start. System gehts unstable and reloads services every 1-2 minutes.
tried cli uninstall / install
removed elasticsearch user/group
removed left over files and directories.
reinstalled through GUI
still no luck.
Had 2 boxes in the office 1 was OK the other failed as well.
-
Hi @Armin,
For the early updaters, It seems that the new repo is installing without any issue. But It should be uninstalled from Zenarmor GUI (Configuration - Uninstall). Otherwise, some files don't remove and occur error.
And @Armin has still an error after reboot. There was a problem with index files and resolved by resetting report files (Configuration - Reporting & Data - Reset Reporting).
-
Hi,
thanks, that did the trick for me!
br
-
bad news..
issue came back after antoher reboot. Even worse. System resettet itself every 1-2 minutes.
was pretty hard to get it back to work.
Had to
service elasticsearch stop
rm -rf /etc/rc.conf.d/elasticsearch
rm -rf /usr/local/opnsense/service/templates/OPNsense/Sensei/rc.conf.d/elasticsearch
rm -rf /usr/local/etc/elasticsearch
rm -rf /usr/local/etc/rc.d/elasticsearch
rm -rf /usr/local/libexec/elasticsearch
rm -rf /usr/local/libexec/elasticsearch/bin/elasticsearch
rm -rf /usr/local/lib/elasticsearch
rm -rf /usr/local/lib/elasticsearch/bin/elasticsearch
rm -rf /usr/local/sensei/scripts/database/templates/rc.conf.d/elasticsearch
rm -rf /usr/local/sensei/scripts/updater/elasticsearch
rm -rf /usr/local/sensei/scripts/installers/elasticsearch
rm -rf /usr/local/datastore/elasticsearch
rm -rf /var/mail/elasticsearch
rm -rf /var/run/elasticsearch
rm -rf /var/db/elasticsearch
rm -rf /var/tmp/elasticsearch
rm -rf /var/log/elasticsearch
directly from cli. Right after the next reset it came back stable.
I am not sure what this is all about.
What i can say is that this is elasticsearch related.
I guess there is something left over which causes the trouble for elasticsearch.
So worse that it brings the system to reset all the time.
I can run the system stable with zenarmor and mongodb.
It might would be time to reinstall a 22 version and restore the backups.
-
Hi Armin,
Is there any news on the issue after your support email? I think it is up now with Mongo DB. But it should be a system or hardware issue. When we checked it, the indexes were broken for a while.
-
Hi Sy,
it is an elasticsearch issue which came with zenarmor at the time.
At the moment zenarmor runs with mongodb.
From system side of view i wrote that i might install a fresh build as the system was upgraded several times.
Just to strike out all old left over files of elasticsearch or zenarmor which cause the issue if reinstalled.
thanks for your help and support.
armin
-
I got the same issue on 22.1 after I updated sunnyvalley to 1.2.1, os-sesi to 1.10.1.
So, I uninstalled and reinstalled zenarmor, while coiffuring ZI get the error in hardware configuration "We could not locate ubench package, tried to install it, and installation did not work.".
Help pls.
-
I got the same issue on 22.1 after I updated sunnyvalley to 1.2.1, os-sesi to 1.10.1.
So, I uninstalled and reinstalled zenarmor, while coiffuring ZI get the error in hardware configuration "We could not locate ubench package, tried to install it, and installation did not work.".
Help pls.
I'm seeing the same thing wondering if it's possibly the repo.
-
Hi,
Can you share the commands output;
pkg clean
pkg update
pkg search ubench
-
Hi,
Can you share the commands output;
pkg clean
pkg update
pkg search ubench
I have reinstalled Zenarmor but did not configure again yet because of the ubench error I got yesterday.
pkg clean output
root@router:~ # pkg clean
The following package files will be deleted:
/var/cache/pkg/os-sunnyvalley-1.2_1~c903d6c917.txz
/var/cache/pkg/os-sensei-1.10.1~2e63a9cffd.txz
/var/cache/pkg/os-sunnyvalley-1.2_1.txz
/var/cache/pkg/os-sensei-updater-1.10.txz
/var/cache/pkg/os-sensei-1.10.1.txz
/var/cache/pkg/os-sensei-updater-1.10~9249ba3dac.txz
/var/cache/pkg/ubench-0.32~bbd4f2ad21.txz
/var/cache/pkg/ubench-0.32.txz
/var/cache/pkg/os-sensei-agent-1.10.5~f5aa1cfce9.txz
/var/cache/pkg/os-sensei-agent-1.10.5.txz
The cleanup will free 48 MiB
Proceed with cleaning the cache? [y/N]: y
oot@router:~ # pkg update
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
SunnyValley repository is up to date.
All repositories are up to date.
pkg search ubench
ubench-0.32 Unix Benchmark Utility for CPU(s) and memory
By the way, I hit Y to pkg clean and cleaned out the pkgs.
-
Hi,
What if you install ubench manually?
pkg install -fy ubench
-
Hi,
What if you install ubench manually?
pkg install -fy ubench
When I try to install ubench manually it gave the beloww out put, but I think after running pkg clean something good happened. The configuration went on and completed.
pkg install -y ubench
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
SunnyValley repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The most recent versions of packages are already installed
Thank you for the pkg clean suggestion, I believe that worked.
-
I just waited and I can run through no issues today.
-
Ok, i tried it now with
Elasticsearch
MongoDB
SQLite
All installations start failing after a while.
I guess it is time to reinstall OPNSense.
SQLite worked the longest time. 1-2 month. While the other two failed after days/weeks.
Reduce reporting to one day. Resetted, reinstalled etc...
-
Just like Ying & Yang - always something good on something bad.
this morning i reinstalled the firewall completely.
Was pretty straight forward with the configuration import.
Took 30 minutes..
So for now fresh system and Zenarmor is installed with the default
Elasticsearch
100 Devices
5 Days reporting data
As ADS and Tracker can be blocked by Zen i removed DNSBL.
And added all the o365 ranges for better traffic.
Lets see how it runs from now on.
-
Well, right after one day elasticsearch died again....
Reinstalled yesterday
BUT restored configuration. Not sure if this can cause an error.
Zenarmor packed engine is running.
Could this be a hardware issue?
Am i somehow able to gather more details, logs etc?
-
Hi,
Please send a bug report from the upper right corner of Zenarmor GUI.
-
Hi Sy,
did and dropped a mail to Matt as well.
No: 4669
In the meanwhile i reinstalled it from scratch.
This time without the configuration import.
Also replaced the harddisk as the ssd was not healthy in smarttools.
-
I have been following this post and dealing with similar challenges with ZA and elasticsearch. Right now I am reverting back to using Sqlite, which has worked in the past. Lately, I cannot get ZA to stay open more than 24 hours with elasticsearch. ArminF, have you found a solution these past few days?
-
It's most likely not a hardware issue because I'm also experiencing this issue. Everything is similar to yours.
-
Hi,
What is the error in elasticsearch log when the issue has occurred?
-
I have the same issue... I always have to restart it every morning. Sometimes it don't even start again until reboot. Kinda annoying, I pay 10$/month for a unstable service.
Version 22.7.4
Thank you.
-
Hi,
Can you share a bug report?