[Solved] netdata not working

[opns_neuling]

Hello!
After update to 20.7.1 netdata is not working...

ps awux |grep -i netdata
netdata  5317   0.0  0.0   18472   8116  -  SN   14:01       0:00.08 /usr/local/libexec/netdata/plugins.d/apps.plugin 1
netdata 14509   0.0  0.1   54720  30480  -  SN   14:01       0:01.19 /usr/local/bin/python3.7 /usr/local/libexec/netdata/plugins.d/python.d.plugin 1
netdata 17179   0.0  0.1   84900  23820  -  IN   14:00       0:13.80 /usr/local/sbin/netdata -u netdata -P /var/db/netdata/netdata.pid
netdata 23227   0.0  0.0   18420   8416  -  IN   14:00       0:01.03 /usr/local/sbin/netdata --special-spawn-server
root    50945   0.0  0.0 1060960   3244  0  R+   14:09       0:00.00 grep -i netdata


netstat -l -n -4 |grep 10999 (not opened port) !!! 10999


dmesg
pid 30949 (netdata), jid 0, uid 302: exited on signal 11 (core dumped)


removed, removed netdata cache dirs, etc, and new install ....

Cheers

[Davesworld]

I thought I read that nobody is maintaining the plugin but for some reason mine is working now. Is yours still broken after the 20.7.1 update?

[mimugmail]

There is another post here with solution, you have to remove a special dir. Just search the forums

[opns_neuling]

tested ...
https://github.com/netdata/netdata/issues/9521
But this seems to not resolve the issue.

[Fright]

"netstat -l -n -4 |grep 10999 (not opened port) !!! 10999"
you sure about port number?

[opns_neuling]

yes, and before upgrade has worked :-)

[Fright]

is there anything interesting in /var/log/netdata/error.log ?

[opns_neuling]

Hi !

what I've tested so far

pkg delete os-netdata & pkg delete netdata
(manual remove from netdata rest files also cache dirs, logs dirs, db dir, etc ...)
pkg install os-netdata

Log Files ..

error.log only ...
access und debug are empty ...

2020-09-01 10:57:05: netdata INFO  : MAIN : SIGNAL: Received SIGTERM. Cleaning up to exit...
2020-09-01 10:57:05: netdata ERROR : PLUGINSD[apps] : read failed: end of file (errno 9, Bad file descriptor)
2020-09-01 10:57:05: netdata INFO  : PLUGINSD[apps] : PARSER ended
2020-09-01 10:57:05: netdata INFO  : MAIN : Shutting down command server.
2020-09-01 10:57:05: netdata ERROR : PLUGINSD[python.d] : read failed: end of file (errno 9, Bad file descriptor)
2020-09-01 10:57:05: netdata INFO  : MAIN : Shutting down command event loop.
2020-09-01 10:57:05: netdata ERROR : PLUGINSD[apps] : '/usr/local/libexec/netdata/plugins.d/apps.plugin' (pid 2073) disconnected after 41699 successful data collections (ENDs).
2020-09-01 10:57:05: netdata INFO  : PLUGINSD[python.d] : PARSER ended
2020-09-01 10:57:05: netdata ERROR : PLUGINSD[python.d] : '/usr/local/libexec/netdata/plugins.d/python.d.plugin' (pid 77871) disconnected after 10820 successful data collections (ENDs).
2020-09-01 10:57:05: netdata INFO  : MAIN : Shutting down command loop complete.
2020-09-01 10:57:05: netdata INFO  : MAIN : Command server has stopped.
2020-09-01 10:57:05: netdata INFO  : MAIN : EXIT: netdata prepares to exit with code 0...
2020-09-01 10:57:05: netdata INFO  : MAIN : /usr/local/libexec/netdata/plugins.d/anonymous-statistics.sh 'EXIT' 'OK' '-'
2020-09-01 10:57:05: netdata ERROR : PLUGINSD[apps] : child pid 2073 killed by signal 15.
2020-09-01 10:57:05: netdata INFO  : PLUGINSD[apps] : '/usr/local/libexec/netdata/plugins.d/apps.plugin' (pid 2073) was killed with SIGTERM. Disabling it.
2020-09-01 10:57:05: netdata INFO  : PLUGINSD[apps] : thread with task id 101071 finished
2020-09-01 10:57:05: netdata ERROR : PLUGINSD[python.d] : child pid 77871 killed by signal 15.
2020-09-01 10:57:05: netdata INFO  : PLUGINSD[python.d] : '/usr/local/libexec/netdata/plugins.d/python.d.plugin' (pid 77871) was killed with SIGTERM. Disabling it.
2020-09-01 10:57:05: netdata INFO  : PLUGINSD[python.d] : thread with task id 101080 finished
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: cleaning up the database...
2020-09-01 10:57:06: netdata INFO  : MAIN : Cleaning up database [1 hosts(s)]...
2020-09-01 10:57:06: netdata INFO  : MAIN : Cleaning up database of host 'hrouter1.xxx'...
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: stopping static threads...
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: Stopping main thread: PLUGIN[freebsd]
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: Stopping main thread: PLUGIN[idlejitter]
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: Stopping main thread: STATSD
2020-09-01 10:57:06: netdata INFO  : PLUGIN[freebsd] : cleaning up...
2020-09-01 10:57:06: netdata INFO  : PLUGIN[freebsd] : thread with task id 101044 finished
2020-09-01 10:57:06: netdata INFO  : PLUGIN[idlejitter] : cleaning up...
2020-09-01 10:57:06: netdata INFO  : PLUGIN[idlejitter] : thread with task id 101045 finished
2020-09-01 10:57:06: netdata INFO  : STATSD : cleaning up...
2020-09-01 10:57:06: netdata INFO  : STATSD : STATSD: stopping data collection thread 1...
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: Stopping main thread: WEB_SERVER[static1]
2020-09-01 10:57:06: netdata INFO  : STATSD : STATSD: closing sockets...
2020-09-01 10:57:06: netdata INFO  : STATSD_COLLECTOR[1] : cleaning up...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : freeing local web clients cache...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : stopped after 0 connects, 0 disconnects (max concurrent 0), 0 receptions and 0 sends
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : stopping worker 2
2020-09-01 10:57:06: netdata INFO  : STATSD : STATSD: cleanup completed.
2020-09-01 10:57:06: netdata INFO  : STATSD : thread with task id 101046 finished
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static2] : freeing local web clients cache...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static2] : stopped after 0 connects, 0 disconnects (max concurrent 0), 0 receptions and 0 sends
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static2] : thread with task id 101070 finished
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: Stopping main thread: PLUGINSD
2020-09-01 10:57:06: netdata INFO  : STATSD_COLLECTOR[1] : thread with task id 101081 finished
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : stopping worker 3
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: Stopping main thread: HEALTH
2020-09-01 10:57:06: netdata INFO  : PLUGINSD : cleaning up...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : stopping worker 4
2020-09-01 10:57:06: netdata INFO  : MAIN : Waiting 6 threads to finish...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static4] : freeing local web clients cache...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static4] : stopped after 0 connects, 0 disconnects (max concurrent 0), 0 receptions and 0 sends
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static4] : thread with task id 101074 finished
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : stopping worker 5
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static3] : freeing local web clients cache...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static5] : freeing local web clients cache...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static5] : stopped after 0 connects, 0 disconnects (max concurrent 0), 0 receptions and 0 sends
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static5] : thread with task id 101076 finished
2020-09-01 10:57:06: netdata INFO  : PLUGINSD : cleanup completed.
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static3] : stopped after 0 connects, 0 disconnects (max concurrent 0), 0 receptions and 0 sends
2020-09-01 10:57:06: netdata INFO  : HEALTH : cleaning up...
2020-09-01 10:57:06: netdata INFO  : HEALTH : thread with task id 101052 finished
2020-09-01 10:57:06: netdata INFO  : PLUGINSD : thread with task id 101051 finished
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static3] : thread with task id 101072 finished
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : stopping worker 6
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : Waiting 5 static web threads to finish...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static6] : freeing local web clients cache...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static6] : stopped after 0 connects, 0 disconnects (max concurrent 0), 0 receptions and 0 sends
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static6] : thread with task id 101078 finished
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : closing all web server sockets...
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : all static web threads stopped.
2020-09-01 10:57:06: netdata INFO  : WEB_SERVER[static1] : thread with task id 101049 finished
2020-09-01 10:57:06: netdata INFO  : MAIN : All threads finished.
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: freeing database memory...
2020-09-01 10:57:06: netdata INFO  : MAIN : Freeing all memory for host 'hrouter1.xxx'...
2020-09-01 10:57:06: netdata INFO  : MAIN : Shutting down RRD engine event loop.
2020-09-01 10:57:06: netdata INFO  : MAIN : Shutting down RRD engine event loop complete.
2020-09-01 10:57:06: netdata INFO  : MAIN : Shutting down RRD metadata log event loop.
2020-09-01 10:57:06: netdata INFO  : MAIN : Shutting down RRD metadata log loop complete.
2020-09-01 10:57:06: netdata INFO  : MAIN : Freed 7976400 bytes of memory from page cache.
2020-09-01 10:57:06: netdata INFO  : MAIN : SYSTEM_INFO: free 0x451a67fc300
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: removing netdata PID file '/var/db/netdata/netdata.pid'...
2020-09-01 10:57:06: netdata INFO  : MAIN : EXIT: all done - netdata is now exiting - bye bye...
EOF found in spawn pipe.
Shutting down spawn server event loop.
Shutting down spawn server loop complete.
2020-09-01 10:57:07: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 937368, max = 937368
2020-09-01 10:57:07: netdata ERROR : MAIN : Out-Of-Memory (OOM) score setting is not supported on this system. (errno 2, No such file or directory)
2020-09-01 10:57:07: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to other (2), with priority 0. Falling back to nice. (errno 1, Operation not permitted)
2020-09-01 10:57:07: netdata INFO  : MAIN : Running with process scheduling policy 'other', nice level 19
2020-09-01 10:57:07: netdata INFO  : MAIN : netdata started on pid 37061.
2020-09-01 10:57:07: netdata INFO  : MAIN : Initializing spawn client.
2020-09-01 10:57:07: netdata INFO  : MAIN : Executing /usr/local/libexec/netdata/plugins.d/system-info.sh
Spawn server is up.
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_NAME=none
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID=none
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID_LIKE=none
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION=none
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION_ID=none
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_DETECTION=none
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_HOST_OS_NAME=FreeBSD
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_HOST_OS_ID=FreeBSD
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_HOST_OS_ID_LIKE=FreeBSD
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION=12.1-RELEASE-p8-HBSD
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION_ID=unknown
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_HOST_OS_DETECTION=uname
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_NAME=FreeBSD
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=1201000
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_ARCHITECTURE=amd64
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=none
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=none
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER=unknown
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=none
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=8
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_VENDOR=unknown
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_MODEL=Intel(R) Xeon(R) CPU E3-1230 v5 @ 3.40GHz
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_FREQ=unknown
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_DETECTION=sysctl
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_RAM=34128646144
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_RAM_DETECTION=sysctl
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_DISK_SIZE=224189005824
2020-09-01 10:57:08: netdata INFO  : MAIN : NETDATA_SYSTEM_DISK_DETECTION=df
2020-09-01 10:57:08: netdata INFO  : MAIN : Configuring locking mechanism for global GUID map
2020-09-01 10:57:08: netdata INFO  : MAIN : Cannot open the file /var/db/netdata/health.silencers.json, so Netdata will work with the default health configuration.
2020-09-01 10:57:08: netdata INFO  : MAIN : CONFIG: cannot load user config '/usr/local/etc/netdata/stream.conf'. Will try stock config.
2020-09-01 10:57:08: netdata ERROR : MAIN : HEALTH [hrouter1.xxx]: cannot open health file: /var/db/netdata/health/health-log.db.old (errno 2, No such file or directory)
2020-09-01 10:57:08: netdata INFO  : MAIN : Found 3 files in path /var/cache/netdata/dbengine
2020-09-01 10:57:08: netdata INFO  : MAIN : Scanning file "/var/cache/netdata/dbengine/datafile-1-0000000001.ndf"
2020-09-01 10:57:08: netdata INFO  : MAIN : Matched file "/var/cache/netdata/dbengine/datafile-1-0000000001.ndf"
2020-09-01 10:57:08: netdata INFO  : MAIN : Scanning file "/var/cache/netdata/dbengine/journalfile-1-0000000001.njf"
2020-09-01 10:57:08: netdata INFO  : MAIN : Scanning file "/var/cache/netdata/dbengine/metadatalog-00000-00001.mlf"
2020-09-01 10:57:08: netdata INFO  : MAIN : Initializing data file "/var/cache/netdata/dbengine/datafile-1-0000000001.ndf".
2020-09-01 10:57:08: netdata INFO  : MAIN : Data file "/var/cache/netdata/dbengine/datafile-1-0000000001.ndf" initialized (size:2260992).
2020-09-01 10:57:08: netdata INFO  : MAIN : Loading journal file "/var/cache/netdata/dbengine/journalfile-1-0000000001.njf".
2020-09-01 10:57:08: netdata INFO  : MAIN : Journal file "/var/cache/netdata/dbengine/journalfile-1-0000000001.njf" loaded (size:208896).
2020-09-01 10:57:08: netdata INFO  : MAIN : Found 3 files in path /var/cache/netdata/dbengine
2020-09-01 10:57:08: netdata INFO  : MAIN : Scanning file "/var/cache/netdata/dbengine/datafile-1-0000000001.ndf"
2020-09-01 10:57:08: netdata INFO  : MAIN : Scanning file "/var/cache/netdata/dbengine/journalfile-1-0000000001.njf"
2020-09-01 10:57:08: netdata INFO  : MAIN : Scanning file "/var/cache/netdata/dbengine/metadatalog-00000-00001.mlf"
2020-09-01 10:57:08: netdata INFO  : MAIN : Matched file "/var/cache/netdata/dbengine/metadatalog-00000-00001.mlf"
2020-09-01 10:57:08: netdata INFO  : MAIN : Loading metadata log "/var/cache/netdata/dbengine/metadatalog-00000-00001.mlf".
EOF found in spawn pipe.
Shutting down spawn server event loop.
Shutting down spawn server loop complete.

[opns_neuling]

Observation:
after reboot symbolic links refer each other !

ls -altr /root/var/cache/netdata /root/var/log/netdata /root/var/db/netdata
lrwxr-xr-x  1 root  wheel  23 Sep  3 10:59 /root/var/cache/netdata -> /root/var/cache/netdata
lrwxr-xr-x  1 root  wheel  20 Sep  3 10:59 /root/var/db/netdata -> /root/var/db/netdata
lrwxr-xr-x  1 root  wheel  21 Sep  3 10:59 /root/var/log/netdata -> /root/var/log/netdata

ls -latr /var/log/netdata /var/cache/netdata /var/db/netdata
lrwxr-xr-x  1 root  wheel  23 Sep  4 12:19 /var/cache/netdata -> /root/var/cache/netdata
lrwxr-xr-x  1 root  wheel  20 Sep  4 12:19 /var/db/netdata -> /root/var/db/netdata
lrwxr-xr-x  1 root  wheel  21 Sep  4 12:19 /var/log/netdata -> /root/var/log/netdata

[Fright]

little confused
can you delete error.log
try to start netdata two or three times and attach error.log?

[opns_neuling]

Solution

pkg delete os-netdata
pkg delete netdata
rm -fr /var/log/netdata /var/db/netdata var/cache/netdata
rm -fr /root/var/cache/netdata /root/var/db/netdata /root/var/log/netdata
rm -fr /usr/local/etc/netdata /var/cache/netdata /var/mail/netdata

pkg install os-netdata
service netdata start

Also, solved :-)