20.1.4 - ntopng

[gpb]

Thanks Franco!  Just updated no issues, but am seeing higher than normal CPU that looks to be attributable to ntopng.  It varies between 50 and 100 (gui) vs normal idle of between 0 and 25.  Looking at it via "top" it looks very high (50 to 300%).  Not sure...maybe others can chime in...I've disabled it for now.  Looks like that's an updated package too, but no mention in the release notes (?).  Again, thanks and stay healthy!

[cryogenic666]

It's the new 4.0 version of NtopNG, actually, and I'm seeing the same thing myself. It also doesn't use the same GeoIP databases and doesn't include the GeoIP update stuff to do all of that for you.

[cryogenic666]

I'm also getting permission denied in the log files for ntopng for basically everything under /var/db/ntopng/#/rrd (substitute # with numbers 0-5). They're owned by ntopng/wheel, so I'm not sure what's going on, but it can't seem to update rrd files

[gpb]

I copied the GeoIP databases to the same folder as usual (manually, as below).  That seemed to work although the map view now is different and doesn't have the slick graphics it did before, feels like a step backward.  I need to read up on it...maybe I'm missing something.

Copy files to /usr/local/share/ntopng/httpdocs/geoip/

GeoLite2-ASN.mmdb
GeoLite2-Country.mmdb
GeoLite2-City.mmdb

chmod +x /usr/local/opnsense/scripts/OPNsense/Ntopng/generate_certs.php

Then restarting ntopng worked.  I didn't notice the log file issue.

[mimugmail]

I'm also getting permission denied in the log files for ntopng for basically everything under /var/db/ntopng/#/rrd (substitute # with numbers 0-5). They're owned by ntopng/wheel, so I'm not sure what's going on, but it can't seem to update rrd files

There are some strange files with only write permissions for the group. I will update the setup.sh script to change permissions recursive

[mimugmail]

I copied the GeoIP databases to the same folder as usual (manually, as below).  That seemed to work although the map view now is different and doesn't have the slick graphics it did before, feels like a step backward.  I need to read up on it...maybe I'm missing something.

Copy files to /usr/local/share/ntopng/httpdocs/geoip/

GeoLite2-ASN.mmdb
GeoLite2-Country.mmdb
GeoLite2-City.mmdb

chmod +x /usr/local/opnsense/scripts/OPNsense/Ntopng/generate_certs.php

Then restarting ntopng worked.  I didn't notice the log file issue.

Can you send me the exact command? Then I can try to automate it in setup script

[mimugmail]

Thanks Franco!  Just updated no issues, but am seeing higher than normal CPU that looks to be attributable to ntopng.  It varies between 50 and 100 (gui) vs normal idle of between 0 and 25.  Looking at it via "top" it looks very high (50 to 300%).  Not sure...maybe others can chime in...I've disabled it for now.  Looks like that's an updated package too, but no mention in the release notes (?).  Again, thanks and stay healthy!

Last quote for the day, I can also reproduce the CPU spikes. I'm already in touch with ntopng team, we will work on this.

If your system is unusable you can always revert to old version via CLI:

opnsense-revert -r 20.1.3 ntopng

[gpb]

I copied the GeoIP databases to the same folder as usual (manually, as below).  That seemed to work although the map view now is different and doesn't have the slick graphics it did before, feels like a step backward.  I need to read up on it...maybe I'm missing something.

Copy files to /usr/local/share/ntopng/httpdocs/geoip/

GeoLite2-ASN.mmdb
GeoLite2-Country.mmdb
GeoLite2-City.mmdb

chmod +x /usr/local/opnsense/scripts/OPNsense/Ntopng/generate_certs.php

Then restarting ntopng worked.  I didn't notice the log file issue.

Can you send me the exact command? Then I can try to automate it in setup script

Thanks but not sure what you can change aside from the chmod command above.  Some background...whenever ntopng is updated (or maybe other times too) the files mentioned above get removed.  These are the geo IP data files which are not apparently distributed with the package (which makes sense since they are time sensitive and now require an account to access).  I just ftp them over to the folder mentioned above and execute the chmod command...these are details I found in a youtube video explaining how to enable geo-features in ntopng in OPNsense.  It worked so I just do this process when needed.  Thanks!

[niziak]

The same. Huge CPU usage after usage.
Package ntopng version 4.0.d20200326,1.

Lots of error in ntop log:

Code: [Select]


tail -f /var/db/ntopng/ntopng.log

21/Apr/2020 19:56:01 [minute.lua:25] [rrd.lua:413] ERROR: rrd_update_r() [/var/db/ntopng/0/rrd/FTP_CONTROL.rrd][1587498960:786] failed [opening '/var/db/ntopng/0/rrd/FTP_CONTROL.rrd': Permission denied]
21/Apr/2020 19:56:01 [minute.lua:25] [rrd.lua:413] ERROR: rrd_update_r() [/var/db/ntopng/0/rrd/MQTT.rrd][1587498960:2854] failed [opening '/var/db/ntopng/0/rrd/MQTT.rrd': Permission denied]
21/Apr/2020 19:56:01 [minute.lua:25] [rrd.lua:413] ERROR: rrd_update_r() [/var/db/ntopng/0/rrd/WindowsUpdate.rrd][1587498960:775] failed [opening '/var/db/ntopng/0/rrd/WindowsUpdate.rrd': Permission denied]
21/Apr/2020 19:56:01 [minute.lua:25] [rrd.lua:413] ERROR: rrd_update_r() [/var/db/ntopng/0/rrd/SSH.rrd][1587498960:219766] failed [opening '/var/db/ntopng/0/rrd/SSH.rrd': Permission denied]


Code: [Select]

ls -l /var/db/ntopng/0/rrd/       

-rw-------   1 ntopng  wheel    34640 Dec 10 01:58 AFP.rrd
-rw-------   1 ntopng  wheel    34640 Dec 10 01:58 AJP.rrd
----rw-rw-   1 ntopng  wheel    34640 Apr  2  2019 Apple.rrd
-rw-------   1 ntopng  wheel    34640 Apr 21 19:07 ApplePush.rrd
-rw-------   1 ntopng  wheel    34640 Feb 13 07:13 AppleStore.rrd
-rw-------   1 ntopng  wheel    34640 Apr 21 19:38 AppleiCloud.rrd
----rw-rw-   1 ntopng  wheel    34640 Apr  2  2019 AppleiTunes.rrd
-rw-------   1 ntopng  wheel    34640 Oct 24 06:27 BGP.rrd
----rw-rw-   1 ntopng  wheel    34640 Apr  2  2019 BJNP.rrd
-rw-------   1 ntopng  wheel    34640 Apr 21 19:07 BitTorrent.rrd
...

RW permission for ntopng is missing on some RRD files, but this not cause of CPU usage. After fixing permission problems, no error reported, but CPU usage stays on the same almost 100% level.


Removal of /var/db/ntopng and /var/db/redis doesn't help also.

Finally

Code: [Select]

opnsense-revert -r 20.1.3 ntopng helps

[michelvankessel]

I did a revert.. ntopng starts, but fails after a few minutes and I get the following in the logs

2020-04-25T18:22:12kernel: pid 29452 (ntopng), uid 288: exited on signal 1120
20-04-25T18:20:47kernel: -> pid: 29452 ppid: 1 p_pax:0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
2020-04-25T18:20:47kernel: [HBSD SEGVGUARD] [ntopng (29452)] Suspension expired.

running 20.1.5 and reverted using "opnsense-revert -r 20.1.3 ntopng"

any ideas?
many thanks for your help

[michelvankessel]

reinstalled ntopng and redis and now running 20.1.6.

but also with clean install and  "opnsense-revert -r 20.1.3 ntopng" I keep getting similar errors and the process ends

2020-05-09T20:44:12   kernel: pid 99966 (ntopng), uid 288: exited on signal 11
2020-05-09T20:40:31   kernel: -> pid: 99966 ppid: 1 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
2020-05-09T20:40:31   kernel: [HBSD SEGVGUARD] [ntopng (99966)] Suspension expired.

Does anyone know what I am doing wrong?

[gpb]

I wonder if it is related to a particular setting...reason I say that is I did find someone a couple years ago noting that disabling alerts in ntopng was causing a crash.  I would hope that has been long solved by now.  In opnsense I have "none" selected for interfaces (advanced mode) which I found a bit confusing (none seems to mean all or don't limit it to any).  I think those settings are all the defaults and haven't had a problem aside from that high CPU issue.

[patcsy88]

May I know if there is any progress on the high CPU usage front? Am on 20.1.6 and am observing the same with load hovering between 2.0 - 2.5

[miruoy]

Hi @all

Issue still present on 20.1.6 with ntop-ng 1.2 and redis 1.1

Not sure if this is related but syslog indicates

Code: [Select]

ntopng: [Utils.cpp:3351] WARNING: ntopng has not been compiled with libcap-dev
ntopng logs does not show any errors.

Is there already a ticket logged for this to the maintainer m.muenz or can we add him to the conversation?

[edit] Removed dev email

[mimugmail]

I'm here
The problem is ntop itself. Not related to the plugin or OPNsense.
Version 4 doesn't run really well on FreeBSD