Thanks Franco! Just updated no issues, but am seeing higher than normal CPU that looks to be attributable to ntopng. It varies between 50 and 100 (gui) vs normal idle of between 0 and 25. Looking at it via "top" it looks very high (50 to 300%). Not sure...maybe others can chime in...I've disabled it for now. Looks like that's an updated package too, but no mention in the release notes (?). Again, thanks and stay healthy!
It's the new 4.0 version of NtopNG, actually, and I'm seeing the same thing myself. It also doesn't use the same GeoIP databases and doesn't include the GeoIP update stuff to do all of that for you.
I'm also getting permission denied in the log files for ntopng for basically everything under /var/db/ntopng/#/rrd (substitute # with numbers 0-5). They're owned by ntopng/wheel, so I'm not sure what's going on, but it can't seem to update rrd files
I copied the GeoIP databases to the same folder as usual (manually, as below). That seemed to work although the map view now is different and doesn't have the slick graphics it did before, feels like a step backward. I need to read up on it...maybe I'm missing something.
Copy files to /usr/local/share/ntopng/httpdocs/geoip/
GeoLite2-ASN.mmdb
GeoLite2-Country.mmdb
GeoLite2-City.mmdb
chmod +x /usr/local/opnsense/scripts/OPNsense/Ntopng/generate_certs.php
Then restarting ntopng worked. I didn't notice the log file issue.
Quote from: cryogenic666 on April 09, 2020, 02:11:29 AM
I'm also getting permission denied in the log files for ntopng for basically everything under /var/db/ntopng/#/rrd (substitute # with numbers 0-5). They're owned by ntopng/wheel, so I'm not sure what's going on, but it can't seem to update rrd files
There are some strange files with only write permissions for the group. I will update the setup.sh script to change permissions recursive
Quote from: gpb on April 09, 2020, 03:01:50 AM
I copied the GeoIP databases to the same folder as usual (manually, as below). That seemed to work although the map view now is different and doesn't have the slick graphics it did before, feels like a step backward. I need to read up on it...maybe I'm missing something.
Copy files to /usr/local/share/ntopng/httpdocs/geoip/
GeoLite2-ASN.mmdb
GeoLite2-Country.mmdb
GeoLite2-City.mmdb
chmod +x /usr/local/opnsense/scripts/OPNsense/Ntopng/generate_certs.php
Then restarting ntopng worked. I didn't notice the log file issue.
Can you send me the exact command? Then I can try to automate it in setup script
Quote from: gpb on April 08, 2020, 11:25:48 PM
Thanks Franco! Just updated no issues, but am seeing higher than normal CPU that looks to be attributable to ntopng. It varies between 50 and 100 (gui) vs normal idle of between 0 and 25. Looking at it via "top" it looks very high (50 to 300%). Not sure...maybe others can chime in...I've disabled it for now. Looks like that's an updated package too, but no mention in the release notes (?). Again, thanks and stay healthy!
Last quote for the day, I can also reproduce the CPU spikes. I'm already in touch with ntopng team, we will work on this.
If your system is unusable you can always revert to old version via CLI:
opnsense-revert -r 20.1.3 ntopng
Quote from: mimugmail on April 09, 2020, 07:59:05 AM
Quote from: gpb on April 09, 2020, 03:01:50 AM
I copied the GeoIP databases to the same folder as usual (manually, as below). That seemed to work although the map view now is different and doesn't have the slick graphics it did before, feels like a step backward. I need to read up on it...maybe I'm missing something.
Copy files to /usr/local/share/ntopng/httpdocs/geoip/
GeoLite2-ASN.mmdb
GeoLite2-Country.mmdb
GeoLite2-City.mmdb
chmod +x /usr/local/opnsense/scripts/OPNsense/Ntopng/generate_certs.php
Then restarting ntopng worked. I didn't notice the log file issue.
Can you send me the exact command? Then I can try to automate it in setup script
Thanks but not sure what you can change aside from the chmod command above. Some background...whenever ntopng is updated (or maybe other times too) the files mentioned above get removed. These are the geo IP data files which are not apparently distributed with the package (which makes sense since they are time sensitive and now require an account to access). I just ftp them over to the folder mentioned above and execute the chmod command...these are details I found in a youtube video explaining how to enable geo-features in ntopng in OPNsense. It worked so I just do this process when needed. Thanks!
The same. Huge CPU usage after usage.
Package ntopng version 4.0.d20200326,1.
Lots of error in ntop log:
tail -f /var/db/ntopng/ntopng.log
21/Apr/2020 19:56:01 [minute.lua:25] [rrd.lua:413] ERROR: rrd_update_r() [/var/db/ntopng/0/rrd/FTP_CONTROL.rrd][1587498960:786] failed [opening '/var/db/ntopng/0/rrd/FTP_CONTROL.rrd': Permission denied]
21/Apr/2020 19:56:01 [minute.lua:25] [rrd.lua:413] ERROR: rrd_update_r() [/var/db/ntopng/0/rrd/MQTT.rrd][1587498960:2854] failed [opening '/var/db/ntopng/0/rrd/MQTT.rrd': Permission denied]
21/Apr/2020 19:56:01 [minute.lua:25] [rrd.lua:413] ERROR: rrd_update_r() [/var/db/ntopng/0/rrd/WindowsUpdate.rrd][1587498960:775] failed [opening '/var/db/ntopng/0/rrd/WindowsUpdate.rrd': Permission denied]
21/Apr/2020 19:56:01 [minute.lua:25] [rrd.lua:413] ERROR: rrd_update_r() [/var/db/ntopng/0/rrd/SSH.rrd][1587498960:219766] failed [opening '/var/db/ntopng/0/rrd/SSH.rrd': Permission denied]
ls -l /var/db/ntopng/0/rrd/
-rw------- 1 ntopng wheel 34640 Dec 10 01:58 AFP.rrd
-rw------- 1 ntopng wheel 34640 Dec 10 01:58 AJP.rrd
----rw-rw- 1 ntopng wheel 34640 Apr 2 2019 Apple.rrd
-rw------- 1 ntopng wheel 34640 Apr 21 19:07 ApplePush.rrd
-rw------- 1 ntopng wheel 34640 Feb 13 07:13 AppleStore.rrd
-rw------- 1 ntopng wheel 34640 Apr 21 19:38 AppleiCloud.rrd
----rw-rw- 1 ntopng wheel 34640 Apr 2 2019 AppleiTunes.rrd
-rw------- 1 ntopng wheel 34640 Oct 24 06:27 BGP.rrd
----rw-rw- 1 ntopng wheel 34640 Apr 2 2019 BJNP.rrd
-rw------- 1 ntopng wheel 34640 Apr 21 19:07 BitTorrent.rrd
...
RW permission for ntopng is missing on some RRD files, but this not cause of CPU usage. After fixing permission problems, no error reported, but CPU usage stays on the same almost 100% level.
Removal of /var/db/ntopng and /var/db/redis doesn't help also.
Finally opnsense-revert -r 20.1.3 ntopng helps :)
I did a revert.. ntopng starts, but fails after a few minutes and I get the following in the logs
2020-04-25T18:22:12kernel: pid 29452 (ntopng), uid 288: exited on signal 1120
20-04-25T18:20:47kernel: -> pid: 29452 ppid: 1 p_pax:0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
2020-04-25T18:20:47kernel: [HBSD SEGVGUARD] [ntopng (29452)] Suspension expired.
running 20.1.5 and reverted using "opnsense-revert -r 20.1.3 ntopng"
any ideas?
many thanks for your help
reinstalled ntopng and redis and now running 20.1.6.
but also with clean install and "opnsense-revert -r 20.1.3 ntopng" I keep getting similar errors and the process ends
2020-05-09T20:44:12 kernel: pid 99966 (ntopng), uid 288: exited on signal 11
2020-05-09T20:40:31 kernel: -> pid: 99966 ppid: 1 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
2020-05-09T20:40:31 kernel: [HBSD SEGVGUARD] [ntopng (99966)] Suspension expired.
Does anyone know what I am doing wrong?
I wonder if it is related to a particular setting...reason I say that is I did find someone a couple years ago noting that disabling alerts in ntopng was causing a crash. I would hope that has been long solved by now. In opnsense I have "none" selected for interfaces (advanced mode) which I found a bit confusing (none seems to mean all or don't limit it to any). I think those settings are all the defaults and haven't had a problem aside from that high CPU issue.
May I know if there is any progress on the high CPU usage front? Am on 20.1.6 and am observing the same with load hovering between 2.0 - 2.5
Hi @all
Issue still present on 20.1.6 with ntop-ng 1.2 and redis 1.1
Not sure if this is related but syslog indicates
ntopng: [Utils.cpp:3351] WARNING: ntopng has not been compiled with libcap-dev
ntopng logs does not show any errors.
Is there already a ticket logged for this to the maintainer m.muenz or can we add him to the conversation?
[edit] Removed dev email
I'm here :)
The problem is ntop itself. Not related to the plugin or OPNsense.
Version 4 doesn't run really well on FreeBSD
Quote from: mimugmail on May 15, 2020, 03:02:39 PM
I'm here :)
Ah good to know you're in the loop and already investigating :D
We will be patient until you provide more feedback. Let us know if we can do some additional tests to assist in solving this.
Is there a way to install ntopng 3.8 ? I believe this version was ok with freebsd
Quote from: andrema2 on June 02, 2020, 04:39:44 PM
Is there a way to install ntopng 3.8 ? I believe this version was ok with freebsd
Mimugmail has posted a temporary workaround in this thread to revert to the old version:
If your system is unusable you can always revert to old version via CLI:
opnsense-revert -r 20.1.3 ntopngYour mileage may vary though. On my end the old version on fails after about 5 minutes of runtime.
I'm already in touch with ntop devs, hopefully this gets fixed soon
Quote from: mimugmail on June 02, 2020, 09:24:57 PM
I'm already in touch with ntop devs, hopefully this gets fixed soon
awesome.. the brief few minutes i got to work to version 4, i really liked the new UI for it.. too bad the process was at 93% CPU utilization .. lol
cant' wait!
well.. unfortunately, after several attempts to get ntopng running again after upgrading I have not been able to get it running for longer than a few minutes.
i upgraded to 20.1 and then to 20.1.7 today... ntopng was running just fine previously.
i reverted (as indicated earlier in the thread) to 20.1.3 and i start to see the following logs:
2020-06-06T22:27:01 kernel: pflog0: promiscuous mode enabled
2020-06-06T22:27:01 kernel: pflog0: promiscuous mode disabled
2020-06-06T22:23:56 kernel: em0: promiscuous mode disabled
2020-06-06T22:23:56 kernel: pid 58039 (ntopng), uid 288: exited on signal 11
2020-06-06T22:23:42 kernel: -> pid: 58039 ppid: 1 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
2020-06-06T22:23:42 kernel: [HBSD SEGVGUARD] [ntopng (58039)] Suspension expired.
2020-06-06T22:23:41 ntopng: [Utils.cpp:3056] WARNING: Network discovery and other privileged activities will fail
2020-06-06T22:23:41 kernel: em0: promiscuous mode enabled
2020-06-06T22:23:41 ntopng: [Utils.cpp:3055] WARNING: ntopng has not been compiled with libcap-dev
2020-06-06T22:19:41 kernel: em0: promiscuous mode disabled
2020-06-06T22:19:41 kernel: pid 11183 (ntopng), uid 288: exited on signal 11
2020-06-06T22:19:38 ntopng: [Utils.cpp:3056] WARNING: Network discovery and other privileged activities will fail
2020-06-06T22:19:38 kernel: em0: promiscuous mode enabled
2020-06-06T22:19:38 ntopng: [Utils.cpp:3055] WARNING: ntopng has not been compiled with libcap-dev
i guess it will just remain disabled for a while
fyi - 20.1.7 same issue.
Upstream patch is available, couple of weeks to go
Quote from: mimugmail on June 15, 2020, 06:13:16 PM
Upstream patch is available, couple of weeks to go
Great news && thanks for the update! Let us know if you need testers.
Quote from: mimugmail on June 15, 2020, 06:13:16 PM
Upstream patch is available, couple of weeks to go
Thank you for the update!
I can confirm, after upgrade to 20.1.8 CPU load is way better.
Confirmed that the issue is fixed after upgrade to version OPNsense 20.1.8_1
how can i update ntopng ?
A new ntopng stable version (v.4.2.0) is available for download: please upgrade.
It has problems withing FreeBSD ... not yet.
Quote from: mimugmail on November 11, 2020, 11:04:24 AM
It has problems withing FreeBSD ... not yet.
Would be possible to have a plugin for nprobe so we could forward the information for an external ntopng ?
I think nprobe isnt Open source