OPNsense Forum

Archive => 15.7 Legacy Series => Topic started by: BertM on December 11, 2015, 05:36:16 pm

Title: Missing RRD Graphs
Post by: BertM on December 11, 2015, 05:36:16 pm
After updating to 15.7.22, the rrd graphs are no longer there.
Is this intentional, or just an unfortunate mishap and will they come back?

Kind regards,
Bert
Title: Re: Missing RRD Graphs
Post by: Andreas on December 11, 2015, 05:41:15 pm
Hi Bert
The Rrd graphs have been taken out from the system since the 15.7.21 release
OPNsense 15.7.21 released
https://forum.opnsense.org/index.php?topic=1795


andreas
Title: Re: Missing RRD Graphs
Post by: phoenix on December 11, 2015, 05:43:26 pm
As mentioned above, they're gone but they have been replaced by System Health. I'd suggest that monitoring your firewall should be the role of some other server on your LAN with something like LibreNMS. :)
Title: Re: Missing RRD Graphs
Post by: franco on December 11, 2015, 09:52:30 pm
The won't come back. I would like to focus on why they might still be needed and how we can improve System Heath to meet all the requirements.
Title: Re: Missing RRD Graphs
Post by: rost on December 21, 2015, 07:13:51 am
I am very upset that rrd is no more in this version.
There were normal tables and graphs that can be easily saved as a picture.
And now I have to look at here is!
Title: Re: Missing RRD Graphs
Post by: franco on December 22, 2015, 08:18:40 am
Hmm, that looks like a picture to me. But on a more serious note, what browser are you using? This doesn't look 100% correct...
Title: Re: Missing RRD Graphs
Post by: rost on December 22, 2015, 09:03:42 am
Chrome 47.0.2526.106
Lunux 64bit
How can I return rrd?

System Health to me is absolutely useless!
There are all indicators of traffic does not meet the reality.
Why is there megabytes, not megabits?
The rest of the graphics at all empty and I do not understand why they are needed.

Please return rrd
p.s. I was hoping to get statistics on the load channel for the month and the year. But no luck, you erased rrd!
Title: Re: Missing RRD Graphs
Post by: phoenix on December 22, 2015, 12:05:37 pm
As mentioned several times, the rrd graphs are not coming back and my opinion would be to use some other server on your system for a more complete monitoring system - is there any reason you don't want to do that?
Title: Re: Missing RRD Graphs
Post by: rost on December 22, 2015, 01:01:25 pm
I think is not appropriate to set a separate monitoring system in order to have statistics on the utilization of channels.
To put another OS in a small network? What for?
Where I have a large network, I use zabbix, but I have a small company with a small server, which is nothing other than to monitor "whether enough Internet channel and how the data are on the VPN"

Is RRD demanded a lot of resources?
Title: Re: Missing RRD Graphs
Post by: phoenix on December 22, 2015, 01:14:40 pm
I think is not appropriate to set a separate monitoring system in order to have statistics on the utilization of channels.
Why not, don't you monitor all the machines in your network?

To put another OS in a small network? What for?
As I've just mentioned, to monitor your complete network and you could always add monitoring software to an already existing server.

Where I have a large network, I use zabbix, but I have a small company with a small server, which is nothing other than to monitor "whether enough Internet channel and how the data are on the VPN"

Is RRD demanded a lot of resources?
Well, we could debate this point for ever but the position is quite clear and the rrd graphs are gone. Whatever you decide to do, it's your choice in the end. ;)
Title: Re: Missing RRD Graphs
Post by: franco on December 22, 2015, 02:02:17 pm
Okay, maybe some misconceptions here... We removed the RRD frontend graphics generation, because ever since we started people have said they don't like it for one reason or another. What we have done is replace the frontend only. The RRD data that is being collected is still there, and you can easily export it. There is no loss in data at all. It looks different any maybe a label is wrong. These things can be fixed.

We've also talked about static reports, PDF export and so on, but we have much larger dragons to slay with the GUI itself. Internal discussion on System Health went the way of "it's good that we did this, but it already cost a lot of time that could have been spent elsewhere".

The RRD frontend code can also be brought back as a plugin. I'll happily merge a pull request, but this won't be carried out by any of our project's resources.

So, which way do we want to go?
Title: Re: Missing RRD Graphs
Post by: franco on December 22, 2015, 02:12:01 pm
PS, can you humour me and use Firefox for the page view instead? It should really look like this... https://opnsense.org/system-health-whats-next/
Title: Re: Missing RRD Graphs
Post by: rost on December 22, 2015, 03:08:47 pm
Well, I try not to use the system-health in the chrome traffic for more than 60 hours.
Then there are still shortcomings (maybe it's not just me)
When I go to the tab "Traffic" is displayed by default ipsec.
I never set up ipsec and it is off in this system.
I turned to the setting is per IPv6, why the graphs displaying charts ipv6?

Title: Re: Missing RRD Graphs
Post by: franco on December 22, 2015, 03:16:12 pm
Welcome to the wonders of our code heritage. The backend always did this, the frontend simply hid it. We've cleaned up such things in the last 12 months and this is no exception. :)

The interface selection has a select bug and could reorder the entries, we can fix that.

I am guessing you mean you don't use IPv6? The data points are always there even if not enabled, why should we hide them? If not, please help me to understand better.
Title: Re: Missing RRD Graphs
Post by: rost on December 22, 2015, 03:28:40 pm
System: Settings: Networking
Allow IPv6  (box is unchecked)
All IPv6 traffic will be blocked by the firewall unless this box is checked.
NOTE: This does not disable any IPv6 features on the firewall, it only blocks traffic.

All IPv6 traffic will be blocked  I was thinking that why me these charts if they will always be zero.
Title: Re: Missing RRD Graphs
Post by: franco on December 22, 2015, 03:32:31 pm
The option does just that, block IPv6. And that only on non-local traffic. You will always have loopback traffic on IPv6 as long as the system is built as being IPv6 capable. Case in point, squid (proxy server) does this. So even when you disable IPv6, it'll still pass (and block) IPv6 traffic. Hope that helps.
Title: Re: Missing RRD Graphs
Post by: giovino on December 23, 2015, 01:44:28 pm
I too miss the RRD graphs and would enable them if that was an option. What do I miss?

1. It was very easy to see 5m, 1h, 1d, 30d, 365d graphs.

2. Traffic Graphs: I like the table stats below the graph (maximum, average, current, period, 95th percentile)
 * I realize you can enable "show tables" but you don't get the measurement per sec (MB/s), period or 95th percentile

Note: RRD graphs are not important enough for me to bring up on separate infrastructure in my home network.



Title: Re: Missing RRD Graphs
Post by: BertM on December 25, 2015, 12:05:30 am
The main reason why I liked the RRD Graphs so much is that it was so easy to look at them (traffic) to identify trends in bandwidth usage, or to look at quality to have a historical overview of the stability of the WAN connection, or to see a trend in blocked packets, etc.

And setting-up another machine, just to monitor? We already have capacity problems in our VMware environment, so that is not likely to happen.
But even if we did, it would mean putting time and effort in setting-up yet another system I am not familiar with (LibreNMS was suggested).
And for what?
To have an additional device to query while I am already looking at the firewall involved?

The convenient thing about the RRD Graphs was that they were always available on the same firewall you were already looking at. Having to go elsewhere just to have a look at this just does not make sense.
And it would also mean that our firewalls in branch offices such as in Japan, Russia, Italy, Spain, Germany, etc. would all have to report back to the server that is collecting the info.
In other words: I would have to setup an additional infrastructure just for the purpose of having a look at things that have always been there since the early days of pfSense and also in the first releases of OPNsense.

I am sad they are gone. I consider the system heath to be a bit clumsy. I don't think it is meanth for the average laptop screen. (1366x768)
With RRD Graphs, you just had to scroll up and down to see the graphs for the different periods in different resolutions.
With the System Healt page, I just have to click to select what I want to see, then scroll down to actually see it, and if I want to see another period or resolution, I have to scroll up again, make the changes in my selection and scroll down again to see the picture.

I initially hoped that the fact that they were missing in 15.7.22 was some mistake or error, and that they would return 15.7.23.
It is really sad to hear that such a useful functionality is gone, and will not return.

Title: Re: Missing RRD Graphs
Post by: AdSchellevis on December 25, 2015, 11:37:53 am
System health uses the exact same data as the old rrd graphs did, so technically the functionality is not gone only the presentation is  different.

That wouldn't mean we can't optimize the system health to address some of the current concerns, at least the screen size issue and the amount of scrolling could probably be improved a bit.

I will try to look into this.
Title: Re: Missing RRD Graphs
Post by: franco on December 25, 2015, 02:46:16 pm
Okay, so as a requirement I'm seeing "5m, 1h, 1d, 30d, 365d graphs" templates, a multi-graph view and a better screen utilisation. I think that can be arranged in a sort of "compatibility view mode", but I'm not exactly sure about the actual use case other than "better overview". It sounds like a dashboard functionality, not a drill-down page issue?
Title: Re: Missing RRD Graphs
Post by: BertM on January 06, 2016, 09:48:46 pm
Considering the fact that the roadmap reads for the 16.1 version: "Replace RRD frontend using a modern alternative", I have to assume this is it.
Maybe I am just an old fossil having problems getting used to "modern".

I know I can use the bar below the graph to define an area to zoom-in to, but not in one nice view.
The problem with that is that it does not show more detail, it just stretches the graph horizontally.
But if you want to look at details, even with the zoom level set to 20 hours and the resolution set to high, zooming-in using the bar below the graph gives the impression (I am not really sure) that there is less detail than with the old RRD graphs.

I guess I will just have to live with this.
For me personally, the actual functionality of OPNsense is more important than the graph.

I am much more eager to learn more about the upcoming IDS/IPS and, for the future maybe, something in the area of "parental control" that would allow me to prevent users from listening to internet radio or visiting sites like youtube over already congested internet links.

Let's not put to much energy in discussing something like graphs. They will always be subject to personal preferences.




Title: Re: Missing RRD Graphs
Post by: AdSchellevis on January 07, 2016, 09:43:59 am
We have done some work to improve the readability a bit, made all a bit less space consuming and provide the option to collapse the options.

Below some examples:

20 hour dataset (high resolution):
(https://forum.opnsense.org/images/systemhealth_20_hour.png)
1 year dataset (high resolution):
(https://forum.opnsense.org/images/systemhealth_1_year.png)

In case you want to try this on your machine before the release, just download this single template:
Code: [Select]
curl -o /usr/local/opnsense/mvc/app/views/OPNsense/Diagnostics/systemhealth.volt https://raw.githubusercontent.com/opnsense/core/master/src/opnsense/mvc/app/views/OPNsense/Diagnostics/systemhealth.volt


I did check the detail level on my end as well, but the level of detail really is the same as in the raw data from the rrd data itself (which hasn't changed) for the time period selected as long as it fits the screen.
(high res for the full 20 hours changes from 1 minute to 2 minutes, zooming results to 1 minute again).

The raw data can be exported using rrdtool (just for reference):

Code: [Select]
/usr/local/bin/rrdtool dump /var/db/rrd/


I totally agree, a lot of things are subjected to preference, although we can try to improve some of it to make it more acceptable for you too :)

Title: Re: Missing RRD Graphs
Post by: jstrebel on January 07, 2016, 10:12:40 am
@BertM, in regard of parental control and service filtering.
OPNsense is actually very good and efficient at this. There is a OpenDNS Client in it which allows you to determine which content and services you allow. Further on it acts also as a excellent phishing filter.
You will find this service entry under Services:DNS tools: Filter
Jakob
Title: Re: Missing RRD Graphs
Post by: BertM on January 07, 2016, 10:46:07 pm
@jstrebel:Good suggestion. I will certainly try to play around with OpenDNS.
On some locations in our network it will certainly be an option.

Unfortunately not for all because that will only work if the computers in your network use the OPNsense for DNS.

On locations where we have a Windows DC, DNS from Windows will be used, and that synchronizes with all other DNS servers in our worldwide spread Active Directory. If only one of the DNS servers in the AD knows about the sites you want to block, it will still be reachable.
 :'(
Title: Re: Missing RRD Graphs
Post by: jstrebel on January 08, 2016, 08:37:38 am
BertM, may be you could try to install the OpenDNS client in the win DC. So this win DC will get its "external" address resolutions from OpenDNS. (I do not have WIN AD know kow )


Gesendet von iPhone mit Tapatalk
Title: Re: Missing RRD Graphs
Post by: BertM on January 08, 2016, 08:03:12 pm
@jstrebel:
I'm not allowed to do that (corp. policy)
However, some time in Q3 this year I will be able to setup a "test lab", so by then I could start playing around with it in a test environment.
 
Title: Re: Missing RRD Graphs
Post by: franco on January 09, 2016, 01:00:04 am
Ad, I think the data load message is in the wrong place as it keeps "jumping" the graph since it's shown/hidden... it should be static or incorporated into the static header... :)
Title: Re: Missing RRD Graphs
Post by: AdSchellevis on January 09, 2016, 08:59:27 am
Maybe it's better to replace it with a single icon in the title and drop the text (<i class="fa fa-spinner fa-spin"></i>), what do you think?
Title: Re: Missing RRD Graphs
Post by: franco on January 09, 2016, 09:51:52 am
That sounds great. :)
Title: Re: Missing RRD Graphs
Post by: AdSchellevis on January 09, 2016, 06:06:34 pm
it looked a bit odd without the text, so I kept the text and moved all to the same panel (because of processing order, it doesn't have a title when performing the initial fetch). The jumping behaviour is fixed now.
Title: Re: Missing RRD Graphs
Post by: rost on February 20, 2016, 08:53:11 am
I'm sorry, but I could not get used to the "System health".
Return Please RRD.  :'(

When I opened the rrd, I immediately saw all the necessary information.
And now, every time I have to do 10 clicks to get at least some informative graph.   >:(
And why is everything in "System health" is measured in bytes, when the network has always been a bit?
Title: Re: Missing RRD Graphs
Post by: AdSchellevis on February 20, 2016, 01:20:30 pm
The old graphs aren't coming back, if there are solvable issues with the current implementation we will definitely look at it.

All data is presented in the form it is recorded into rrd, for example the traffic graph uses this command:
Code: [Select]
pfctl -vvsI -i em1
Which outputs bytes, and there for is presented in bytes. A good addition might be to add a multiplier for some of the graphs, sometimes indeed it is more logical to present bits.

If you really want to use the old pages, you could download them from an earlier distribution and maintain them yourself.
Code: [Select]
curl -o /usr/local/www/status_rrd_graph_tabs.inc https://raw.githubusercontent.com/opnsense/core/15.7.10/src/www/status_rrd_graph_tabs.inc
curl -o /usr/local/www/status_rrd_graph.php https://raw.githubusercontent.com/opnsense/core/15.7.10/src/www/status_rrd_graph.php
curl -o /usr/local/www/status_rrd_graph_img.php https://raw.githubusercontent.com/opnsense/core/15.7.10/src/www/status_rrd_graph_img.php

Title: Re: Missing RRD Graphs
Post by: franco on February 22, 2016, 07:29:24 am
If someone steps up to maintain a plugin for this we're happy to merge and publish it as such. It's not that this isn't trivially easy to bring back now, the problem is it will break later on and needs maintenance, which takes time for reaching roadmap goals away from the core team.

https://docs.opnsense.org/development/examples/helloworld.html

https://github.com/opnsense/plugins