Network Insight - Further additions

[pricklydevil]

Heya guys,

I originally raised the request for this in the 16.1 forum (https://forum.opnsense.org/index.php?topic=2211.msg7139#msg7139 and I've got to admit what you guys have created so far is absolutely amazing!

However, I think we're a little bit short of the original request. Whilst we can now see all the traffic from specific addresses on the network as well as breaking it down into individual protocols, the missing part is combining that into easily readable information, such as what Trav1sty suggested, in that it would be easy to see a total of what has been pushed/pulled per interface.

Unless I'm using Insight wrong (and I admit, I'm a bit of a newbie at it) I cannot seem to find just the basic information saying that say eth0 has pushed Xgb of data and received Xgb of data over a 1/7/30 day period, such as the following:



Would that be something that could be included under the Totals tab as a separate table? I really don't want to use a 3rd party product to do all of this via SNMP as it seems a waste to not have this in the core router functionality.

I guess from there things like maximum, minimum and average speeds can also be added as separate columns.

Happy to get involved and add more flesh around it, just can't do any of the actual coding

Cheers
Dean

[AdSchellevis]

Hi Dean,

Thank you very much for your positive feedback!

Reporting was a very often requested feature from different users and customers from the beginning of our project, we just didn't want to use the standard darkstat or ntop because it wouldn't integrate very nicely with the rest of our system.
The netflow / flowd + own ui has multiple advantages, like being able to use the same datastream for your enterprise netflow solution as our ui uses.

I'm not sure what extra functionality we're going to put in at the moment, we already invested a huge amount of time building this component.

Let me first try to explain what is already in, and the parts from your message which will most likely make it for 16.7.

1) Accumulated totals per port / address (the two pie charts), these represent the total data processed (in+out) for the selected time period at the top left corner of the screen and the selected interface just above the charts.

** we probably should add totals (in/out) beneath the charts like you suggested. (likely in 16.7)
   (Our system collects and aggregates data instantly, totals per interface are already stored.)

** I think we also should add some additional timespans in the selection in the top right corner

-- More charts for different timespans at once are not in our plans at the moment

2) Details per day for different selections, if you click on an address or port (pie piece) you will automatically be directed to the next tab which holds the totals for your selection.
   A top X is displayed, the rest will be captured in "(other)"
   
3) API support to collect data yourself
   Like all components we develop, its pretty easy to use them from your own applications using our API system, easiest way to learn the capabilities is to look at our template code which calls the api.
   
4) More detailed data export
   All data that is collected and stored in the different containers, can be extracted in csv format using the last tab "Export", using this data and excel you can build any presentation you like.
   The resolutions in which the data is stored (in seconds) for the different types will be automatically visible.
   

-- Maximums, minimums and averages are not in our scope at the moment, max/min/avg depends on the timespan you inspect and will probably only hold value if you save detailed data over a longer period of time. For interfaces it may be partly available, but going deeper into your data it kind of explodes.

Thank you very much for taking the time to test and provide feedback,

Best regards,

Ad

[pricklydevil]

Hi Ad,

Thanks for coming back to me so quickly

Wholly agree that the standard darkstat/ntop/pfInfo/vnStat certainly wouldn't match either the developers expectations but also the end users who appreciate a finer quality product than just attaching bolt-ons that doesn't match what we've all come to love about OPNSense.

The totals are probably the more important feature, especially with people that have limited connections etc. I'm lucky in that I don't need that, so it's more being nosey as to how much data is being pushed/pulled. The table as you suggested is best as shouldn't require "intensive" development, outside of the already extensive work that you guys have actually done .. which I can only imagine was no mean feat! The expanded time frames of 1/7/14/30/60/90/182/365 would be fantastic additions and gives granularity similar to other products.

The Max/Min/Average was more a throwout there comment but appreciate that it would require a lot of data storage, probably more than most people actually want to dedicate to that cause and would use something else for such things rather than the router .. keeping with the small but powerful motif.

Cheers
Dean

[franco]

Hi guys,

I really like what Ad has delivered here. The framework is flexible and has more under the hood than it shows. That makes it easy to shape UX and add further report details until we hit the official 16.7 mark with welcome feedback such as this.

I agree that there are some totals missing, spread out to total in / out per interface period.

Min/max/avg. can probably go to the totals at least too, that's just a bit of number magic with the present values.


Cheers,
Franco

[AdSchellevis]

Hi guys,

The totals and additional time selections are in the development version, for easy testing (when running OPNsense-dev), just execute the following in a shell:

Code: [Select]

curl -o /usr/local/opnsense/mvc/app/views/OPNsense/Diagnostics/networkinsight.volt https://raw.githubusercontent.com/opnsense/core/master/src/opnsense/mvc/app/views/OPNsense/Diagnostics/networkinsight.volt
Totals are shown for the same time/interface selection as the pie charts above it.

Best regards,

Ad

[pricklydevil]

Hi Ad,

Many many thanks for getting onto this and apologies for the slow feedback.

Works absolutely perfectly!

However, I have identified what could be considered a bug but maybe more because of my lack of data in Insight. Anything over 30 days, 24 hour average does not update the running totals, but the graphs change. I'll keep playing with it though as the more data the better

Can I also just check something .. where it says 24 hour average do this apply to the graphs or is that a true total? It could be slightly misread or me attempting to read when I haven't had enough sleep!

Many thanks
Dean

[AdSchellevis]

Hi Dean,

No problem, we're all busy 

The graphs change when there is no data, because the first date is plotted as 0 if not available (not really a bug).
About the averages, this is actually the sample resolution, so 1 running total per day. The term averages might be a bit confusing sometimes.

Best regards,

Ad

[pricklydevil]

Cool, thanks Ad. That makes a lot more sense.

Now to be even more cheeky , would there be any consensus over doing historicals on this ... by that I mean, Jan was XXX, Feb XXX et al?

Admittedly that means a lot more work and also wouldn't fit in with the current layout (probably another tab on the top, next to export) would "feel" better as it's not something that would be used all the time.

Considering the workload already on 16.7, it could come as a further enhancement.

Many thanks
Dean

[AdSchellevis]

Hi Dean,

No concrete plans for historical data yet, other then the api support and export which are already in there 
Although I agree that if we add it... we should add another tab to keep the rest clean like it is now.

Thanks for your feedback,

Best regards,

Ad