Generating a historical report on bandwidth speed

[kwo1]

Hi everyone,

I work for an organization which is paying $$$$ for their WAN internet connection.  It would be beneficial to generate a report to show the org what their actual usage is.  My needs:

• Peak usage, in megabits/sec
• Avg usage, preferably with an interval that can be defined, eg: avg usage over 24 hours, avg over 5 days, etc
• Be able to specify a start/end date for report to evaluate historical figures
• Identify the src or dest. hostname, IP, or OPNSense interface, to categorize and include/exclude them from the report

This report would help us determine if we're overutilizing/underutilizing the WAN connection, on what dates, by which devices, and their use case. 

I've evaluated the following that's native to OPNSense:
Reporting > Traffic - This shows data only in real time.  There's no reporting option available to look at historical metrics. 
Reporting > Insight - This covers most of my asks, but the biggest downside is that the .csv reports it generates does not show speed in terms of bytes, but in packets.  The 'Details' tab in the web GUI shows bytes, but it's bandwidth usage, not bandwidth speed. 

VnStat plugin - This shows bandwidth speed.  I can see HH/DD/MM/YY stats, which is awesome.  I can't specify exact dates/times, which is fine.  The drawback is it doesn't list the hostname/IP of that traffic, so I don't know what is consuming the data. 

I *think* there's no perfect option that meets my needs, and I'm left to just take all the various reports and try to generate my own report manually.  I hope I'm wrong and someone here can suggest something that can answer my needs?

Thank you very much for your time. 

[Patrick M. Hausen]

Peak, average, selection of time frame can all be done with an SNMP based network management system (NMS) like LibreNMS or Observium. I prefer Observium.

You need a separate machine or VM inside your network to run this software.

They (Observium) have complete guides for installation on Debian/Ubuntu and I wrote an equivalent complete guide for installation on FreeBSD.

The last point in your requirements list needs some bigger guns, e.g. ElastiFlow. Resource requirements are way heavier than for an NMS but you get more detailed traffic/volume reports so pick your poison. Heavier means 16 GB of memory and according to their docs 8 cores for the VM. Plus a couple of hundred of GB of virtual disk drive. I run it smoothly with 16 GB (no way around that, because "Elasticsearch") and 4 cores plus 300 GB of virtual disk.

They have a free "basic" license for up to 4000 flows (think "connections") per second - which is quite a bit so it fits probably every home lab and many smaller companies.

They have a nice quickstart guide for Ubuntu.

You might even want to deploy both, because an NMS not only gives you traffic measurements but also hardware stats like CPU, memory, disk, temperatures etc.

HTH,
Patrick

[kwo1]

Hi Patrick, thanks for your response. 

Per your suggestion, I looked at both LibreNMS and Observium.  I appreciate how they both link to a live demo system for me to evaluate their software in action.  However, I don't see any report on min/max/avg. bandwidth speeds.  The closest I could find is the transfer graph in Observium which shows total bandwidth transferred, but no mention of speed.  In other words, I can see that on XYZ day, 5.3GB was transferred, but I don't know how fast it was, the peak, or avg, in Mbps.

Maybe this is due to my unfamiliarity with using either app?  I'm reluctant to invest the time and money to install a VM and learn the apps only to find out it still doesn't do what I need.

I found bandwidthd which may have what I need (minus the ability to select a start/end time to generate a report): https://bandwidthd.sourceforge.net/demo/.

Thank you again.

[Patrick M. Hausen]

Here is the traffic graph from Observium for my VLAN 1.

You can see over a period of 24 hours:

- maximum in: 190 Mbit/s
- maximum out: 280 Mbit/s
- 95 percentile in: 32 Mbit/s
- 95 percentile out: 12 Mbit/s

and in the top row you can specify arbitrary time frames for which you need these values.

This is possible for all interfaces.

That's why I said an SNMP based NMS can deliver what you asked for per interface - but not per client or server system.

Kind regards,
Patrick

[kwo1]

Hi, is that feature part of the paid subscription of Observium?  Thank you

[Patrick M. Hausen]

No - that's a standard feature.

In the demo e.g.

- pick "Devices > All Devices" from the top menu
- pick "omega.observium.org"
- pick "Ports > Details" from the device's menu
- click on e.g. "vnet0"
- in the "Traffic" line of graphs click on the first one to zoom in

Almost everything is "clickable" in Observium for further exploration/drill-down.

Also in the dashboard you are shown after the login to the demo installation: click on any of the traffic graphs to arrive exactly at that location for the particular device and port. You can create a personal dashboard with all the ports you are specifically interested in.

HTH,
Patrick

EDIT: 95 percentile seems to be a paid feature. If that is important for your business, the professional edition is only 240€ per year. And that offers even more accounting features like automated billing. But the peak bandwidth per interface and time period is available in the CE.

[kwo1]

I installed Observium and found the graph as you described.  I'm performing speedtests from a computer but Observium's graph do not reflect the same figures.

Here's my data path:
Workstation (LAN) using OPNSense as it's default gateway.
OPNsense using it's WAN interface to go to the internet.

On my workstation, I'm performing a Google speedtest which is reporting 765 Mbps download and 207 Mbps upload.

My expectation is that when I look at the traffic graph in Observium corresponding to the WAN port of the OPNSense firewall, I should see similar corresponding figures, since traffic has to leave the WAN port of OPNSense to actually perform the speedtest.  I should see a max of around 700 Mbps in, and a 200 Mbps out.  Instead, the graph shows only a max of 43 Mbps in, and 31 Mbps out.

[Patrick M. Hausen]

Granularity is 5 minutes average. Run your speed test for 15 minutes and you will see.

[Patrick M. Hausen]

I just enabled the billing module in my subscription version and pretended to be billing for all Internet traffic by measuring pppoe0 on OPNsense. Looks good. Of course there is no historical data, yet.