Track historical usage per user

[fawzy46]

I use ntopng and i love it as it shows total bandwidth used, currently active downloader, along with some other great features.

But it do not save the data for previous / old records. I have enabled the cache and changed the time to different time / days. But only thing it shows is active devices. If a device goes offline or firewall reboot then all the data is gone, deleted / removed by software.

I need to provide the daily report of total bandwidth used by the devices in network. I used bandwidthd before in PFSense and i know it will solve this issue.

Still if you know any way around in ntogng to save data and to get past records for report on daily basis then i am good to go with ntopng.

i have the same issue

[mimugmail]

For daily or monthly stats vnstat-plugin?

[c00lfloyd]

hey guys, i just stumbled on this thread while having the same problem, wanting to use bandwidthd. and also followed them directions by 'dedman'. and also had it not work...  BUT...

I figured it out! and got it working just fine, so if anyone still cares about doing this, here it is..

what needs changed is step #3, he had:

ln -s /usr/local/bandwidthd/htdocs/ bandwidthd

it should be :

Code Select Expand


ln -s /usr/local/bandwidthd/htdocs/ /usr/local/www/bandwidthd

-----
you can see why it wouldn't work. it was running and making stats but putting the html output in the ./bandwidthd folder off whatever folder you were in at the time you ran the command...

so there it is.,

Also i figured out how to autostart the bandwidthd service at startup.

Add a file here:

Code Select Expand


/usr/local/etc/rc.syshook.d/start


name it something like '91-bwdstartup'
then put these contents in that file you just created:

Code Select Expand


#!/bin/sh

echo -n "Starting BandwidthD..."

/usr/local/etc/rc.d/bandwidthd onestart


-------------



welp hope this helps someone....

good luck!

-R

[mimugmail]

Is bandwidthd really still a thing? Wasnt updated since when? 2016?

[sclawrenc]

Is bandwidthd still the best option for tracking data usage per device per month?  I've seen vnstat metnioned too, but I am not sure if it provides usage per device.  I have consistently been going over the Xfinity 1.2 TB data cap, and I would like to have a better idea of the main culprits. Thanks.

[rman50]

I run ntopng and store my timeseries data in InfluxDB for easy reporting/graphing in Grafana for client bandwidth usage statistics as well as application/category statistics. I only store the data for 2 weeks but could easily keep it for 30+ days if needed. Attached is a screenshot from my Grafana dashboard for the last 7 days for my ntopng data.

I also send OPNsense metrics to InfluxDB via Telegraf and I use Packetbeats to monitor DNS traffic. It makes for a pretty comprehensive dashboard for my home network. I run this all on a Raspberry Pi 4 with a mirror port since I didn't want to interfere with the firewall except for the Telegraf agent. I am assuming it could all be run the firewall itself as well using mimugmail's repository for InfluxDB & Grafana.

[mimugmail]

Grafana Dashboard please :)

[rman50]

Attached is my Grafana dashboard for the OPNsense Telegraf metrics and ntopng timeseries. It has two InfluxDB datasource variables to point to for OPNsense telegraf and ntopng databases. I stripped out all panels that require the custom Telegraf things I do for DHCP and DNS/Packetbeats.

For ntopng, I have the following timeseries settings:

• Resolution set to 1 minute
• Interface Layer-7 applications set to Both
• Host Timeseries set to Full
• Host Layer-7 applications set to Both

For the OPNsense CPU temperature, I manually added the following to the telegraf script:
[[inputs.exec]]
  commands = ["/sbin/sysctl dev.cpu | grep temperature | sed 's/[a-z\.]*/systemp cpu/;s/\.[a-z]*\: /=/;s/.$//'"]
  timeout = "5s"
  data_format = "influx"

[mimugmail]

Thx for sharing, I will have a look (with Plugins-only :) )

[sclawrenc]

Thanks rman50 and mimugmail!  This seems a bit over my head at the moment.  I do have a HP 730t for Opensense, and I have a couple of Raspberry Pi 4's so I could attempt this since it seems pretty cool to have that level of visibility into usage.

I'm currently on the Xfinity XB7 in router mode since the unlimited data plan is $25 with their gateway and $30 if I use my own modem, but I plan on spinning up OPNsense and putting the XB7 in bridge mode which I've done many times before without issue. 

So I guess bandwidthd is the best option outside of using ntopng piped to a Raspberry Pi?

[sclawrenc]

I am running the latest OPNsense, and I setup bandwidthd based on the previous comments in this topic.  I will give this a try for a while to see if it provides the information I'm looking for, but on first glance it looks good.  Thanks again!

[sclawrenc]

I have been using bandwidthd for the last couple days, and it does great on ipv4, but it completely misses all of my ipv6 traffic.  I'm assuming I will need to use ntopng to capture both ipv4 and ipv6.

I'm going to look into ntopng tutorials next.  Would ya'll recommend I store the ntopng data on the OPNsense firewall since I only plan on using it for a couple of months or should I send it to a raspberry pi 4?  I have over 200GB storage and 8GB of memory on the firewall (HP 730t) so I could probably use that if I wanted, but I also don't want to risk slowing down my 1200-1400 mbps connection.  I'm not running any IDS/IPS.

[rman50]

ntopng does support both ipv4 and ipv6. influxdb is very efficient at storing timeseries data since that is all it does. Also, ntopng provides several options for what data to retain and how often to write it out (every minute or every 5 minutes). Other than the timeseries data, ntopng maintains everything else in memory. I just reset my environment but I was using almost no disk space to store 2 weeks worth of data and I enable most of the timeseries details. I doubt you would have any issue running ntopng, influxdb & grafana (if you want your own custom dashboards which I recommend) on your firewall with those specs.

[sclawrenc]

Thanks so much rman50!! 

Hopefully, I will have some time over the weekend to work on getting this setup on my HP 730t running OPNsense.  Am I understanding correctly that the json file you provided I could use with my grafana installation?

I did notice that there is an option for enabling nProbe in ntopng.  Do you think this is something that I look into or just leave it disabled?  It sounds interesting from a data collection standpoint, but I don't think I would use the nDPI function.

I'm also curious in how you are sending OPNsense metrics to influxdb via telegraph.  I'm going to look into that too. 

Thanks again!  I'm really glad that I decided to go with OPNsense based on the friendly and knowledgeable people like you in this forum and the options that OPNsense provides.  :)

[rman50]

I don't use nProbe. It isn't required for the client statistics. Also I think it requires a license.

You should be able to import my dashboard json into grafana and then customize it as needed.

There is a telegraf plugin for opnsense which makes it simple to send the firewall metrics to influx.