OPNsense
  • Home
  • Help
  • Search
  • Login
  • Register

  • OPNsense Forum »
  • Archive »
  • 16.7 Legacy Series »
  • [SOLVED] Netflow Insight No Data Available
« previous next »
  • Print
Pages: 1 [2] 3

Author Topic: [SOLVED] Netflow Insight No Data Available  (Read 55811 times)

franco

  • Administrator
  • Hero Member
  • *****
  • Posts: 15133
  • Karma: 1309
    • View Profile
Re: Netflow Insight No Data Available
« Reply #15 on: September 01, 2016, 03:21:44 pm »
Hmm, the flowd* dump files hold all the traffic data. Removing the file would clear any chance of restoring that data, but likely brings back the graphs for any new fresh data. This more or less equals a full reset that you can do from the Reporting: Settings page if one isn't attached to the historic data.

For permanent storage of NetFlow data we strongly recommend sending it to an external server for collection.
Logged

Zapp

  • Newbie
  • *
  • Posts: 36
  • Karma: 1
    • View Profile
Re: Netflow Insight No Data Available
« Reply #16 on: September 01, 2016, 03:33:35 pm »
Hi! Me again.  ;D

Rebuilding of DB is still running and data for today are starting to show up in the Details tab.

Still no graphs for Last 2 hours, 30 second average but on some of the drop-downs I get graphs. Historical graphs still is missing also.

   /Jonas...
Logged

phoenix

  • Hero Member
  • *****
  • Posts: 520
  • Karma: 58
    • View Profile
Re: Netflow Insight No Data Available
« Reply #17 on: September 01, 2016, 03:53:48 pm »
Thanks Franco

I'd missed that option on the settings page, I'll clear everything and, as I said, I use other tools to monitor my LAN so the historical data isn't that important to me - I'll see how that goes. :)
Logged
Regards


Bill

franco

  • Administrator
  • Hero Member
  • *****
  • Posts: 15133
  • Karma: 1309
    • View Profile
Re: Netflow Insight No Data Available
« Reply #18 on: September 01, 2016, 03:58:38 pm »
Bill, it's new since 16.7 and not completely obvious. It's our fault really.

There may be something to read through the flowd file and partially restore it if it was damaged:

# flowd-reader /var/log/flowd.log

Jonas, do you get any error readings on that?

Best to keep the indexing running for now.
Logged

Zapp

  • Newbie
  • *
  • Posts: 36
  • Karma: 1
    • View Profile
Re: Netflow Insight No Data Available
« Reply #19 on: September 01, 2016, 04:02:30 pm »
Quote from: franco on September 01, 2016, 03:58:38 pm
Bill, it's new since 16.7 and not completely obvious. It's our fault really.

There may be something to read through the flowd file and partially restore it if it was damaged:

# flowd-reader /var/log/flowd.log

Jonas, do you get any error readings on that?

Best to keep the indexing running for now.

Update: I have graphs for Last 2 hours now! Historical graphs start to show up also. WIll probably take some time to rebuild everything. I'll just be patient here...

Thanks so far.

   /Jonas...
Logged

Zapp

  • Newbie
  • *
  • Posts: 36
  • Karma: 1
    • View Profile
Re: Netflow Insight No Data Available
« Reply #20 on: September 01, 2016, 04:06:00 pm »
Ahh...

No errors with the following command, just takes very long to process...

Code: [Select]
# flowd-reader /var/log/flowd.log

Exits like this.
Code: [Select]
...
FLOW recv_time 2016-09-01T16:02:42.659872 proto 17 tcpflags 00 tos 00 agent [127.0.0.1] src [10.42.50.254]:27402 dst [10.42.50.11]:53 packets 1 octets 87
FLOW recv_time 2016-09-01T16:02:42.659872 proto 17 tcpflags 00 tos 00 agent [127.0.0.1] src [10.42.50.254]:41226 dst [10.42.50.10]:53 packets 1 octets 75
FLOW recv_time 2016-09-01T16:02:42.659872 proto 17 tcpflags 00 tos 00 agent [127.0.0.1] src [10.42.50.254]:41226 dst [10.42.50.11]:53 packets 1 octets 75
root@OPNsense:~ #
Logged

franco

  • Administrator
  • Hero Member
  • *****
  • Posts: 15133
  • Karma: 1309
    • View Profile
Re: Netflow Insight No Data Available
« Reply #21 on: September 01, 2016, 04:07:27 pm »
Very odd. :(
Logged

Zapp

  • Newbie
  • *
  • Posts: 36
  • Karma: 1
    • View Profile
Re: Netflow Insight No Data Available
« Reply #22 on: September 01, 2016, 04:10:11 pm »
Some pictures on how it looks right now.

   /Jonas...
Logged

phoenix

  • Hero Member
  • *****
  • Posts: 520
  • Karma: 58
    • View Profile
Re: Netflow Insight No Data Available
« Reply #23 on: September 01, 2016, 04:17:16 pm »
Franco

Thanks for that tip on the settings page, I reset everything and it's now showing graphs for the last thirty minutes. :)
Logged
Regards


Bill

franco

  • Administrator
  • Hero Member
  • *****
  • Posts: 15133
  • Karma: 1309
    • View Profile
Re: Netflow Insight No Data Available
« Reply #24 on: September 02, 2016, 08:58:02 am »
Hi Jonas,

The graphs look ok to me. There are data points for all of the 2h interval and your data collection started on August 31?


Cheers,
Franco
Logged

Zapp

  • Newbie
  • *
  • Posts: 36
  • Karma: 1
    • View Profile
Re: Netflow Insight No Data Available
« Reply #25 on: September 02, 2016, 09:51:29 am »
Quote from: franco on September 02, 2016, 08:58:02 am
...
The graphs look ok to me. There are data points for all of the 2h interval and your data collection started on August 31?
...

Hmm, yes. Looks like I lost all old data in this process for the Insight graphs. The DB rebuild looks like it's finished so I guess I have to live with that unless there is some way of recreation it. Looking at the timestamps in the flowd logfiles i guess it's not possible.
All new data shows up in the graphs as expected.

   /Jonas...
Logged

franco

  • Administrator
  • Hero Member
  • *****
  • Posts: 15133
  • Karma: 1309
    • View Profile
Re: Netflow Insight No Data Available
« Reply #26 on: September 02, 2016, 10:22:29 am »
What "all old data" are we talking about in time ranges? Your oldest entry on the disk was:

-rw-------  1 root  wheel    11M Aug 31 02:22 /var/log/flowd.log.000010

As we don't touch the flowd log files at all, this brings me to the question: are you using Nano? Output of the following would shed more light on it:

# df -h


Cheers,
Franco
Logged

Zapp

  • Newbie
  • *
  • Posts: 36
  • Karma: 1
    • View Profile
Re: Netflow Insight No Data Available
« Reply #27 on: September 02, 2016, 12:41:36 pm »
I'm a bit puzzled myself here. I'll backtrack my steps to see if we have removed anything there.

This is the output.
Code: [Select]
root@OPNsense:~ # df -h
Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/ada0s1a     77G    1.7G     69G     2%    /
devfs           1.0K    1.0K      0B   100%    /dev
devfs           1.0K    1.0K      0B   100%    /var/dhcpd/dev
root@OPNsense:~ #
Logged

Zapp

  • Newbie
  • *
  • Posts: 36
  • Karma: 1
    • View Profile
Re: Netflow Insight No Data Available
« Reply #28 on: September 02, 2016, 12:47:35 pm »
I've been up for about 40 days with this install and I might not have enabled NetFlow from day one but it was certainly before 31/8.
Somehow the flowd logs have been removed. It might even have been me that have made that happen but I don't know how. Is it possible to make it happen from the GUI? Like when you press Apply in the NetFlow Capture window again?
I know that I have not removed any files in the console (SSH), that's for sure.

   /Jonas...
Logged

Zapp

  • Newbie
  • *
  • Posts: 36
  • Karma: 1
    • View Profile
Re: Netflow Insight No Data Available
« Reply #29 on: September 02, 2016, 01:09:14 pm »
Just a thought...

To me this looks like a logrotated set up. If it is then I would understand that my old data is gone since I have ditched the SQL DB that hold that data. (To solve the previous problems)

Code: [Select]
root@OPNsense:/var/log # ls -lah /var/log/flowd*
-rw-------  1 root  wheel   2.1M Sep  2 12:50 /var/log/flowd.log
-rw-------  1 root  wheel    11M Sep  2 12:21 /var/log/flowd.log.000001
-rw-------  1 root  wheel    11M Sep  2 10:25 /var/log/flowd.log.000002
-rw-------  1 root  wheel    11M Sep  2 08:12 /var/log/flowd.log.000003
-rw-------  1 root  wheel    11M Sep  2 05:58 /var/log/flowd.log.000004
-rw-------  1 root  wheel    11M Sep  2 03:29 /var/log/flowd.log.000005
-rw-------  1 root  wheel    11M Sep  2 00:58 /var/log/flowd.log.000006
-rw-------  1 root  wheel    11M Sep  1 22:46 /var/log/flowd.log.000007
-rw-------  1 root  wheel    11M Sep  1 20:17 /var/log/flowd.log.000008
-rw-------  1 root  wheel   107M Sep  1 17:56 /var/log/flowd.log.000009
-rw-------  1 root  wheel    11M Aug 31 20:39 /var/log/flowd.log.000010
root@OPNsense:/var/log #

   /Jonas...
Logged

  • Print
Pages: 1 [2] 3
« previous next »
  • OPNsense Forum »
  • Archive »
  • 16.7 Legacy Series »
  • [SOLVED] Netflow Insight No Data Available
 

OPNsense is an OSS project © Deciso B.V. 2015 - 2023 All rights reserved
  • SMF 2.0.19 | SMF © 2021, Simple Machines
    Privacy Policy
    | XHTML | RSS | WAP2