[SOLVED] Netflow Insight No Data Available

[franco]

Hmm, the flowd* dump files hold all the traffic data. Removing the file would clear any chance of restoring that data, but likely brings back the graphs for any new fresh data. This more or less equals a full reset that you can do from the Reporting: Settings page if one isn't attached to the historic data.

For permanent storage of NetFlow data we strongly recommend sending it to an external server for collection.

[Zapp]

Hi! Me again. 

Rebuilding of DB is still running and data for today are starting to show up in the Details tab.

Still no graphs for Last 2 hours, 30 second average but on some of the drop-downs I get graphs. Historical graphs still is missing also.

   /Jonas...

[phoenix]

Thanks Franco

I'd missed that option on the settings page, I'll clear everything and, as I said, I use other tools to monitor my LAN so the historical data isn't that important to me - I'll see how that goes.

[franco]

Bill, it's new since 16.7 and not completely obvious. It's our fault really.

There may be something to read through the flowd file and partially restore it if it was damaged:

# flowd-reader /var/log/flowd.log

Jonas, do you get any error readings on that?

Best to keep the indexing running for now.

[Zapp]

Bill, it's new since 16.7 and not completely obvious. It's our fault really.

There may be something to read through the flowd file and partially restore it if it was damaged:

# flowd-reader /var/log/flowd.log

Jonas, do you get any error readings on that?

Best to keep the indexing running for now.

Update: I have graphs for Last 2 hours now! Historical graphs start to show up also. WIll probably take some time to rebuild everything. I'll just be patient here...

Thanks so far.

   /Jonas...

[Zapp]

Ahh...

No errors with the following command, just takes very long to process...

Code: [Select]

# flowd-reader /var/log/flowd.log

Exits like this.

Code: [Select]

...
FLOW recv_time 2016-09-01T16:02:42.659872 proto 17 tcpflags 00 tos 00 agent [127.0.0.1] src [10.42.50.254]:27402 dst [10.42.50.11]:53 packets 1 octets 87
FLOW recv_time 2016-09-01T16:02:42.659872 proto 17 tcpflags 00 tos 00 agent [127.0.0.1] src [10.42.50.254]:41226 dst [10.42.50.10]:53 packets 1 octets 75
FLOW recv_time 2016-09-01T16:02:42.659872 proto 17 tcpflags 00 tos 00 agent [127.0.0.1] src [10.42.50.254]:41226 dst [10.42.50.11]:53 packets 1 octets 75
root@OPNsense:~ #


[franco]

Very odd.

[Zapp]

Some pictures on how it looks right now.

   /Jonas...

[phoenix]

Franco

Thanks for that tip on the settings page, I reset everything and it's now showing graphs for the last thirty minutes.

[franco]

Hi Jonas,

The graphs look ok to me. There are data points for all of the 2h interval and your data collection started on August 31?


Cheers,
Franco

[Zapp]

...
The graphs look ok to me. There are data points for all of the 2h interval and your data collection started on August 31?
...

Hmm, yes. Looks like I lost all old data in this process for the Insight graphs. The DB rebuild looks like it's finished so I guess I have to live with that unless there is some way of recreation it. Looking at the timestamps in the flowd logfiles i guess it's not possible.
All new data shows up in the graphs as expected.

   /Jonas...

[franco]

What "all old data" are we talking about in time ranges? Your oldest entry on the disk was:

-rw-------  1 root  wheel    11M Aug 31 02:22 /var/log/flowd.log.000010

As we don't touch the flowd log files at all, this brings me to the question: are you using Nano? Output of the following would shed more light on it:

# df -h


Cheers,
Franco

[Zapp]

I'm a bit puzzled myself here. I'll backtrack my steps to see if we have removed anything there.

This is the output.

Code: [Select]

root@OPNsense:~ # df -h
Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/ada0s1a     77G    1.7G     69G     2%    /
devfs           1.0K    1.0K      0B   100%    /dev
devfs           1.0K    1.0K      0B   100%    /var/dhcpd/dev
root@OPNsense:~ #


[Zapp]

I've been up for about 40 days with this install and I might not have enabled NetFlow from day one but it was certainly before 31/8.
Somehow the flowd logs have been removed. It might even have been me that have made that happen but I don't know how. Is it possible to make it happen from the GUI? Like when you press Apply in the NetFlow Capture window again?
I know that I have not removed any files in the console (SSH), that's for sure.

   /Jonas...

[Zapp]

Just a thought...

To me this looks like a logrotated set up. If it is then I would understand that my old data is gone since I have ditched the SQL DB that hold that data. (To solve the previous problems)

Code: [Select]

root@OPNsense:/var/log # ls -lah /var/log/flowd*
-rw-------  1 root  wheel   2.1M Sep  2 12:50 /var/log/flowd.log
-rw-------  1 root  wheel    11M Sep  2 12:21 /var/log/flowd.log.000001
-rw-------  1 root  wheel    11M Sep  2 10:25 /var/log/flowd.log.000002
-rw-------  1 root  wheel    11M Sep  2 08:12 /var/log/flowd.log.000003
-rw-------  1 root  wheel    11M Sep  2 05:58 /var/log/flowd.log.000004
-rw-------  1 root  wheel    11M Sep  2 03:29 /var/log/flowd.log.000005
-rw-------  1 root  wheel    11M Sep  2 00:58 /var/log/flowd.log.000006
-rw-------  1 root  wheel    11M Sep  1 22:46 /var/log/flowd.log.000007
-rw-------  1 root  wheel    11M Sep  1 20:17 /var/log/flowd.log.000008
-rw-------  1 root  wheel   107M Sep  1 17:56 /var/log/flowd.log.000009
-rw-------  1 root  wheel    11M Aug 31 20:39 /var/log/flowd.log.000010
root@OPNsense:/var/log #

   /Jonas...