OPNsense Forum
Archive => 16.7 Legacy Series => Topic started by: adcwb on August 25, 2016, 06:09:54 pm
-
OPNsense was running ok until yesterday, as of today the Insight page just show "No Data Available".
The log shows the following:
flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 79, in aggregate_flowd stream_agg_object.add(flow_record_cpy) File "/usr/local/opnsense/scripts/netflow/lib/aggregates/source.py", line 105, in add super(FlowSourceAddrDetails, self).add(flow) File "/usr/local/opnsense/scripts/netflow/lib/aggregate.py", line 258, in add self._update_cur.execute(self._update_stmt, flow) DatabaseError: database disk image is malformed
In the Netflow -> Cache page i can see that it is tracking.
The flowd-reader command seems fine too, outputting the traffic.
Tried restarting the flowd_aggregator service, without success.
Any thoughts?
-
Hi there,
Which version are you using? What image flavour did you use?
We do have an auto-repair, but it's not in the release it. You can apply it on top of 16.7.2 using:
# opnsense-patch 5ec2101
The respective commit is:
https://github.com/opnsense/core/commit/5ec2101
If this works we can queue it up for 16.7.3 next week.
Cheers,
Franco
-
Bumping this.
As per my upgrade to 16.7.3 yesterday (31/8) I now have the No Data Available in my Insight display. 31/8 was ok and have data up until time of upgrade then nothing. 1/9 have NULL entries!
In NetFlow section Cache tab pressing refresh makes the numbers change. In my world that means something is happening but does not get reflected in Insight.
NetFlow Capture is on and Capture local checked, 127.0.0.1:2056 for Destination and v9 for Version.
I'd be happy to supply more information but right now I don't know what. :-\
/Jonas...
-
Hi Jonas,
You might want to try this patch we've given a longer test interval:
https://github.com/opnsense/core/commit/5ec2101ac2
You can install and activate it from the command like this
# opnsense-patch 5ec2101ac2
# service flowd_aggregate restart
The graphs should come back to life immediately.
I haven't seen any glitches there in the last month (I'm using the patched opnsense-devel version).
Cheers,
Franco
-
...
You can install and activate it from the command like this
# opnsense-patch 5ec2101ac2
# service flowd_aggregate restart
The graphs should come back to life immediately.
...
Sorry, no joy!
Edit /etc/motd to change this login announcement.
0) Logout 7) Ping host
1) Assign Interfaces 8) Shell
2) Set interface(s) IP address 9) pfTop
3) Reset the root password 10) Filter Logs
4) Reset to factory defaults 11) Restart web interface
5) Power off system 12) Upgrade from console
6) Reboot system 13) Restore a configuration
Enter an option: 8
root@OPNsense:~ # opnsense-patch 5ec2101ac2
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 5ec2101ac2c67824b94306fe01d1ed97211ea730 Mon Sep 17 00:00:00 2001
|From: Ad Schellevis <ad@opnsense.org>
|Date: Sun, 7 Aug 2016 16:10:19 +0200
|Subject: [PATCH] (insight) check database integrity before start, repair if
| broken
|
|---
| src/opnsense/scripts/netflow/flowd_aggregate.py | 5 +-
| src/opnsense/site-python/sqlite3_helper.py | 69 +++++++++++++++++++++++++
| 2 files changed, 73 insertions(+), 1 deletion(-)
| create mode 100644 src/opnsense/site-python/sqlite3_helper.py
|
|diff --git a/src/opnsense/scripts/netflow/flowd_aggregate.py b/src/opnsense/scripts/netflow/flowd_aggregate.py
|index 8915576..1368046 100755
|--- a/src/opnsense/scripts/netflow/flowd_aggregate.py
|+++ b/src/opnsense/scripts/netflow/flowd_aggregate.py
--------------------------
Patching file opnsense/scripts/netflow/flowd_aggregate.py using Plan A...
Hunk #1 succeeded at 28.
Hunk #2 succeeded at 36.
Hunk #3 succeeded at 130.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/src/opnsense/site-python/sqlite3_helper.py b/src/opnsense/site-python/sqlite3_helper.py
|new file mode 100644
|index 0000000..6c32f7e
|--- /dev/null
|+++ b/src/opnsense/site-python/sqlite3_helper.py
--------------------------
(Creating file opnsense/site-python/sqlite3_helper.py...)
Patching file opnsense/site-python/sqlite3_helper.py using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
done
All patches have been applied successfully. Have a nice day.
root@OPNsense:~ # service flowd_aggregate restart
flowd_aggregate not running? (check /var/run/flowd_aggregate.pid).
Starting flowd_aggregate.
root@OPNsense:~ # service flowd_aggregate status
flowd_aggregate is running as pid 3230.
root@OPNsense:~ #
Still No Data Available i graphs and still nothing in Details either.
/Jonas...
-
It's interesting that flowd_aggregate wasn't running. Can you please check the system log for configd log messages?
-
Let me rephrase that. Now I don't have any data in Details tab of Insight! :|
-
A screenshot would make this as unambiguous as possible.
I remember another thing that could happen is the netflow dump grew to GB proportions so that flowd_aggregate dies while loading the dump:
# ls -lah /var/log/flowd*
-
Like this?
root@OPNsense:/var/log # grep configd system.log
Sep 1 14:27:02 OPNsense configd.py: [8c342056-8a26-44c0-ae6b-d981b679f975] retrieve flow cache statistics
Sep 1 14:27:03 OPNsense configd.py: [d7ebf617-1294-4f55-ad1b-fdcc295a9b3a] retrieve flow cache statistics
Sep 1 14:27:04 OPNsense configd.py: [1b911f1b-91c1-4f33-9d3a-b277539a3513] retrieve flow cache statistics
Sep 1 14:27:05 OPNsense configd.py: [411f13af-6197-4cb8-b80c-64ac0171ba81] retrieve flow cache statistics
Sep 1 14:27:06 OPNsense configd.py: [1acdfd9f-0ec3-402c-b8f8-7fe958ef956a] retrieve flow cache statistics
Sep 1 14:27:12 OPNsense configd.py: [fb34970c-ed49-4e68-af22-c887f843d8f5] request netflow data aggregator timeseries for FlowInterfaceTotals
Sep 1 14:27:12 OPNsense configd.py: [1361e984-a95d-4cab-a5ab-e6535430d168] request netflow data aggregator top usage for FlowDstPortTotals
Sep 1 14:27:12 OPNsense configd.py: [b372ae49-6fd6-4d75-b865-663b3fb6094c] request netflow data aggregator top usage for FlowSourceAddrTotals
Sep 1 14:27:18 OPNsense configd.py: [b372ae49-6fd6-4d75-b865-663b3fb6094c] Script action failed with Command '/usr/local/opnsense/scripts/netflow/get_top_usage.py /provider "FlowSourceAddrTotals" /start_time "1472724000" /end_time "1472732832" /key_fields "src_addr" /value_field "octets" /filter "if=igb2" /max_hits "25"' returned non-zero exit status 1 at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 477, in execute stdout=output_stream, stderr=error_stream) File "/usr/local/lib/python2.7/subprocess.py", line 541, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '/usr/local/opnsense/scripts/netflow/get_top_usage.py /provider "FlowSourceAddrTotals" /start_time "1472724000" /end_time "1472732832" /key_fields "src_addr" /value_field "octets" /filter "if=igb2" /max_hits "25"' returned non-zero exit status 1
Sep 1 14:27:18 OPNsense configd.py: [36e69f2b-e378-4c3c-bc63-17281d3949b9] request netflow data aggregator top usage for FlowInterfaceTotals
Sep 1 14:27:18 OPNsense configd.py: [3ffe4cb6-33b2-4a5c-aaaa-ecfa8f74b607] request netflow data aggregator top usage for FlowInterfaceTotals
Sep 1 14:27:18 OPNsense configd.py: [20965b29-5041-4182-9f05-a24412707e2c] request netflow data aggregator metadata
Sep 1 14:27:21 OPNsense configd.py: [d3ae606f-35d0-412d-9bf5-5bfa532baf06] request netflow data aggregator top usage for FlowSourceAddrDetails
Sep 1 14:27:26 OPNsense configd.py: [d3ae606f-35d0-412d-9bf5-5bfa532baf06] Script action failed with Command '/usr/local/opnsense/scripts/netflow/get_top_usage.py /provider "FlowSourceAddrDetails" /start_time "1472601600" /end_time "1472774399" /key_fields "service_port,protocol,if,src_addr,dst_addr" /value_field "octets" /filter "if=igb2" /max_hits "100"' returned non-zero exit status 1 at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 477, in execute stdout=output_stream, stderr=error_stream) File "/usr/local/lib/python2.7/subprocess.py", line 541, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '/usr/local/opnsense/scripts/netflow/get_top_usage.py /provider "FlowSourceAddrDetails" /start_time "1472601600" /end_time "1472774399" /key_fields "service_port,protocol,if,src_addr,dst_addr" /value_field "octets" /filter "if=igb2" /max_hits "100"' returned non-zero exit status 1
Sep 1 14:29:06 OPNsense configd.py: [853a42a1-b5f4-41f7-b7bf-73a88d5214b9] request netflow data aggregator timeseries for FlowInterfaceTotals
Sep 1 14:29:06 OPNsense configd.py: [353e5e7d-98ad-445b-b57f-ff30c57485e1] request netflow data aggregator top usage for FlowDstPortTotals
Sep 1 14:29:06 OPNsense configd.py: [f20bf34b-c7b5-462f-9f92-b8444fb94d3a] request netflow data aggregator top usage for FlowInterfaceTotals
Sep 1 14:29:06 OPNsense configd.py: [189c0c02-246f-4383-a536-e132671533bd] request netflow data aggregator metadata
Sep 1 14:29:06 OPNsense configd.py: [20bc5ecb-3d1f-4980-b139-2c51acc3feab] request netflow data aggregator top usage for FlowSourceAddrTotals
Sep 1 14:29:06 OPNsense configd.py: [9cdb69f0-b4df-4607-b772-568a1a3c91af] request netflow data aggregator top usage for FlowInterfaceTotals
Sep 1 14:29:13 OPNsense configd.py: [6d482a0f-a957-4f8a-9b5e-131dde3b9aea] request netflow data aggregator top usage for FlowSourceAddrDetails
-
A screenshot would make this as unambiguous as possible.
I remember another thing that could happen is the netflow dump grew to GB proportions so that flowd_aggregate dies while loading the dump:
# ls -lah /var/log/flowd*
How do I attach/include a picture???
root@OPNsense:/var/log # ls -lah /var/log/flowd*
-rw------- 1 root wheel 86M Sep 1 14:36 /var/log/flowd.log
-rw------- 1 root wheel 11M Aug 31 20:39 /var/log/flowd.log.000001
-rw------- 1 root wheel 11M Aug 31 18:55 /var/log/flowd.log.000002
-rw------- 1 root wheel 11M Aug 31 16:42 /var/log/flowd.log.000003
-rw------- 1 root wheel 11M Aug 31 15:15 /var/log/flowd.log.000004
-rw------- 1 root wheel 11M Aug 31 13:11 /var/log/flowd.log.000005
-rw------- 1 root wheel 12M Aug 31 11:06 /var/log/flowd.log.000006
-rw------- 1 root wheel 11M Aug 31 09:06 /var/log/flowd.log.000007
-rw------- 1 root wheel 11M Aug 31 06:57 /var/log/flowd.log.000008
-rw------- 1 root wheel 11M Aug 31 04:39 /var/log/flowd.log.000009
-rw------- 1 root wheel 11M Aug 31 02:22 /var/log/flowd.log.000010
-
Data is back in Details tab, just took a VERY long time to show up. Still no data for today and after 21:50 (isch) yesterday.
-
Thanks for the log, Ad will take a look at this when we is back on Monday.
Click the "Attachments and other options" link directly beneath the text area when writing a reply. :)
Flowd dump size looks ok all the data from yesterday is there (although it seems like quite a lot for a single day)
The last thing we ca try is to remove the sqlite database and keeping the flowd data, it should recalculate your graphs unless there is a problem with the dumps themselves.
# service flowd_aggregate stop
# rm /var/netflow/*
# service flowd_aggregate start
Cheers,
Franco
-
Oh, ok, then there's an issue with the current /var/log/flowd file, it's grown beyond the 11 MB rotation limit.
Let's try bumping flowd as a last resort...
# service flowd restart
-
Oh, ok, then there's an issue with the current /var/log/flowd file, it's grown beyond the 11 MB rotation limit.
Let's try bumping flowd as a last resort...
# service flowd restart
Ok, will do that. Should I wait for the DB rebuild to finnish first?
-
Oh, ok, then there's an issue with the current /var/log/flowd file, it's grown beyond the 11 MB rotation limit.
I've just been looking at this as I also don't have any graphs (I use other tools to monitor the firewall) and my log file seems to have grown rather large:
ls -l /var/log/flowd*
-rw------- 1 root wheel 3545620816 Sep 1 15:10 /var/log/flowd.logI guess I can just stop the service and remove this file and it will rebuild it?
-
Hmm, the flowd* dump files hold all the traffic data. Removing the file would clear any chance of restoring that data, but likely brings back the graphs for any new fresh data. This more or less equals a full reset that you can do from the Reporting: Settings page if one isn't attached to the historic data.
For permanent storage of NetFlow data we strongly recommend sending it to an external server for collection.
-
Hi! Me again. ;D
Rebuilding of DB is still running and data for today are starting to show up in the Details tab.
Still no graphs for Last 2 hours, 30 second average but on some of the drop-downs I get graphs. Historical graphs still is missing also.
/Jonas...
-
Thanks Franco
I'd missed that option on the settings page, I'll clear everything and, as I said, I use other tools to monitor my LAN so the historical data isn't that important to me - I'll see how that goes. :)
-
Bill, it's new since 16.7 and not completely obvious. It's our fault really.
There may be something to read through the flowd file and partially restore it if it was damaged:
# flowd-reader /var/log/flowd.log
Jonas, do you get any error readings on that?
Best to keep the indexing running for now.
-
Bill, it's new since 16.7 and not completely obvious. It's our fault really.
There may be something to read through the flowd file and partially restore it if it was damaged:
# flowd-reader /var/log/flowd.log
Jonas, do you get any error readings on that?
Best to keep the indexing running for now.
Update: I have graphs for Last 2 hours now! Historical graphs start to show up also. WIll probably take some time to rebuild everything. I'll just be patient here...
Thanks so far.
/Jonas...
-
Ahh...
No errors with the following command, just takes very long to process...
# flowd-reader /var/log/flowd.log
Exits like this.
...
FLOW recv_time 2016-09-01T16:02:42.659872 proto 17 tcpflags 00 tos 00 agent [127.0.0.1] src [10.42.50.254]:27402 dst [10.42.50.11]:53 packets 1 octets 87
FLOW recv_time 2016-09-01T16:02:42.659872 proto 17 tcpflags 00 tos 00 agent [127.0.0.1] src [10.42.50.254]:41226 dst [10.42.50.10]:53 packets 1 octets 75
FLOW recv_time 2016-09-01T16:02:42.659872 proto 17 tcpflags 00 tos 00 agent [127.0.0.1] src [10.42.50.254]:41226 dst [10.42.50.11]:53 packets 1 octets 75
root@OPNsense:~ #
-
Very odd. :(
-
Some pictures on how it looks right now.
/Jonas...
-
Franco
Thanks for that tip on the settings page, I reset everything and it's now showing graphs for the last thirty minutes. :)
-
Hi Jonas,
The graphs look ok to me. There are data points for all of the 2h interval and your data collection started on August 31?
Cheers,
Franco
-
...
The graphs look ok to me. There are data points for all of the 2h interval and your data collection started on August 31?
...
Hmm, yes. Looks like I lost all old data in this process for the Insight graphs. The DB rebuild looks like it's finished so I guess I have to live with that unless there is some way of recreation it. Looking at the timestamps in the flowd logfiles i guess it's not possible.
All new data shows up in the graphs as expected.
/Jonas...
-
What "all old data" are we talking about in time ranges? Your oldest entry on the disk was:
-rw------- 1 root wheel 11M Aug 31 02:22 /var/log/flowd.log.000010
As we don't touch the flowd log files at all, this brings me to the question: are you using Nano? Output of the following would shed more light on it:
# df -h
Cheers,
Franco
-
I'm a bit puzzled myself here. I'll backtrack my steps to see if we have removed anything there.
This is the output.
root@OPNsense:~ # df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ada0s1a 77G 1.7G 69G 2% /
devfs 1.0K 1.0K 0B 100% /dev
devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev
root@OPNsense:~ #
-
I've been up for about 40 days with this install and I might not have enabled NetFlow from day one but it was certainly before 31/8.
Somehow the flowd logs have been removed. It might even have been me that have made that happen but I don't know how. Is it possible to make it happen from the GUI? Like when you press Apply in the NetFlow Capture window again?
I know that I have not removed any files in the console (SSH), that's for sure.
/Jonas...
-
Just a thought...
To me this looks like a logrotated set up. If it is then I would understand that my old data is gone since I have ditched the SQL DB that hold that data. (To solve the previous problems)
root@OPNsense:/var/log # ls -lah /var/log/flowd*
-rw------- 1 root wheel 2.1M Sep 2 12:50 /var/log/flowd.log
-rw------- 1 root wheel 11M Sep 2 12:21 /var/log/flowd.log.000001
-rw------- 1 root wheel 11M Sep 2 10:25 /var/log/flowd.log.000002
-rw------- 1 root wheel 11M Sep 2 08:12 /var/log/flowd.log.000003
-rw------- 1 root wheel 11M Sep 2 05:58 /var/log/flowd.log.000004
-rw------- 1 root wheel 11M Sep 2 03:29 /var/log/flowd.log.000005
-rw------- 1 root wheel 11M Sep 2 00:58 /var/log/flowd.log.000006
-rw------- 1 root wheel 11M Sep 1 22:46 /var/log/flowd.log.000007
-rw------- 1 root wheel 11M Sep 1 20:17 /var/log/flowd.log.000008
-rw------- 1 root wheel 107M Sep 1 17:56 /var/log/flowd.log.000009
-rw------- 1 root wheel 11M Aug 31 20:39 /var/log/flowd.log.000010
root@OPNsense:/var/log #
/Jonas...
-
From the GUI this would only be possible with Reporting: Settings: "Reset Netflow Data".
This would, however, make a log entry in the system log. Search for "flush netflow".
The flowd data is decoupled from the database: flowd records the netflow records, the database never deletes them, only processes them.
And if your uptime is greater than the earliest log file we are not looking at a file system corruption either.
Your install is ok too, you're not using a memory file system for /var which would delete all flowd records on boot.
-
I can't recall that I pushed the "Reset Netflow Data" button.
There is no "flush netflow" in system.log. There is not even any "flush" in system.log...
There are however netflow entries as long back as 25/8 in system.log. (I don't know if it's relevant)
Aug 25 22:36:40 OPNsense configd.py: [13707223-0fe2-4f3d-8efb-2ddb7f7b13a1] request netflow data aggregator timeseries for FlowInterfaceTotals
Aug 25 22:36:41 OPNsense configd.py: [f20f15a8-f87e-4cce-9b24-d0fc76ea3bac] request netflow data aggregator top usage for FlowDstPortTotals
Aug 25 22:36:41 OPNsense configd.py: [6fc0132c-0251-4824-95c6-fe4a7a8590ea] request netflow data aggregator top usage for FlowSourceAddrTotals
Aug 25 22:36:41 OPNsense configd.py: [2f73198c-0c5e-442d-bdc9-4f37b9ceb6c1] request netflow data aggregator top usage for FlowInterfaceTotals
Aug 25 22:36:42 OPNsense configd.py: [dc3bab3e-56d6-46bd-b5aa-5c5f3460be54] request netflow data aggregator top usage for FlowInterfaceTotals
Aug 25 22:36:42 OPNsense configd.py: [14cbc454-8242-460d-aa01-7685e70afd36] request netflow data aggregator metadata
Aug 25 22:37:02 OPNsense configd.py: [1df39fb7-ebbb-414e-9e55-e55101168b30] request netflow data aggregator top usage for FlowSourceAddrDetails
Aug 26 12:25:48 OPNsense configd.py: [da581fa7-52a4-4a88-84d1-819b653fdd85] retrieve flow cache statistics
Looks like we are not able to find the cause of this unless you have any more clevver way of checking things.
It's not like the end of the day if I don't have the data, it just bugs me that I don't know why they disappeared.
/Jonas...
-
root@OPNsense:/var/log # ls -lah /var/log/flowd*
-rw------- 1 root wheel 2.1M Sep 2 12:50 /var/log/flowd.log
-rw------- 1 root wheel 11M Sep 2 12:21 /var/log/flowd.log.000001
-rw------- 1 root wheel 11M Sep 2 10:25 /var/log/flowd.log.000002
-rw------- 1 root wheel 11M Sep 2 08:12 /var/log/flowd.log.000003
-rw------- 1 root wheel 11M Sep 2 05:58 /var/log/flowd.log.000004
-rw------- 1 root wheel 11M Sep 2 03:29 /var/log/flowd.log.000005
-rw------- 1 root wheel 11M Sep 2 00:58 /var/log/flowd.log.000006
-rw------- 1 root wheel 11M Sep 1 22:46 /var/log/flowd.log.000007
-rw------- 1 root wheel 11M Sep 1 20:17 /var/log/flowd.log.000008
-rw------- 1 root wheel 107M Sep 1 17:56 /var/log/flowd.log.000009
-rw------- 1 root wheel 11M Aug 31 20:39 /var/log/flowd.log.000010
root@OPNsense:/var/log #
Something strange is going on here! You say that the flowd.log files are not touched or deleted. Compare the above from earlier today with this from now.
root@OPNsense:/var/log # ls -lah /var/log/flowd*
-rw------- 1 root wheel 8.4M Sep 2 16:03 /var/log/flowd.log
-rw------- 1 root wheel 11M Sep 2 14:25 /var/log/flowd.log.000001
-rw------- 1 root wheel 11M Sep 2 12:21 /var/log/flowd.log.000002
-rw------- 1 root wheel 11M Sep 2 10:25 /var/log/flowd.log.000003
-rw------- 1 root wheel 11M Sep 2 08:12 /var/log/flowd.log.000004
-rw------- 1 root wheel 11M Sep 2 05:58 /var/log/flowd.log.000005
-rw------- 1 root wheel 11M Sep 2 03:29 /var/log/flowd.log.000006
-rw------- 1 root wheel 11M Sep 2 00:58 /var/log/flowd.log.000007
-rw------- 1 root wheel 11M Sep 1 22:46 /var/log/flowd.log.000008
-rw------- 1 root wheel 11M Sep 1 20:17 /var/log/flowd.log.000009
-rw------- 1 root wheel 107M Sep 1 17:56 /var/log/flowd.log.000010
root@OPNsense:/var/log #
Who rotates my flowd logs?
/Jonas...
-
Hmm...
Looks like /usr/local/opnsense/scripts/netflow/flowd_aggregate.py has something to do with this...
...
MAX_FILE_SIZE_MB=10
MAX_LOGS=10
...
def check_rotate():
""" Checks if flowd log needs to be rotated, if so perform rotate.
We keep [MAX_LOGS] number of logs containing approx. [MAX_FILE_SIZE_MB] data, the flowd data probably contains
more detailed data then the stored aggregates.
:return: None
...
That's why my old data went away when I ditched the SQL DB. Only 10 flowd.log files are kept to not flood the filesystem. The rest of the aggregated data is in the SQL DB, and nowhere else.
I think that solves the mystery.
Comments?
/Jonas...
-
Hmmmmm, it looks like I lied. Good question indeed, the log files are rotated by flowd_aggregate so that makes our code the prime suspect.
https://github.com/opnsense/core/blob/master/src/opnsense/scripts/netflow/flowd_aggregate.py#L90
And specifically:
https://github.com/opnsense/core/blob/master/src/opnsense/scripts/netflow/flowd_aggregate.py#L47
That makes a current cap of 100 MB of flowd files, not accounting for glitches like your 107 MB record file.
Mystery solved. Now we need to discuss how to handle this more gracefully in the future...
-
You beat me to it, nice! :)
-
Ok, so case closed and some leftover work for you guys to decide on how to proceed.
Thanks again for a super awesome piece of software and the same for your time and effort put into supporting it here in the forums.
Now I have to try to find the next anomaly that I can dig deeper into...
/Jonas...
-
Hi Jonas,
I've added a question ticket to GitHub to keep track of it: https://github.com/opnsense/core/issues/1170
Thanks for the help,
Franco
-
same issue on 17.7
Versions OPNsense 17.7-amd64
FreeBSD 11.0-RELEASE-p11
OpenSSL 1.0.2l 25 May 2017
Aug 24 07:02:57 flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 148, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 79, in aggregate_flowd stream_agg_object.add(flow_record_cpy) File "/usr/local/opnsense/scripts/netflow/lib/aggregates/interface.py", line 70, in add super(FlowInterfaceTotals, self).add(flow) File "/usr/local/opnsense/scripts/netflow/lib/aggregate.py", line 258, in add self._update_cur.execute(self._update_stmt, flow) DatabaseError: database disk image is malformed
should I open a new post?
-
and also:
Aug 24 09:42:54 configd.py: [215ffaee-e444-4af3-828a-ebe97916bf06] Script action failed with Command '/usr/local/opnsense/scripts/netflow/get_top_usage.py /provider "FlowInterfaceTotals" /start_time "1503550800" /end_time "1503560572" /key_fields "direction" /value_field "packets" /filter "if=igb0" /max_hits "25"' returned non-zero exit status 1 at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 477, in execute stdout=output_stream, stderr=error_stream) File "/usr/local/lib/python2.7/subprocess.py", line 541, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '/usr/local/opnsense/scripts/netflow/get_top_usage.py /provider "FlowInterfaceTotals" /start_time "1503550800" /end_time "1503560572" /key_fields "direction" /value_field "packets" /filter "if=igb0" /max_hits "25"' returned non-zero exit status 1
-
No, just go to Reporting: Settings and reset the Netflow/Insight data from there.
Cheers,
Franco
-
solved, thank you so much!