OPNsense Forum

English Forums => General Discussion => Topic started by: onazari on July 10, 2016, 10:51:40 am

Title: flowd_aggregate is not running?
Post by: onazari on July 10, 2016, 10:51:40 am: Hi;
flowd_aggregate is not running, How to fix it?
Title: Re: flowd_aggregate is not running?
Post by: onazari on July 10, 2016, 10:58:45 am: flowd_aggregate output:
# service flowd_aggregate start
Starting flowd_aggregate.
# service flowd_aggregate start
flowd_aggregate already running? (pid=74084).
# service flowd_aggregate status
flowd_aggregate is not running.
# service flowd_aggregate status
flowd_aggregate is not running.
Title: Re: flowd_aggregate is not running?
Post by: AdSchellevis on July 10, 2016, 02:34:08 pm: Hi,

can you try to run this in a console and paste the output here?

Code: [Select]
/usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
Best regards,

Ad
Title: Re: flowd_aggregate is not running?
Post by: onazari on July 11, 2016, 05:43:11 am: Quote from: AdSchellevis on July 10, 2016, 02:34:08 pm
Hi,

can you try to run this in a console and paste the output here?

Code: [Select]
/usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
Best regards,

Ad
Hi, Tanks For Your Replay

# /usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
# /usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
# /usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
Nothing output
Title: Re: flowd_aggregate is not running?
Post by: AdSchellevis on July 11, 2016, 08:44:58 am: ok, that's not much :)

can you try to grep your syslog?

Code: [Select]
clog /var/log/system.log | grep flowd
Title: Re: flowd_aggregate is not running?
Post by: onazari on July 11, 2016, 09:57:24 am: Quote from: AdSchellevis on July 11, 2016, 08:44:58 am
ok, that's not much :)

can you try to grep your syslog?

Code: [Select]
clog /var/log/system.log | grep flowd

Output is:
Code: [Select]
# clog /var/log/system.log | grep flowd Jul 3 09:00:34 OPNsense root: /usr/local/etc/rc.d/flowd: WARNING: failed to start flowd Jul 9 16:32:14 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 11:49:45 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 12:03:40 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 12:04:56 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 13:18:38 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 13:18:53 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 13:19:10 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 13:21:57 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 13:22:00 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 13:22:07 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 13:23:52 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 13:24:37 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 13:24:53 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 10 13:24:56 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 11 07:19:24 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 11 08:09:32 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 11 08:11:11 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 11 08:11:14 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 11 11:35:04 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 11 11:35:07 bop root: /usr/local/etc/rc.d/flowd_aggregate: WARNING: failed to start flowd_aggregate Jul 11 11:35:08 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 11 11:35:15 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small Jul 11 11:35:21 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run aggregate_flowd(do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd for flow_record in parse_flow(prev_recv): File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Title: Re: flowd_aggregate is not running?
Post by: AdSchellevis on July 11, 2016, 10:27:19 am: It looks like a corruption in your flowd log files, you could remove all collected data and start over.

Code: [Select]
rm /var/log/flowd.log* rm /var/netflow/*.sqlite
The /var/netflow/ contains the already parsed data (if any), you could first try to keep that one, and if that doesn't work remove those too.
Title: Re: flowd_aggregate is not running?
Post by: franco on July 11, 2016, 11:00:50 am: We do need a GUI reset for this, I've recorded this here: https://github.com/opnsense/core/issues/983
Title: Re: flowd_aggregate is not running?
Post by: Supermule on March 30, 2020, 07:53:16 pm: Same error today

Mar 30 19:39:54 /flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 160, in run aggregate_flowd(self.config, do_vacuum) File…

Version 20.1.3
Title: Re: flowd_aggregate is not running?
Post by: adiz0r on May 15, 2020, 10:00:07 pm: flowd_aggregate regularly dies. I had to create a monit task for it to restart and keep it alive.
Title: Re: flowd_aggregate is not running?
Post by: fog on July 20, 2020, 03:39:12 pm: Quote
I had to create a monit task for it to restart and keep it alive.
I have the same problem. The flowd_aggregate service stop after 1-7 days.
My working solution: In Services: Monit: Settings create new service:

Enable service checks: checked
Name: flowd_aggregate_monitor
Type: Process
PID file: /var/run/flowd_aggregate.pid
Start: /usr/sbin/service flowd_aggregate start
Stop: /usr/sbin/service flowd_aggregate stop
Tests: Nothing selected
Depends: Nothing selected

If the email settings are correct set under Services: Monit: Settings -> General settings and Alert Settings mails are sent:

Does not exist Service flowd_aggregate_monitor

   Date: Sun, 19 Jul 2020 06:00:02
   Action: restart
   Host: ****
   Description: process is not running

Your faithful employee,
Monit

Exists Service flowd_aggregate_monitor

   Date: Sun, 19 Jul 2020 06:02:10
   Action: alert
   Host: ****
   Description: process is running with pid 58260

Your faithful employee,
Monit
Title: Re: flowd_aggregate is not running?
Post by: GuruLee on November 06, 2020, 12:05:58 pm: Quote from: franco on July 11, 2016, 11:00:50 am
We do need a GUI reset for this, I've recorded this here: https://github.com/opnsense/core/issues/983
Thank you, this fixed the no insight data for me.