Text only | Text with Images

OPNsense Forum

English Forums => General Discussion => Topic started by: onazari on July 10, 2016, 10:51:40 AM

Title: flowd_aggregate is not running?
Post by: onazari on July 10, 2016, 10:51:40 AM
Hi;
flowd_aggregate is not running, How to fix it?
Title: Re: flowd_aggregate is not running?
Post by: onazari on July 10, 2016, 10:58:45 AM
flowd_aggregate output:
# service flowd_aggregate start
Starting flowd_aggregate.
# service flowd_aggregate start
flowd_aggregate already running?  (pid=74084).
# service flowd_aggregate status
flowd_aggregate is not running.
# service flowd_aggregate status
flowd_aggregate is not running.
Title: Re: flowd_aggregate is not running?
Post by: AdSchellevis on July 10, 2016, 02:34:08 PM
Hi,

can you try to run this in a console and paste the output here?

Code Select
/usr/local/opnsense/scripts/netflow/flowd_aggregate.py console

Best regards,

Ad
Title: Re: flowd_aggregate is not running?
Post by: onazari on July 11, 2016, 05:43:11 AM
Quote from: AdSchellevis on July 10, 2016, 02:34:08 PM
Hi,

can you try to run this in a console and paste the output here?

Code Select
/usr/local/opnsense/scripts/netflow/flowd_aggregate.py console

Best regards,

Ad
Hi, Tanks For Your Replay


# /usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
# /usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
# /usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
Nothing output

Title: Re: flowd_aggregate is not running?
Post by: AdSchellevis on July 11, 2016, 08:44:58 AM
ok, that's not much  :)

can you try to grep your syslog?

Code Select
clog /var/log/system.log | grep flowd


Title: Re: flowd_aggregate is not running?
Post by: onazari on July 11, 2016, 09:57:24 AM
Quote from: AdSchellevis on July 11, 2016, 08:44:58 AM
ok, that's not much  :)

can you try to grep your syslog?

Code Select
clog /var/log/system.log | grep flowd



Output is:
Code Select
# clog /var/log/system.log | grep flowd
Jul  3 09:00:34 OPNsense root: /usr/local/etc/rc.d/flowd: WARNING: failed to start flowd
Jul  9 16:32:14 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 11:49:45 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 12:03:40 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 12:04:56 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:18:38 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:18:53 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:19:10 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:21:57 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:22:00 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:22:07 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:23:52 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:24:37 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:24:53 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:24:56 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 07:19:24 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 08:09:32 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 08:11:11 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 08:11:14 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 11:35:04 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 11:35:07 bop root: /usr/local/etc/rc.d/flowd_aggregate: WARNING: failed to start flowd_aggregate
Jul 11 11:35:08 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 11:35:15 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 11:35:21 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Title: Re: flowd_aggregate is not running?
Post by: AdSchellevis on July 11, 2016, 10:27:19 AM
It looks like a corruption in your flowd log files, you could remove all collected data and start over.


Code Select

rm /var/log/flowd.log*
rm /var/netflow/*.sqlite


The /var/netflow/ contains the already parsed data (if any), you could first try to keep that one, and if that doesn't work remove those too.
Title: Re: flowd_aggregate is not running?
Post by: franco on July 11, 2016, 11:00:50 AM
We do need a GUI reset for this, I've recorded this here: https://github.com/opnsense/core/issues/983
Title: Re: flowd_aggregate is not running?
Post by: Supermule on March 30, 2020, 07:53:16 PM
Same error today

Mar 30 19:39:54    /flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 160, in run aggregate_flowd(self.config, do_vacuum) File...

Version 20.1.3
Title: Re: flowd_aggregate is not running?
Post by: adiz0r on May 15, 2020, 10:00:07 PM
flowd_aggregate regularly dies. I had to create a monit task for it to restart and keep it alive.
Title: Re: flowd_aggregate is not running?
Post by: fog on July 20, 2020, 03:39:12 PM
QuoteI had to create a monit task for it to restart and keep it alive.
I have the same problem. The flowd_aggregate service stop after 1-7 days.
My working solution: In Services: Monit: Settings create new service:

Enable service checks: checked
Name: flowd_aggregate_monitor
Type: Process
PID file: /var/run/flowd_aggregate.pid
Start: /usr/sbin/service flowd_aggregate start
Stop: /usr/sbin/service flowd_aggregate stop
Tests: Nothing selected
Depends: Nothing selected

If the email settings are correct set under Services: Monit: Settings -> General settings and Alert Settings mails are sent:

Does not exist Service flowd_aggregate_monitor

   Date:        Sun, 19 Jul 2020 06:00:02
   Action:      restart
   Host:        ****
   Description: process is not running

Your faithful employee,
Monit

Exists Service flowd_aggregate_monitor

   Date:        Sun, 19 Jul 2020 06:02:10
   Action:      alert
   Host:        ****
   Description: process is running with pid 58260

Your faithful employee,
Monit



Title: Re: flowd_aggregate is not running?
Post by: GuruLee on November 06, 2020, 12:05:58 PM
Quote from: franco on July 11, 2016, 11:00:50 AM
We do need a GUI reset for this, I've recorded this here: https://github.com/opnsense/core/issues/983
Thank you, this fixed the no insight data for me.
Text only | Text with Images