flowd_aggregate is not running?

[onazari]

Hi;
flowd_aggregate is not running, How to fix it?

[onazari]

flowd_aggregate output:
# service flowd_aggregate start
Starting flowd_aggregate.
# service flowd_aggregate start
flowd_aggregate already running?  (pid=74084).
# service flowd_aggregate status
flowd_aggregate is not running.
# service flowd_aggregate status
flowd_aggregate is not running.

[AdSchellevis]

Hi,

can you try to run this in a console and paste the output here?

Code Select Expand

/usr/local/opnsense/scripts/netflow/flowd_aggregate.py console

Best regards,

Ad

[onazari]

Hi,

can you try to run this in a console and paste the output here?

Code Select Expand

/usr/local/opnsense/scripts/netflow/flowd_aggregate.py console

Best regards,

Ad

Hi, Tanks For Your Replay


# /usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
# /usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
# /usr/local/opnsense/scripts/netflow/flowd_aggregate.py console
Nothing output

[AdSchellevis]

ok, that's not much  :)

can you try to grep your syslog?

Code Select Expand

clog /var/log/system.log | grep flowd


[onazari]

ok, that's not much  :)

can you try to grep your syslog?

Code Select Expand

clog /var/log/system.log | grep flowd


Output is:

Code Select Expand

# clog /var/log/system.log | grep flowd
Jul  3 09:00:34 OPNsense root: /usr/local/etc/rc.d/flowd: WARNING: failed to start flowd
Jul  9 16:32:14 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 11:49:45 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 12:03:40 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 12:04:56 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:18:38 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:18:53 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:19:10 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:21:57 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:22:00 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:22:07 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:23:52 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:24:37 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:24:53 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 10 13:24:56 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 07:19:24 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 08:09:32 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 08:11:11 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 08:11:14 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 11:35:04 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 11:35:07 bop root: /usr/local/etc/rc.d/flowd_aggregate: WARNING: failed to start flowd_aggregate
Jul 11 11:35:08 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 11:35:15 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small
Jul 11 11:35:21 bop flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last):   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 145, in run     aggregate_flowd(do_vacuum)   File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 67, in aggregate_flowd     for flow_record in parse_flow(prev_recv):   File "/usr/local/opnsense/scripts/netflow/lib/parse.py", line 92, in parse_flow     for flow in flog: ValueError: store_read_flow: Internal error: flow buffer too small



[AdSchellevis]

It looks like a corruption in your flowd log files, you could remove all collected data and start over.

Code Select Expand


rm /var/log/flowd.log*
rm /var/netflow/*.sqlite


The /var/netflow/ contains the already parsed data (if any), you could first try to keep that one, and if that doesn't work remove those too.

[franco]

We do need a GUI reset for this, I've recorded this here: https://github.com/opnsense/core/issues/983

[Supermule]

Same error today

Mar 30 19:39:54    /flowd_aggregate.py: flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 160, in run aggregate_flowd(self.config, do_vacuum) File...

Version 20.1.3

[adiz0r]

flowd_aggregate regularly dies. I had to create a monit task for it to restart and keep it alive.

[fog]

I had to create a monit task for it to restart and keep it alive.

I have the same problem. The flowd_aggregate service stop after 1-7 days.
My working solution: In Services: Monit: Settings create new service:

Enable service checks: checked
Name: flowd_aggregate_monitor
Type: Process
PID file: /var/run/flowd_aggregate.pid
Start: /usr/sbin/service flowd_aggregate start
Stop: /usr/sbin/service flowd_aggregate stop
Tests: Nothing selected
Depends: Nothing selected

If the email settings are correct set under Services: Monit: Settings -> General settings and Alert Settings mails are sent:

Does not exist Service flowd_aggregate_monitor

   Date:        Sun, 19 Jul 2020 06:00:02
   Action:      restart
   Host:        ****
   Description: process is not running

Your faithful employee,
Monit

Exists Service flowd_aggregate_monitor

   Date:        Sun, 19 Jul 2020 06:02:10
   Action:      alert
   Host:        ****
   Description: process is running with pid 58260

Your faithful employee,
Monit

[GuruLee]

We do need a GUI reset for this, I've recorded this here: https://github.com/opnsense/core/issues/983

Thank you, this fixed the no insight data for me.