Netflow sqlite file 33GB and full disk

[slair]

Having an issue where /var/netflow/src_addr_details_086400.sqlite has grown to 33GB.  Then flowd_aggregate.py runs and fills up the whole disk then crashes.  Any idea why that file has grown to 33GB?  How is that sqlite database related to Insight and flowd_agggregate.py? 

Thanks for any help/insight you can give!  OPNsense 20.7.4

Here are some additional details:

Code: [Select]

# ls -lh /var/netflow
total 36154824
-rw-r-----  1 root  wheel    12M Dec 16 01:47 dst_port_000300.sqlite
-rw-r-----  1 root  wheel    14M Dec 16 01:47 dst_port_003600.sqlite
-rw-r-----  1 root  wheel   270M Dec 16 01:47 dst_port_086400.sqlite
-rw-r-----  1 root  wheel   2.5M Dec 16 01:47 interface_000030.sqlite
-rw-r-----  1 root  wheel   1.2M Dec 16 01:47 interface_000300.sqlite
-rw-r-----  1 root  wheel   456K Dec 16 01:47 interface_003600.sqlite
-rw-r-----  1 root  wheel    60K Dec 16 01:47 interface_086400.sqlite
-rw-r-----  1 root  wheel    12K Dec 16 01:47 metadata.sqlite
-rw-r-----  1 root  wheel   303M Dec 16 01:47 src_addr_000300.sqlite
-rw-r-----  1 root  wheel   122M Dec 16 01:47 src_addr_003600.sqlite
-rw-r-----  1 root  wheel   568M Dec 16 01:47 src_addr_086400.sqlite
-rw-r-----  1 root  wheel    33G Dec 16 02:37 src_addr_details_086400.sqlite


Code: [Select]

# ls -lh /var/log/flowd.log*
-rw-------  1 root  wheel    67M Dec 16 02:27 /var/log/flowd.log
-rw-------  1 root  wheel   5.6G Dec 16 01:20 /var/log/flowd.log.000001
-rw-------  1 root  wheel    12M Dec 15 14:11 /var/log/flowd.log.000002
-rw-------  1 root  wheel    21M Dec 15 14:09 /var/log/flowd.log.000003
-rw-------  1 root  wheel    13M Dec 15 14:06 /var/log/flowd.log.000004
-rw-------  1 root  wheel    15M Dec 15 14:05 /var/log/flowd.log.000005
-rw-------  1 root  wheel    13M Dec 15 14:02 /var/log/flowd.log.000006
-rw-------  1 root  wheel    14M Dec 15 13:59 /var/log/flowd.log.000007
-rw-------  1 root  wheel    18M Dec 15 13:56 /var/log/flowd.log.000008
-rw-------  1 root  wheel    11M Dec 15 13:52 /var/log/flowd.log.000009
-rw-------  1 root  wheel    14M Dec 15 13:50 /var/log/flowd.log.000010


Code: [Select]

2020-12-16T02:37:26 /flowd_aggregate.py[81444] flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 160, in run aggregate_flowd(self.config, do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 86, in aggregate_flowd stream_agg_object.cleanup(do_vacuum) File "/usr/local/opnsense/scripts/netflow/lib/aggregates/__init__.py", line 213, in cleanup self._update_cur.execute('vacuum') sqlite3.OperationalError: database or disk is full
...
...
2020-12-16T02:27:34 kernel pid 49300 (suricata), uid 0 inumber 13643558 on /mnt: filesystem full
2020-12-16T02:27:25 kernel pid 49300 (suricata), uid 0 inumber 13643558 on /mnt: filesystem full
2020-12-16T02:27:22 kernel pid 81444 (python3.7), uid 0 inumber 13563330 on /mnt: filesystem full
2020-12-16T02:22:00 kernel pid 66066 (dd), uid 2 inumber 13563481 on /mnt: filesystem full
2020-12-16T01:46:56 /flowd_aggregate.py[81444] vacuum src_addr_details_086400.sqlite


[benibilme]

Hello,

I have the similar problem, I receive "gzip is failed" error during updates. Did you find a solution to your problem?