Netflow - continous 50-60 Mb/s disk writes

[molnart]

I have a long running issue with netflow - literally for years. netflow starts to hog up my disk and cpu usage with huge amount of writes. restarting opnsense takes 20-30 minutes because of backing up and restoring netflow data.
Usually its sqlite accessing some .sqlite.fix file.
Every time this happens, i need to reset netflow logs from the UI, manually stop the sqlite process and delete the contents of the netflow directory.

Is there a better way to handle this, while keeping long term netflow statistics (10+ years)? 

[meyergru]

Do you use a ZFS install? Things like that usually happen when there are filesystem inconsistencies created during hard stops of the netflow process. That cannot happen with ZFS.

[Patrick M. Hausen]

Send the netflow data to a different system.

[pfry]

[...]Is there a better way to handle this, while keeping long term netflow statistics (10+ years)?

Send the netflow data to a different system.

I rather like consolidation, but given your desired retention time, I have to agree. You'd probably end up with a write-only setup anyway, rather like folks I knew who recorded hundreds of hours of TV, films, etc. (Flow records would at least be usefully searchable, but still.)