Netflow - again high I/O

[GreenMatter]

So it seems all started with upgrade to 26.1 and at that time (January) issues with Neighbours Discovery. That was sorted out by the next upgrades/fixes but I started having similar issues with netflow/rrd. Once they are enabled, they cause high I/O and CPU demand. I tried, repair and reset netflow/rrd data plus manual removal of content of /various/netflow/ folder (with stopped neighbours discovery and netflow). Nothing helps; currently system is up to date: 26.1.6.
How to fix it?

[GreenMatter]

Writing above post triggered better thinking ;-). The culprit was selection of VPNs' interfaces in netflow settings. Once they've been removed, all went back to normal...

[Patrick M. Hausen]

Don't save netflow data on OPNsense. Export to a netflow collector like Elastiflow and save your SSD 🙂

[nero355]

Don't save netflow data on OPNsense. Export to a netflow collector like Elastiflow and save your SSD 🙂

Looks seriously sweet as far as I can tell from your other recent post : https://forum.opnsense.org/index.php?msg=264974

Should I need something like that I will definitely consider it! :)


For now I have got almost all logging disabled in OPNsense since I barely need any of it.

[Patrick M. Hausen]

Netflow is a behemoth designed for large ISPs. It will scrub your SSD to death in weeks if you log locally and have some considerable amount of traffic.

It was designed from the start to just collect the data on the (at the time) seriously underpowered control plane of the (Cisco) device and get it off the box to some collector as fast as possible.

The smallest recommended deployment - even for a home lab - for Elastiflow is 4 cores, 16 G of RAM, a couple of hundred G of disk.
Repeating myself - you don't want that on OPNsense proper.

But it works as advertised. I get the same beautiful graphs from my OPNsense to my Ubuntu VM running the stack. And noticed as written in that other post some odd traffic on UDP/1194 immediately ;-)

They promise the license will be free forever, they just want you to register an account and extend the license once per year. Like e.g. Maxmind, too.
The limit for the free tier is 4000 flow records per second. If you outgrow that, you have bigger fish to fry ;-)

Kind regards,
Patrick

[nero355]

Netflow is a behemoth designed for large ISPs. It will scrub your SSD to death in weeks if you log locally and have some considerable amount of traffic.

The smallest recommended deployment - even for a home lab - for Elastiflow is 4 cores, 16 G of RAM, a couple of hundred G of disk.

They promise the license will be free forever, they just want you to register an account and extend the license once per year. Like e.g. Maxmind, too.
The limit for the free tier is 4000 flow records per second. If you outgrow that, you have bigger fish to fry ;-)

Good to know! Thnx! :)