[solved] netflow - bytes in and out for a single device (ip)

[jata]

I have netflow enabled and outputting to influx via telegraf. Seems to be working fine but...

I don't really capture the data i expected so I probably misunderstand and I was hoping to get some help here.

What I want to visualise is internet usage (data rate) for a single client - for testing I am using MB/hr.

To achieve this I am using bytes_in where dest = ip and bytes_out where source = ip (sum grouped by hour)

but I am only getting a fraction of the data rate I expect.

Has anyone set this up and can explain what I am doing wrong?

and I have tried many different combos of bytes_in bytes_out dest / source etc

[jata]

I have been investigating and experimenting with this and I am stuck - appreciate any help.

I have netflow configured in opnsense and it is exporting to influxdb via telegraf - my understanding is that this will provide summary metrics and I should be able to visualise bandwidth etc overall and by client.

I am also using the opnsense telegraf plugin and using this with network input - this will give me total data by interface. This is working fine.


My issue is that the netflow metrics collected seem to only be a fraction of what I expect. Maybe it is due to sampling rate?

See charts attached. The first two are using the opnsense telegraf plugin reporting data total and data rate by hr and are working fine. The issue with this is that I can only report at interface level (not client)

The third chart is using netflow sum bytes_in by hr and I what I see is only a small amount of the data and the data does not seem to align in any way at all.

I think I am doing something really silly and would appreciate any assistance with this please.





 

[Patrick M. Hausen]

I use netflow with the default settings and Elastiflow and I do see detailed data for all internal and external systems.

[jata]

Great. Then using netflow is something I should be able to get working and I am on the right track...

I am using defaults for netflow and everything else that i am pushing through telegraf and into influx is working fine so I think the setup fine.

My network is ipv4 only so I guess I can try switching from netflow v9 to netflow v5 in opnsense and telegraf and see if that helps.

Anything else you can share that might help?

[Patrick M. Hausen]

No, sorry. Never used netflow with anything but Elastiflow.

[jata]

Thanks for the help. I did a bit more research and have found the solution.

The problem is that the data needs to be tagged when doing this with telegraf and influxdb. Records were being overridden in influxdb with default config.

The solution in telegraf is to add a processing step in the pipeline to convert some key fields to tags. Use with caution as this can create cardinality / performance issues due to the number of series that are created :-)

Information linked below to help other folks who are trying something similar.

https://github.com/influxdata/telegraf/blob/release-1.35/plugins/inputs/netflow/README.md#metrics-are-missing-at-the-output