Hi all,
so I finally found a capable Netflow consumer/visualiser to send data to - thanks to @9axqe for the discussion and some pointers. I finally settled on ElastiFlow. Their Docker based quickstart (https://www.elastiflow.com/docs/flowcoll/install_docker_ubuntu_elastic_stack) instructions worked great.
Now apparently OPNsense sends ifName and ifDescr in the flow data with both fields containing e.g. "pppoe0" or "vlan01" or "igb0" ... whatever.
It would of course be great of ifName would contain "pppoe0" and ifDescr "WAN" if that interface is assigned in that way.
But that is still perfectly fine, because ElastiFlow provides a simple method to map interface names like so:
192.168.1.1:
11:
ifName: LAN
internal: true
The IP address is the address of the Netflow sender, the first number is the ifIndex which is identical for both Netflow and SNMP, and then you can change arbitrary fields so the interface name in the ElastiFlow dashboards is "LAN" instead of "vlan01" by the example above.
Great.
Now after I created entries for all my interfaces there is one single ifName for which OPNsense sends flow records to ElastiFlow and that is named - oddly - "index: 0".
What real interface on the OPNsense system is this? So I can adjust the displayed name to something reasonable.
Mind you, this is data sent from OPNsense to ElastiFlow with an interface name that is literally "index: 0" if I am not grossly mistaken. And I just wonder what precisely that data is - yes, there are flows if I filter for "index: 0". But nothing that obviously matches a real interface.
I can send anyone a single flow record in JSON showing that weird ifName field. I just do not want to post that data publicly where it will be indexed by search engines etc. Only IP addresses and DNS names in there, but still.
Thanks,
Patrick
# ifinfo | grep -e ^Int -e index:
I think index: 0 is prohibited. It's really just the current interface linked list in the kernel, which is also why the index is not great for identification but I remember some limitation in that netflow implementation that made it necessary.
Cheers,
Franco
Thanks! So I will look if I can filter that from the Kibana view.
Found it! By analysing the packet contents - isn't a flow analyser a nice tool?
These are packets originating locally.
Thanks for your comment that lead me in the right direction, eventually.
Parick, I did sign up, tried and stopped setting up netflow due to the hardware resources required. More than I had available for a VM for it at the time.
How much memory have you assigned to it, and cpu ? Assuming a VM.
I used the Docker based quick start method linked in my initial post. They recommend 16 GB of memory and 8 CPUs - I got by with 16 GB of memory and 4 CPUs, system runs well.
Ubuntu VM on Proxmox. 300 GB virtual disk, 32 GB swap, rest LVM.
The two open source contenders @9axqe mentioned in our discussion a couple of days ago demand even more memory. As does the complete virtual appliance delivered by ElastiFlow. Plus that one comes with a 2 TB (!) virtual disk.
So manual Ubuntu install and then their all in one Docker based script is what I recommend.
cheers Patrick