OPNsense, prometheus, grafana

[rungekutta]

Hi - does someone use this combination successfully for monitoring and graphing? I can't get it to behave properly but I don't know if I'm misinterpreting metrics or if the FreeBSD implementation of node exporter is just quirky.

So... I was happy enough to find the os-node_exporter plugin. This installs the Prometheus exporter and it runs fine. Prometheus picks up the metrics and all good... except that the node_network_[receive|transmit]_bytes_total metric doesn't actually do seem to reflect anything that goes in and out. No matter what I pump through the firewall, the metrics maintain a very slow, linear increment across all devices and seemingly unrelated to what actually goes on. I can't figure out why - and by comparison, my Linux systems behave exactly as expected from Prometheus documentation and tutorials, so I don't think the problem lies with my queries.

Anyone else managed to get this to work?

[rungekutta]

Actually I just noticed something else. When I go into Reporting->Traffic, those graphs at the top also do not reflect reality and in fact seem to show pretty much the same thing that the Prometheus node exporter does. HOWEVER the tables below, which breaks down traffic by IP, seem to reflect reality. E.g. when a computer on the LAN ramps up an internet download it immediately shows up in the table graphs, however the summarizing graphs above at interface level does not reflect anything.

Also, the graphs in Reporting->Health->Traffic show the right thing.

Plot thickens...? What am I missing? Starting to wonder if I'm missing something obvious here  ;-)

[rungekutta]

Replying to myself again.

I see this in the documentation (https://docs.opnsense.org/manual/reporting_traffic.html):

When an interface doesn’t report traffic and you are certain there should be any, make sure to check if you have any services enabled that use netmap (zero copy) support on the selected interface (such as IPS and Sensei). When zero copy is used, packets won’t by copied in the kernel in which case bpf can’t read from the usual in memory buffer.

I have Suricata enabled. So I guess that's what's going on? When using IPS, the traffic disappears from these reporting graphs just as well as they disappear from the exporter into Prometheus...?

Bit disappointing if I'm forced to choose between one or the other.

[mimugmail]

Known problem with FreeBSD 12.1 introduced with 20.7.
Sensei guys are maybe working in this

[Patrick M. Hausen]

There are collectd and Telegraf plugins - both can send metrics to Influxdb --> Grafana.
Possibly that works better for you?

[rungekutta]

Very interesting, thanks. Do you know if they would be impacted in the same way or not to the (FreeBSD 12.1) problems mentioned above?

[Patrick M. Hausen]

Sorry, no. My OPNsense hardware would not have the power to run one of the IPS packages.
I do use collectd with a graphite compatible connector in InfluxDB:

Code: [Select]

[[graphite]]
  enabled = true
  database = "graphite"
  retention-policy = ""
  bind-address = ":2003"
  protocol = "tcp"
  consistency-level = "one"

  separator = "."

  templates = [
    "servers.* .hostname.resource.instance.measurement*",
  ]

And this configuration in OPNsense - see screenshot. I use the "servers." prefix because my FreeNAS systems do the same out of the box and I want everything in one database.

HTH,
Patrick