Intrusion Detection Alerts not sent by Telegraf

[koushun]

* OPNsense 21.7-amd64
* os-telegraf 1.11.0 (Telegraf 1.19.0)
* InfluxDB 2.0.7

I was hoping this release would fix the problem I have with Intrusion Detection Alerts not beiing sent to InfluxDB.

https://opnsense.org/opnsense-21-7-released/ states that "intrusion detection: fix alert reads from eve.json", but as I was unable to find any more information in the release post, this statement is probably referering to another issue. Anyways, to bump this issue to 21.7, I'll post my findings here on this more active Forum section - I have rambled over at the General Discussion page quite a bit : https://forum.opnsense.org/index.php?topic=16966.0)

1) Suricata is started with the user root
2) Suricata produces events in the /var/log/suricata/ directory which has these permissions

Code: [Select]

drwx------   2 root      wheel       12B Jul 30 00:00 suricata
3) Suricata creates JSON entries in the file /var/log/suricata/eve.json which has these permissions

Code: [Select]

-rwx------  1 root  wheel    30K Jul 30 14:53 /var/log/suricata/eve.json
4) Enabling "Intrusion Detection Alerts" in Telegraf, creates this config in /usr/local/etc/telegraf.conf

Code: [Select]

[[inputs.tail]]
  data_format = "json"
  files = ["/var/log/suricata/eve.json"]
  name_override = "suricata"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
  json_string_fields = ["*"]
5) Telegraf is started by the user telegraf

Code: [Select]

ps aux | grep telegraf
telegraf 12093   0.0  1.1  5040852   92304  -  S    14:02     0:35.47 /usr/local/bin/telegraf --quiet --config=/usr/local/etc/telegraf.conf --config-directory=/usr/local/etc/telegraf.d
6) Telegraf does not have permissions to view the file /var/log/suricata/eve.json

Code: [Select]

sudo -u telegraf more /var/log/suricata/eve.json
/var/log/suricata/eve.json: Permission denied
7) There is no errors in /var/log/telegraf/telegraf.log from [[inputs.tail]] that the file is inaccessible

A quick fix is to add the user telegraf to the wheel group and change permissions. This will not survive a reboot.

Code: [Select]

pw group mod wheel -m telegraf
chmod 750 /var/log/suricata ; chmod 750 /var/log/suricata/eve.json

Then metrics are populated in InfluxDBv2.

Although I have not figured this out entirely, I think the [[inputs.tail]] section would benefit from having "timestamp" from eve.json parsed.

https://docs.influxdata.com/telegraf/v1.19/data_formats/input/json/#json_time_key-json_time_format:

By default the current time will be used for all created metrics, to set the time using the JSON document you can use the json_time_key and json_time_format options together to set the time to a value in the parsed document.

The json_time_key option specifies the key containing the time value and json_time_format must be set to unix, unix_ms, or the Go “reference time” which is defined to be the specific time: Mon Jan 2 15:04:05 MST 2006.

Code: [Select]

[[inputs.tail]]
  data_format = "json"
  files = ["/var/log/suricata/eve.json"]
  name_override = "suricata-alerts"
  tag_keys = ["flow_id","in_iface","event_type","src_ip","src_port","dest_ip","dest_port","proto"]
  json_string_fields = ["*"]
  json_time_key = "timestamp"
  json_time_format = "2006-01-02T15:04:05-0700"

Has anyone got this to work / having the same problems?

[koushun]

To produce metrics of nested JSON objects, we should also switch data_format to "json_v2" - https://github.com/influxdata/telegraf/tree/master/plugins/parsers/json_v2

This will allow us to produce metrics of subitems, such as alert.signature, alert.category - which is not possible at this moment (??).

Screenshot of what I have accomplished so far- I cannot understand how anyone has been using this feature before, without doing group mod and / or chmod.

[koushun]

No one is experiencing the same issues here? 

[mimugmail]

Only a few ppl use telegraf, and less in combinaton with IDS I'd guess

[koushun]

Then what are they using 

[Patrick M. Hausen]

Only few people on the forum are using Suricata at all as I learned in the last user group meeting.

Given your knowledge about how the components are supposed to work and what you would like to see as the output - even if you don't/cannot code - it is probably more productive to open an issue on github describing your concerns.
If you do indeed code, have at it. It's open source 

HTH,
Patrick

[koushun]

They do not use the Intrusion Detection service? Que pasa - can you please enlighten me? How are they securing / monitoring public facing services then?

Yes. There are issues opened on GitHub.


Anyways, thanks for reaching out!

[mimugmail]

Then what are they using 

Pfelk

[koushun]

mimugmail
I do not comprehend your answer. Pfelk is just a set of tools for analyzing this data, which comes from a source, like Suricata. Yes?

The question was meant for the statement 'Only few people on the forum are using Suricata at all as I learned in the last user group meeting.'

If it was meant to say 'Only few people on the forum are using Telegraf at all as I learned in the last user group meeting', I would understand your answer

Have not had the time to look into this. Life and all. Thank you for providing such an awesome products anyways Merry Christmas!