Unable to get Telegraf plugin to send IPS data (suricata)

[koushun]

SOLVED (??):

Check "Enable eve syslog output" under Intrusion Detection > Administration, hit Apply.
Enabling this will create will create JSON entries in the file /var/log/suricata/eve.json whenever there is a new entry / Alert in Intrusion Detection.

To get Telegraf to grab these entries from the eve.json file and send it to your bucket in InfluxDBv2, you will have to enable "Intrusion Detection Alerts". Go to Services > Telegraf > Input and check
* Intrusion Detection Alerts

This will create an entry in your `/usr/local/etc/telegraf.conf` file on the very bottom:

Code: [Select]

[[inputs.tail]]
  data_format = "json"
  files = ["/var/log/suricata/eve.json"]
  name_override = "suricata"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
  json_string_fields = ["*"]

However, change this config (add json_time_key and json_time_format).

This is my [[inputs.tail]] configuration:

Code: [Select]

[[inputs.tail]]
  data_format = "json"
  files = ["/var/log/suricata/eve.json"]
  name_override = "suricata"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
#  json_string_fields = ["*"]
  json_time_key = "timestamp"
  json_time_format = "2006-01-02T15:04:05-0700"

Do not start / stop the Telegraf service from the GUI from now on. This will reset your telegraf.conf file.

However, as root, doing a service telegraf restart does not output any error messages, and everything looks fine. See by yourself with tail -f /var/log/telegraf/telegraf.log.

Although it isn't. The telegraf user does not have access permissions to the eve.json file

To let the service / user telegraf to be able to read the /var/log/suricata/eve.json file, you can do the following.

Add the user telegraf to the group wheel and confirm with pw groupshow;

Code: [Select]

pw group mod wheel -m telegraf
pw groupshow wheel
wheel:*:0:root,telegraf

Change permissions on both the /var/log/suricata directory and the eve.json file:

Code: [Select]

chmod 750 /var/log/suricata
chmod 750 /var/log/suricata/eve.json

Now, restart the telegraf service as root: service telegraf restart

If you are not seeing anything flowing to your bucket, copy a line in the eve.json file, using vi, to the very bottom (to update the file and let [[inputs.tail]] do its job.

Quick tip for copying a line with the vi texteditor:
vi /var/log/suricata/eve.json
Hit shift + g (go to the bottom line)
Hit yy (to copy line)
Hit p (insert the copied line after the current line on which the cursor is resting or type P to insert the copied line before the current line)

Shift + . (colon)
Then wq (write quit)


Data should be flowing to your bucket. From here you could add a cell in your dashboard with the following snippet:

Code: [Select]

from(bucket: "opnsense")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "suricata")
  |> group(columns: ["_time"])
  |> last()
  |> group()
  |> keep(columns: ["_time", "src_ip", "src_port","event_type","dest_ip", "dest_port"])

Change from "Graph" to "Table".
 
PS: If you do not see anything beiing populated in the table, remember to check your timerange "Past 7d, Past 30d" etc.


HOWEVER.

How can we avoid adding telegraf to the wheel group? I have not yet restartet the firewall, so I do not know if this is persistent.

And how on earth has this even worked, before?

[koushun]

SOLVED x2 ?? Here is another method of getting Telegraf plugin to send stats from Suricata.

As this plugin, [[inputs.suricata]], has been newly updated to incorporate events type "Alerts" (https://github.com/influxdata/telegraf/tree/master/plugins/inputs/suricata) , we will just have to wait to use this plugin until Telegraf is updated / OPNsense has a new version of it (for alerts, which is what I want an overview of the most). IF MY UNDERSTANDING IS CORRECT.

Anyways, to get suricata stats sent to InfluxDBv2 with Telegraf, you will have to increase the localhost buffer space to a much higher value than the suggested 16384, from the link above.

Go to System > Settings > Tunables. Click +Add on the top right corner and add the 2 tunables:

* Tunable: net.local.stream.recvspace
* Description: Increase the localhost buffer space
* Value: 24576


* Tunable: net.local.stream.sendspace
* Description: Increase the localhost buffer space
* Value: 24576

Apply and reboot, then do the following:

1. SSH into OPNsense box. As root, service suricata stop & service telegraf stop.

2. Edit /usr/local/etc/telegraf.conf and add:

Code: [Select]

[[inputs.suricata]]
  ## Data sink for Suricata stats log.
  # This is expected to be a filename of a
  # unix socket to be created for listening.
  source = "/tmp/suricata-stats.sock"

  # Delimiter for flattening field keys, e.g. subitem "alert" of "detect"
  # becomes "detect_alert" when delimiter is "_".
  delimiter = "_"

  # Detect alert logs --> this does not work with this version of Telegraf installer on OPNsense
  # alerts = false

Using service telegraf start, a UNIX-socket has been created in the /tmp directory:

Code: [Select]

file /tmp/suricata-stats.sock
/tmp/suricata-stats.sock: socket

https://forum.suricata.io/t/suricata-fails-to-create-socket/170/2 :

suricata doesn’t create the unix socket, but instead if connects to an existing one.

Suricata is started as root, so there are no permission issues for the connection from suricata to this socket.

3. Add a new eve-log output in /usr/local/etc/suricata/suricata.yaml:

Code: [Select]

outputs:
  - eve-log:
      enabled: yes
      filetype: unix_stream
      filename: /tmp/suricata-stats.sock
      types:
        - stats:
           threads: yes
        - alert:
             # packet: yes              # enable dumping of packet (without stream segments)
             # metadata: no             # enable inclusion of app layer metadata with alert. Default yes
             # http-body: yes           # Requires metadata; enable dumping of http body in Base64
             # http-body-printable: yes # Requires metadata; enable dumping of http body in printable format

             # Enable the logging of tagged packets for rules using the
             # "tag" keyword.
             tagged-packets: yes

             http: yes
             tls: yes


Issue service suricata start.

With sockstat | grep surica you can see suricata has connected to this UNIX-socket. Now you will just have to wait and stats will begin to flow in to your InfluxDBv2 via Telegraf.

Caveats:
- This will not survive a reboot
- Alerts are not logged (yet), even though they are sent to the UNIX-socket (I want to have the same overview as the "Alerts" tab under Intrusion Detection - they are not picked up by the [[inputs.suricata]] plugin
- You cannot disable / enable telegraf / suricata from the GUI anymore as it will reset the configuration files
- I have not yet concidered the implications of using the world readable /tmp directory
- The localhost buffer space might even have to be higher;

I had captured the output from sudo -u telegraf nc -U -l /tmp/suricata-stats.sock and searched for "timestamp" throughout the document, it looked as though messages from Suricata were truncated.

The start of the output:

Code: [Select]

{"timestamp":"2021-07-29T16:26:14.841116+0200","event_type":"stat (…)


throughout the output I had
,"invalid{"timestamp":"2021-0 (…)
0,"pseud{"timestam(…)
mcap_drop":0,"ps{"timestamp (…)
emcap_drop"{"timestamp":"2021-07-2

{"timestamp" is a new eve-log entry. Suricata did not have enough space to send the whole event for telegraf to process - it should end with }}}} before a new {"timestamp".

As I started suricata with "type : unix_stream" which made it write to a file instead of a socket (skipping the buffer space completely), I copied the whole first event in to a textfile. This file was about 22000 bytes big. As this was just a "stats" event, I do not know how much more data an "alert" event would add to the bufferspace.

As alerts are not something what this plugin, [[inputs.suricata]], processes just yet, I do not know if 24576 is sufficient.

There are no errors in telegraf (/var/log/telegraf.log) depicting this.

[koushun]

Alerts being sent to the UNIX-socket /tmp/suricata-stats.sock from suricata might not be as relevant to the buffer size as I thought, they are in fact fairly small events.

This is an example of what you'll receive in the UNIX-socket from suricata https://github.com/influxdata/telegraf/blob/master/plugins/inputs/suricata/testdata/test3.json

What I discovered, however, on a firewall with 10+ interfaces - is that the stats event is correspondingly bigger.

A tip is to make suricata write to a file, then wait a bit and then copy a whole event from {"timestamp": (...) to the end, "}}}}}" to a file, and check the size of the file- and then adjust the buffer space accordingly. 

So to make the [[inputs.suricata]] plugin pick up stats events on this particular firewall, I had to increase the bufferspace to 131072 to avoid truncation. I do not know the implications of this, if there are any.

And I do not know yet what to do with all these stats events from suricata

[mimugmail]

https://github.com/opnsense/plugins/issues/2239#issuecomment-912531932