Unable to get Telegraf plugin to send IPS data (suricata)

[md5]

Hi all,
I've been using other FWs for a while and am switching to OPNsense! I'm getting myself up to speed with OPNsense to prepare for a small but mission critical distributed environment. If my project is successful I plan to get some support for OPNsense project from the company.

I've been looking at this issue for a week now. The Telegraf plugin in OPNsense works great and could send all selected input sources to designated server (e.g. influxdb). I'm trying to see if Suricata logs/alerts could also be sent.

On influxdb site there are some descriptions of hooking up Suricata output to Telegraf using a unix_stream socket. I've edited the telegraf.conf and suricata.yaml files and restart the services. Didn't seem to work.

Do I need to install the suricata.go file somewhere on the OPNsense to make the config work? I haven't been able to find and .go files on the OPNsense.

Thanks for help!


My steps of changes:
1) Add config lines to /usr/local/etc/suricata/suricata.yaml:
- eve-log:
    enabled: yes
    type: unix_stream
    filename: /var/run/suricata-stats.sock
    types:
      - stats:
         threads: yes
2) restart suricata by #service suricata restart. Tested the socket by cat /var/run/suricata-stats.sock. A ton of text data looking like suricata output.
3) add lines to /usr/local/etc/telegraf.conf:
[[inputs.suricata]]
  source = "/var/run/suricata-stats.sock"
  delimiter = "_"
4) restart telegraf by #service telegraf restart. Not seeing new measurements (assuming new suricata dataset will create new measurements) being created in influxdb.

General instruction from influxdb site (note there are two minor mistakes. eve-log section of suricata should use "type" instead of "filetype". inputs.suricata statement in suricata.yaml was "input.suricata" without s)
https://github.com/influxdata/telegraf/tree/master/plugins/inputs/suricata


===============Software versions=========
suricata-4.1.6
Name : suricata
Version : 4.1.6
Installed on : Wed Jan 29 16:07:34 2020 EST
Origin : security/suricata
Architecture : FreeBSD:11:amd64
Prefix : /usr/local
Categories : security
Licenses : GPLv2
Maintainer : franco@opnsense.org
Comment : High Performance Network IDS, IPS and Security Monitoring engine
Options :slight_smile:

==============================================
telegraf-1.14.1
Name : telegraf
Version : 1.14.1
Installed on : Fri Apr 24 11:46:09 2020 EDT
Origin : net-mgmt/telegraf
Architecture : FreeBSD:11:amd64
Prefix : /usr/local
Categories : net-mgmt
Licenses : MIT
Maintainer : girgen@FreeBSD.org
Comment : Time-series data collection
Options :
PIE : on
RELRO : on
Annotations :
FreeBSD_version: 1102000
repo_type : binary
repository : OPNsense
Flat size : 63.4MiB

====================================
FreeBSD 11.2-RELEASE-p16-HBSD FreeBSD 11.2-RELEASE-p16-HBSD fc65add89c3(stable/20.1) amd64

[rangojava]

I am having exactly the same issue - were you able to fix this?

[Georges]

Hello, i try to configure it but i have nothing in /var/run/suricata-stats.sock :/ no file at all.

[koushun]

• OPNsense 21.1.8_1-amd64
• InfluxDB 2.0.7
• Telegraf 1.19.0

I am also unable to get this to work. The config file (suricata.yaml - which resides in /usr/local/etc/suricata) is being overwritten by disabling / enabling the service, thus it is not a permanent solution to manually add / alter the configuration, stated here: https://www.influxdata.com/blog/network-security-monitoring-with-suricata-and-telegraf/ / earlier in this post.

A reload does not seem to do anything / apply the "new" configuration (no sock file is created). Disabling / enabling the service overwrites the config file.

Anyways, the information is being written to a file called eve.json in /var/log/suricata/ - but I do not yet know how to get this information over to InfluxDB v2.0

https://github.com/opnsense/core/issues/3401


Is this a bug? Should there be more options when enabling eve syslog output?

[koushun]

I'll try to look more into it using these resources

• https://www.influxdata.com/blog/json-to-influxdb-with-telegraf-and-starlark/
• https://community.influxdata.com/t/suricata-eve-json-input-file/16323

Somehow the JSON file should be converted to "Line Protocol". But, how is this eve.json file going to get pushed to InfluxDB?

Checking "Intrusion Dectection Alerts" in the Telegraf plugin does not seem to do anything :)

I wanted to install Grafana to create a cool dasbhoard, but I was hoping I could just use InfluxDB and create my dashboards there when I saw InfluxDB also supported Dashboards. I have all my other data that I want, but I also want IDS/IPS information.

Hmmm. Someone smarter than me, please help.

[koushun]

Edit: I had misunderstood what the Telegraf config section in InfluxDB tas supposed to do. It just "helps" you create a telegraf.conf. .. Anyways;


Looks like custom config should be added to the file custom.yaml (Edit: This does not survive a reboot)

Code Select Expand

root@opnsense:/usr/local/etc/suricata # service suricata restart
suricata not running? (check /var/run/suricata.pid).
Starting suricata.
23/7/2021 -- 21:29:25 - <Info> - Including configuration file installed_rules.yaml.
23/7/2021 -- 21:29:25 - <Info> - Configuration node 'rule-files' redefined.
23/7/2021 -- 21:29:25 - <Info> - Including configuration file custom.yaml.


Adding this code to custom.yaml, the file suricatat-stats.sock started to populate data after a service suricata restart:

(Edit:  E! [telegraf] Error running agent: starting input inputs.suricata: listen unix /var/run/suricata-stats.sock: bind: permission denied)

Code Select Expand


root@opnsense:/usr/local/etc/suricata # vi custom.yaml
%YAML 1.1
---
# empty stub for custom modifications, add custom persistent config below

  - eve-log:
      enabled: yes
      type: unix_stream
      filename: /tmp/suricata-stats.sock
      types:
        - stats:
             threads: yes


But nothing is showing in InfluxDB.

However, digging a bit further, /usr/local/etc/telegraf.conf held this configuration on the very bottom;

Code Select Expand



[[inputs.tail]]
  data_format = "json"
  files = ["/var/log/suricata/eve.json"]
  name_override = "suricata"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
  json_string_fields = ["*"]


[koushun]

https://github.com/opnsense/plugins/issues/2475

[koushun]

Hm, I thought I was able to get data in to suricata-stats.sock, but know I do not know anymore. I might have been confused rergarding this socket and the file eve.json file. ..

Anyways, further progress - regarding [[inputs.suricata]]:
"Under FreeBSD it is necessary to increase the localhost buffer space to at least 16384, default is 8192 otherwise messages from Suricata are truncated as they exceed the default available buffer space, consequently no statistics are processed by the plugin."

I have increased both net.local.stream.recvspace and net.local.stream.sendspace through tunables. Applied, rebooted and the value is verified through syctl -a | grep net.local.stream

Stated in this post, https://github.com/influxdata/telegraf/issues/7843 it should work like this;
"it's not working as expected, the unix socket is created by telegraf, data is sent by suricata but nothing is sent from telegraf to influxdb"

Having this in /usr/local/etc/telegraf.conf;

Code Select Expand


[[inputs.suricata]]
source = "/tmp/suricata-stats.sock"
delimiter = "_"


Creates this file, /tmp/suricata-stats.sock. But the file is empty, nothing is showing using nc -U /tmp/suricata-stats.sock.

Code Select Expand


srwxr-xr-x  1 telegraf  wheel     0B Jul 26 20:29 /tmp/suricata-stats.sock


Could it be permissions? But Suricata is started by root, so it cannot be permission issues..

[[inputs.tail]] does not produce any metrics in my InfluxDBv2 instance..

[mimugmail]

https://docs.opnsense.org/troubleshooting/hardening.html

User/Group separation

[koushun]

Thanks mimugmail.

Basically, what you are saying - to be able to enable the [[inputs.suricata]] plugin, one would have to "untighten" security? This one?

security.bsd.unprivileged_read_msgbuf [1->0]
Unprivileged processes may read the kernel message buffer

Hmm. Is that a wise solution..

Now, then, if I only could get the eve.json file to work instead..

[mimugmail]

Adding telegraf to wheel is basically the same I guess.

[koushun]

Stop the services:

Code Select Expand


service suricata stop
service telegraf stop


Add the user telegraf to the group wheel and confirm with pw groupshow;

Code Select Expand


pw group mod wheel -m telegraf
pw groupshow wheel
wheel:*:0:root,telegraf


Adding this eve-log directly under outputs in suricata.yaml:

Code Select Expand


outputs:

  - eve-log:
      enabled: yes
      filetype: unix_stream
      filename: /tmp/suricata-stats.sock
      types:
        - stats:
           threads: yes


Adding this in /usr/local/etc/telegraf.conf:

Code Select Expand


[[inputs.suricata]]
  ## Data sink for Suricata stats log.
  # This is expected to be a filename of a
  # unix socket to be created for listening.
  source = "/tmp/suricata-stats.sock"

  # Delimiter for flattening field keys, e.g. subitem "alert" of "detect"
  # becomes "detect_alert" when delimiter is "_".
  delimiter = "_"


Starting telegraf with service telegraf start creates a file in the /tmp directory:

Code Select Expand


root@opnsense:/usr/local/etc/suricata # ls -alh /tmp/suricata-stats.sock
srwxr-xr-x  1 telegraf  wheel     0B Jul 26 23:24 /tmp/suricata-stats.sock


The wheel group does not have write access to this file?

Telegraf seems to start OK, besides this error; https://github.com/influxdata/telegraf/issues/9407

Code Select Expand


time="2021-07-26T23:24:20+02:00" level=error msg="failed to create cache directory. /.cache/snowflake, err: mkdir /.cache: permission denied. ignored\n" func="gosnowflake.(*defaultLogger).Errorf" file="log.go:120"
time="2021-07-26T23:24:20+02:00" level=error msg="failed to open. Ignored. open /.cache/snowflake/ocsp_response_cache.json: no such file or directory\n" func="gosnowflake.(*defaultLogger).Errorf" file="log.go:120"
2021-07-26T21:24:20Z I! Starting Telegraf 1.19.0


No errors when starting suricata:

Code Select Expand


root@opnsense:/usr/local/etc/suricata # service suricata start
Starting suricata.
26/7/2021 -- 23:28:33 - <Info> - Including configuration file installed_rules.yaml.
26/7/2021 -- 23:28:33 - <Info> - Configuration node 'rule-files' redefined.
26/7/2021 -- 23:28:33 - <Info> - Including configuration file custom.yaml.


Besides some warnings:

Code Select Expand


Jul 24 09:32:33 opnsense suricata[42065]: [100950] <Notice> -- This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.IE7.NoRef.NoCookie' is checked but not set. Checked in 2023671 and 4 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.http.binary' is checked but not set. Checked in 2023741 and 4 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.http.javaclient' is checked but not set. Checked in 2017557 and 1 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.JavaArchiveOrClass' is checked but not set. Checked in 2017772 and 1 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.pdf.in.http' is checked but not set. Checked in 2017790 and 0 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'is_proto_irc' is checked but not set. Checked in 2002029 and 4 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] -CLOG;???


Might just be because I've enabled a lot of rules to try to get as many alerts as possible for fault finding this service, and some are not working properly (?)

But, hey - I might be getting somewhere. But I've done so many things now, that I do not know what would have triggered this error message in telegraf.log:

Code Select Expand


2021-07-26T18:30:37Z E! [inputs.suricata] Error in plugin: unexpected end of JSON input


As you can see, this was an error which showed up a bit earlier today.



Can some other folks that do not have sausage fingers like mine try to get this working? :)

The options to get IDS/IPS metrics to eve-log is in the Intrusion Detection service. The option to get Intrusion Detection Alerts as an Input in Telegraf is there. Where is the documentation? What have I done wrong to not get this working.

[koushun]

I've copied the file /var/log/suricata/eve.json to /tmp/eve.json and did a chown telegraf /tmp/eve.json

I've added this in my /usr/local/telegraf.conf file:

Code Select Expand


[[inputs.file]]
  files = ["/tmp/eve.json"]
  data_format = "json"
  json_time_key = "timestamp"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
  json_string_fields = ["*"]


Doing this, producer this error:

Code Select Expand


tail -f /var/log/telegraf/telegraf.log
2021-07-26T22:35:00Z E! [inputs.file] Error in plugin: invalid character '{' after top-level value


There is something "wrong" with the file produced by suricata? Or there is an option which should be defined under [[inputs.file]], which has not been answered before on the internet; https://community.grafana.com/t/suricata-eve-json-input-file/38061

This is a multi-lined JSON?

https://github.com/influxdata/docs.influxdata.com-ARCHIVE/issues/971 :

The above issue (multi-line json when using the tail plugin) is only an issue because the tail plugin reads and parses files one line at a time, hence the need for full and parseable (non-prettyprinted) data, regardless of the format.

[koushun]

Deleting all lines, except for 1 in /tmp/eve.json, which leaves me with this line

Code Select Expand


{"timestamp":"2021-07-26T00:35:09.879746+0200","flow_id":1555495003216570,"in_iface":"re0_vlan50","event_type":"anomaly","src_ip":"192.168.50.100","src_port":28967,"dest_ip":"13.44.15.16","dest_port":57818,"proto":"TCP","app_proto":"tls","anomaly":{"type":"applayer","event":"APPLAYER_DETECT_PROTOCOL_ONLY_ONE_DIRECTION","layer":"proto_detect"}}


And having this config in telegraf.conf;

Code Select Expand


[[inputs.file]]
  files = ["/tmp/eve.json"]
  data_format = "json"
  json_string_fields = ["*"]


Gives me a lot of metrics in InfluxDBv2!

So. Hmm. https://docs.influxdata.com/telegraf/v1.19/data_formats/input/json/

[koushun]

Having this in /usr/local/etc/telegraf.conf ;

Code Select Expand


[[inputs.file]]
  files = ["/tmp/eve.json"]
  data_format = "json"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
  name_override = "suricata"
  json_time_key = "timestamp"
  json_time_format = "2006-01-02T15:04:05-0700"


.. Enables me to create this cell in InfluxDBv2 Dashboard:

Code Select Expand


from(bucket: "opnsense")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "suricata")
  |> group(columns: ["_time"])
  |> last()
  |> group()
  |> keep(columns: ["_time", "src_ip", "src_port","event_type","dest_ip", "dest_port"])


Some kind of progress! Even though I don't fully understand what I am doing just yet :)