OPNsense Forum

Hi all,
I've been using other FWs for a while and am switching to OPNsense! I'm getting myself up to speed with OPNsense to prepare for a small but mission critical distributed environment. If my project is successful I plan to get some support for OPNsense project from the company.

I've been looking at this issue for a week now. The Telegraf plugin in OPNsense works great and could send all selected input sources to designated server (e.g. influxdb). I'm trying to see if Suricata logs/alerts could also be sent.

On influxdb site there are some descriptions of hooking up Suricata output to Telegraf using a unix_stream socket. I've edited the telegraf.conf and suricata.yaml files and restart the services. Didn't seem to work.

Do I need to install the suricata.go file somewhere on the OPNsense to make the config work? I haven't been able to find and .go files on the OPNsense.

Thanks for help!


My steps of changes:
1) Add config lines to /usr/local/etc/suricata/suricata.yaml:
- eve-log:
    enabled: yes
    type: unix_stream
    filename: /var/run/suricata-stats.sock
    types:
      - stats:
         threads: yes
2) restart suricata by #service suricata restart. Tested the socket by cat /var/run/suricata-stats.sock. A ton of text data looking like suricata output.
3) add lines to /usr/local/etc/telegraf.conf:
[[inputs.suricata]]
  source = "/var/run/suricata-stats.sock"
  delimiter = "_"
4) restart telegraf by #service telegraf restart. Not seeing new measurements (assuming new suricata dataset will create new measurements) being created in influxdb.

General instruction from influxdb site (note there are two minor mistakes. eve-log section of suricata should use "type" instead of "filetype". inputs.suricata statement in suricata.yaml was "input.suricata" without s)
https://github.com/influxdata/telegraf/tree/master/plugins/inputs/suricata
(https://github.com/influxdata/telegraf/tree/master/plugins/inputs/suricata)

===============Software versions=========
suricata-4.1.6
Name : suricata
Version : 4.1.6
Installed on : Wed Jan 29 16:07:34 2020 EST
Origin : security/suricata
Architecture : FreeBSD:11:amd64
Prefix : /usr/local
Categories : security
Licenses : GPLv2
Maintainer : franco@opnsense.org
Comment : High Performance Network IDS, IPS and Security Monitoring engine
Options :slight_smile:

==============================================
telegraf-1.14.1
Name : telegraf
Version : 1.14.1
Installed on : Fri Apr 24 11:46:09 2020 EDT
Origin : net-mgmt/telegraf
Architecture : FreeBSD:11:amd64
Prefix : /usr/local
Categories : net-mgmt
Licenses : MIT
Maintainer : girgen@FreeBSD.org
Comment : Time-series data collection
Options :
PIE : on
RELRO : on
Annotations :
FreeBSD_version: 1102000
repo_type : binary
repository : OPNsense
Flat size : 63.4MiB

====================================
FreeBSD 11.2-RELEASE-p16-HBSD FreeBSD 11.2-RELEASE-p16-HBSD fc65add89c3(stable/20.1) amd64

I am having exactly the same issue - were you able to fix this?

Hello, i try to configure it but i have nothing in /var/run/suricata-stats.sock :/ no file at all.

• OPNsense 21.1.8_1-amd64
• InfluxDB 2.0.7
• Telegraf 1.19.0

I am also unable to get this to work. The config file (suricata.yaml - which resides in /usr/local/etc/suricata) is being overwritten by disabling / enabling the service, thus it is not a permanent solution to manually add / alter the configuration, stated here: https://www.influxdata.com/blog/network-security-monitoring-with-suricata-and-telegraf/ / earlier in this post.

A reload does not seem to do anything / apply the "new" configuration (no sock file is created). Disabling / enabling the service overwrites the config file.

Anyways, the information is being written to a file called eve.json in /var/log/suricata/ - but I do not yet know how to get this information over to InfluxDB v2.0

https://github.com/opnsense/core/issues/3401


Is this a bug? Should there be more options when enabling eve syslog output?

I'll try to look more into it using these resources

• https://www.influxdata.com/blog/json-to-influxdb-with-telegraf-and-starlark/
• https://community.influxdata.com/t/suricata-eve-json-input-file/16323

Somehow the JSON file should be converted to "Line Protocol". But, how is this eve.json file going to get pushed to InfluxDB?

Checking "Intrusion Dectection Alerts" in the Telegraf plugin does not seem to do anything :)

I wanted to install Grafana to create a cool dasbhoard, but I was hoping I could just use InfluxDB and create my dashboards there when I saw InfluxDB also supported Dashboards. I have all my other data that I want, but I also want IDS/IPS information.

Hmmm. Someone smarter than me, please help.

Edit: I had misunderstood what the Telegraf config section in InfluxDB tas supposed to do. It just "helps" you create a telegraf.conf. .. Anyways;


Looks like custom config should be added to the file custom.yaml (Edit: This does not survive a reboot)

root@opnsense:/usr/local/etc/suricata # service suricata restart
suricata not running? (check /var/run/suricata.pid).
Starting suricata.
23/7/2021 -- 21:29:25 - <Info> - Including configuration file installed_rules.yaml.
23/7/2021 -- 21:29:25 - <Info> - Configuration node 'rule-files' redefined.
23/7/2021 -- 21:29:25 - <Info> - Including configuration file custom.yaml.


Adding this code to custom.yaml, the file suricatat-stats.sock started to populate data after a service suricata restart:

(Edit:  E! [telegraf] Error running agent: starting input inputs.suricata: listen unix /var/run/suricata-stats.sock: bind: permission denied)

root@opnsense:/usr/local/etc/suricata # vi custom.yaml
%YAML 1.1
---
# empty stub for custom modifications, add custom persistent config below

  - eve-log:
      enabled: yes
      type: unix_stream
      filename: /tmp/suricata-stats.sock
      types:
        - stats:
             threads: yes


But nothing is showing in InfluxDB.

However, digging a bit further, /usr/local/etc/telegraf.conf held this configuration on the very bottom;


[[inputs.tail]]
  data_format = "json"
  files = ["/var/log/suricata/eve.json"]
  name_override = "suricata"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
  json_string_fields = ["*"]


https://github.com/opnsense/plugins/issues/2475

Hm, I thought I was able to get data in to suricata-stats.sock, but know I do not know anymore. I might have been confused rergarding this socket and the file eve.json file. ..

Anyways, further progress - regarding [[inputs.suricata]]:
"Under FreeBSD it is necessary to increase the localhost buffer space to at least 16384, default is 8192 otherwise messages from Suricata are truncated as they exceed the default available buffer space, consequently no statistics are processed by the plugin."

I have increased both net.local.stream.recvspace and net.local.stream.sendspace through tunables. Applied, rebooted and the value is verified through syctl -a | grep net.local.stream

Stated in this post, https://github.com/influxdata/telegraf/issues/7843 it should work like this;
"it's not working as expected, the unix socket is created by telegraf, data is sent by suricata but nothing is sent from telegraf to influxdb"

Having this in /usr/local/etc/telegraf.conf;

[[inputs.suricata]]
source = "/tmp/suricata-stats.sock"
delimiter = "_"


Creates this file, /tmp/suricata-stats.sock. But the file is empty, nothing is showing using nc -U /tmp/suricata-stats.sock.

srwxr-xr-x  1 telegraf  wheel     0B Jul 26 20:29 /tmp/suricata-stats.sock


Could it be permissions? But Suricata is started by root, so it cannot be permission issues..

[[inputs.tail]] does not produce any metrics in my InfluxDBv2 instance..

https://docs.opnsense.org/troubleshooting/hardening.html

User/Group separation

Thanks mimugmail.

Basically, what you are saying - to be able to enable the [[inputs.suricata]] plugin, one would have to "untighten" security? This one?

security.bsd.unprivileged_read_msgbuf [1->0]
Unprivileged processes may read the kernel message buffer

Hmm. Is that a wise solution..

Now, then, if I only could get the eve.json file to work instead..

Adding telegraf to wheel is basically the same I guess.

Stop the services:


service suricata stop
service telegraf stop


Add the user telegraf to the group wheel and confirm with pw groupshow;

pw group mod wheel -m telegraf
pw groupshow wheel
wheel:*:0:root,telegraf


Adding this eve-log directly under outputs in suricata.yaml:

outputs:

  - eve-log:
      enabled: yes
      filetype: unix_stream
      filename: /tmp/suricata-stats.sock
      types:
        - stats:
           threads: yes


Adding this in /usr/local/etc/telegraf.conf:

[[inputs.suricata]]
  ## Data sink for Suricata stats log.
  # This is expected to be a filename of a
  # unix socket to be created for listening.
  source = "/tmp/suricata-stats.sock"

  # Delimiter for flattening field keys, e.g. subitem "alert" of "detect"
  # becomes "detect_alert" when delimiter is "_".
  delimiter = "_"


Starting telegraf with service telegraf start creates a file in the /tmp directory:

root@opnsense:/usr/local/etc/suricata # ls -alh /tmp/suricata-stats.sock
srwxr-xr-x  1 telegraf  wheel     0B Jul 26 23:24 /tmp/suricata-stats.sock


The wheel group does not have write access to this file?

Telegraf seems to start OK, besides this error; https://github.com/influxdata/telegraf/issues/9407

time="2021-07-26T23:24:20+02:00" level=error msg="failed to create cache directory. /.cache/snowflake, err: mkdir /.cache: permission denied. ignored\n" func="gosnowflake.(*defaultLogger).Errorf" file="log.go:120"
time="2021-07-26T23:24:20+02:00" level=error msg="failed to open. Ignored. open /.cache/snowflake/ocsp_response_cache.json: no such file or directory\n" func="gosnowflake.(*defaultLogger).Errorf" file="log.go:120"
2021-07-26T21:24:20Z I! Starting Telegraf 1.19.0


No errors when starting suricata:

root@opnsense:/usr/local/etc/suricata # service suricata start
Starting suricata.
26/7/2021 -- 23:28:33 - <Info> - Including configuration file installed_rules.yaml.
26/7/2021 -- 23:28:33 - <Info> - Configuration node 'rule-files' redefined.
26/7/2021 -- 23:28:33 - <Info> - Including configuration file custom.yaml.


Besides some warnings:

Jul 24 09:32:33 opnsense suricata[42065]: [100950] <Notice> -- This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.IE7.NoRef.NoCookie' is checked but not set. Checked in 2023671 and 4 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.http.binary' is checked but not set. Checked in 2023741 and 4 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.http.javaclient' is checked but not set. Checked in 2017557 and 1 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.JavaArchiveOrClass' is checked but not set. Checked in 2017772 and 1 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.pdf.in.http' is checked but not set. Checked in 2017790 and 0 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'is_proto_irc' is checked but not set. Checked in 2002029 and 4 other sigs
Jul 24 09:33:38 opnsense suricata[98999]: [101381] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] -CLOG;???


Might just be because I've enabled a lot of rules to try to get as many alerts as possible for fault finding this service, and some are not working properly (?)

But, hey - I might be getting somewhere. But I've done so many things now, that I do not know what would have triggered this error message in telegraf.log:

2021-07-26T18:30:37Z E! [inputs.suricata] Error in plugin: unexpected end of JSON input


As you can see, this was an error which showed up a bit earlier today.



Can some other folks that do not have sausage fingers like mine try to get this working? :)

The options to get IDS/IPS metrics to eve-log is in the Intrusion Detection service. The option to get Intrusion Detection Alerts as an Input in Telegraf is there. Where is the documentation? What have I done wrong to not get this working.

I've copied the file /var/log/suricata/eve.json to /tmp/eve.json and did a chown telegraf /tmp/eve.json

I've added this in my /usr/local/telegraf.conf file:

[[inputs.file]]
  files = ["/tmp/eve.json"]
  data_format = "json"
  json_time_key = "timestamp"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
  json_string_fields = ["*"]


Doing this, producer this error:

tail -f /var/log/telegraf/telegraf.log
2021-07-26T22:35:00Z E! [inputs.file] Error in plugin: invalid character '{' after top-level value


There is something "wrong" with the file produced by suricata? Or there is an option which should be defined under [[inputs.file]], which has not been answered before on the internet; https://community.grafana.com/t/suricata-eve-json-input-file/38061

This is a multi-lined JSON?

https://github.com/influxdata/docs.influxdata.com-ARCHIVE/issues/971 :

Quote
The above issue (multi-line json when using the tail plugin) is only an issue because the tail plugin reads and parses files one line at a time, hence the need for full and parseable (non-prettyprinted) data, regardless of the format.

Deleting all lines, except for 1 in /tmp/eve.json, which leaves me with this line

{"timestamp":"2021-07-26T00:35:09.879746+0200","flow_id":1555495003216570,"in_iface":"re0_vlan50","event_type":"anomaly","src_ip":"192.168.50.100","src_port":28967,"dest_ip":"13.44.15.16","dest_port":57818,"proto":"TCP","app_proto":"tls","anomaly":{"type":"applayer","event":"APPLAYER_DETECT_PROTOCOL_ONLY_ONE_DIRECTION","layer":"proto_detect"}}


And having this config in telegraf.conf;

[[inputs.file]]
  files = ["/tmp/eve.json"]
  data_format = "json"
  json_string_fields = ["*"]


Gives me a lot of metrics in InfluxDBv2!

So. Hmm. https://docs.influxdata.com/telegraf/v1.19/data_formats/input/json/

Having this in /usr/local/etc/telegraf.conf ;

[[inputs.file]]
  files = ["/tmp/eve.json"]
  data_format = "json"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
  name_override = "suricata"
  json_time_key = "timestamp"
  json_time_format = "2006-01-02T15:04:05-0700"


.. Enables me to create this cell in InfluxDBv2 Dashboard:

from(bucket: "opnsense")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "suricata")
  |> group(columns: ["_time"])
  |> last()
  |> group()
  |> keep(columns: ["_time", "src_ip", "src_port","event_type","dest_ip", "dest_port"])


Some kind of progress! Even though I don't fully understand what I am doing just yet :)

SOLVED (??):

Check "Enable eve syslog output" under Intrusion Detection > Administration, hit Apply.
Enabling this will create will create JSON entries in the file /var/log/suricata/eve.json whenever there is a new entry / Alert in Intrusion Detection.

To get Telegraf to grab these entries from the eve.json file and send it to your bucket in InfluxDBv2, you will have to enable "Intrusion Detection Alerts". Go to Services > Telegraf > Input and check
* Intrusion Detection Alerts

This will create an entry in your `/usr/local/etc/telegraf.conf` file on the very bottom:

[[inputs.tail]]
  data_format = "json"
  files = ["/var/log/suricata/eve.json"]
  name_override = "suricata"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
  json_string_fields = ["*"]


However, change this config (add json_time_key and json_time_format).

This is my [[inputs.tail]] configuration:

[[inputs.tail]]
  data_format = "json"
  files = ["/var/log/suricata/eve.json"]
  name_override = "suricata"
  tag_keys = ["event_type","src_ip","src_port","dest_ip","dest_port"]
#  json_string_fields = ["*"]
  json_time_key = "timestamp"
  json_time_format = "2006-01-02T15:04:05-0700"


Do not start / stop the Telegraf service from the GUI from now on. This will reset your telegraf.conf file.

However, as root, doing a service telegraf restart does not output any error messages, and everything looks fine. See by yourself with tail -f /var/log/telegraf/telegraf.log.

Although it isn't. The telegraf user does not have access permissions to the eve.json file

To let the service / user telegraf to be able to read the /var/log/suricata/eve.json file, you can do the following.

Add the user telegraf to the group wheel and confirm with pw groupshow;

pw group mod wheel -m telegraf
pw groupshow wheel
wheel:*:0:root,telegraf


Change permissions on both the /var/log/suricata directory and the eve.json file:

chmod 750 /var/log/suricata
chmod 750 /var/log/suricata/eve.json


Now, restart the telegraf service as root: service telegraf restart

If you are not seeing anything flowing to your bucket, copy a line in the eve.json file, using vi, to the very bottom (to update the file and let [[inputs.tail]] do its job.

Quick tip for copying a line with the vi texteditor:
vi /var/log/suricata/eve.json
Hit shift + g (go to the bottom line)
Hit yy (to copy line)
Hit p (insert the copied line after the current line on which the cursor is resting or type P to insert the copied line before the current line)

Shift + . (colon)
Then wq (write quit)


Data should be flowing to your bucket. From here you could add a cell in your dashboard with the following snippet:

from(bucket: "opnsense")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "suricata")
  |> group(columns: ["_time"])
  |> last()
  |> group()
  |> keep(columns: ["_time", "src_ip", "src_port","event_type","dest_ip", "dest_port"])


Change from "Graph" to "Table".

PS: If you do not see anything beiing populated in the table, remember to check your timerange "Past 7d, Past 30d" etc.


HOWEVER.

How can we avoid adding telegraf to the wheel group? I have not yet restartet the firewall, so I do not know if this is persistent.

And how on earth has this even worked, before?

SOLVED x2 ?? Here is another method of getting Telegraf plugin to send stats from Suricata.

As this plugin, [[inputs.suricata]], has been newly updated to incorporate events type "Alerts" (https://github.com/influxdata/telegraf/tree/master/plugins/inputs/suricata) , we will just have to wait to use this plugin until Telegraf is updated / OPNsense has a new version of it (for alerts, which is what I want an overview of the most). IF MY UNDERSTANDING IS CORRECT.

Anyways, to get suricata stats sent to InfluxDBv2 with Telegraf, you will have to increase the localhost buffer space to a much higher value than the suggested 16384, from the link above.

Go to System > Settings > Tunables. Click +Add on the top right corner and add the 2 tunables:

* Tunable: net.local.stream.recvspace
* Description: Increase the localhost buffer space
* Value: 24576


* Tunable: net.local.stream.sendspace
* Description: Increase the localhost buffer space
* Value: 24576

Apply and reboot, then do the following:

1. SSH into OPNsense box. As root, service suricata stop & service telegraf stop.

2. Edit /usr/local/etc/telegraf.conf and add:

[[inputs.suricata]]
  ## Data sink for Suricata stats log.
  # This is expected to be a filename of a
  # unix socket to be created for listening.
  source = "/tmp/suricata-stats.sock"

  # Delimiter for flattening field keys, e.g. subitem "alert" of "detect"
  # becomes "detect_alert" when delimiter is "_".
  delimiter = "_"

  # Detect alert logs --> this does not work with this version of Telegraf installer on OPNsense
  # alerts = false


Using service telegraf start, a UNIX-socket has been created in the /tmp directory:

file /tmp/suricata-stats.sock
/tmp/suricata-stats.sock: socket


https://forum.suricata.io/t/suricata-fails-to-create-socket/170/2 :

Quotesuricata doesn't create the unix socket, but instead if connects to an existing one.

Suricata is started as root, so there are no permission issues for the connection from suricata to this socket.

3. Add a new eve-log output in /usr/local/etc/suricata/suricata.yaml:

outputs:
  - eve-log:
      enabled: yes
      filetype: unix_stream
      filename: /tmp/suricata-stats.sock
      types:
        - stats:
           threads: yes
        - alert:
             # packet: yes              # enable dumping of packet (without stream segments)
             # metadata: no             # enable inclusion of app layer metadata with alert. Default yes
             # http-body: yes           # Requires metadata; enable dumping of http body in Base64
             # http-body-printable: yes # Requires metadata; enable dumping of http body in printable format

             # Enable the logging of tagged packets for rules using the
             # "tag" keyword.
             tagged-packets: yes

             http: yes
             tls: yes



Issue service suricata start.

With sockstat | grep surica you can see suricata has connected to this UNIX-socket. Now you will just have to wait and stats will begin to flow in to your InfluxDBv2 via Telegraf.

Caveats:
- This will not survive a reboot
- Alerts are not logged (yet), even though they are sent to the UNIX-socket (I want to have the same overview as the "Alerts" tab under Intrusion Detection - they are not picked up by the [[inputs.suricata]] plugin
- You cannot disable / enable telegraf / suricata from the GUI anymore as it will reset the configuration files
- I have not yet concidered the implications of using the world readable /tmp directory
- The localhost buffer space might even have to be higher;

I had captured the output from sudo -u telegraf nc -U -l /tmp/suricata-stats.sock and searched for "timestamp" throughout the document, it looked as though messages from Suricata were truncated.

The start of the output:

{"timestamp":"2021-07-29T16:26:14.841116+0200","event_type":"stat (...)



throughout the output I had
,"invalid{"timestamp":"2021-0 (...)
0,"pseud{"timestam(...)
mcap_drop":0,"ps{"timestamp (...)
emcap_drop"{"timestamp":"2021-07-2

{"timestamp" is a new eve-log entry. Suricata did not have enough space to send the whole event for telegraf to process - it should end with }}}} before a new {"timestamp".

As I started suricata with "type : unix_stream" which made it write to a file instead of a socket (skipping the buffer space completely), I copied the whole first event in to a textfile. This file was about 22000 bytes big. As this was just a "stats" event, I do not know how much more data an "alert" event would add to the bufferspace.

As alerts are not something what this plugin, [[inputs.suricata]], processes just yet, I do not know if 24576 is sufficient.

There are no errors in telegraf (/var/log/telegraf.log) depicting this.

Alerts being sent to the UNIX-socket /tmp/suricata-stats.sock from suricata might not be as relevant to the buffer size as I thought, they are in fact fairly small events.

This is an example of what you'll receive in the UNIX-socket from suricata https://github.com/influxdata/telegraf/blob/master/plugins/inputs/suricata/testdata/test3.json

What I discovered, however, on a firewall with 10+ interfaces - is that the stats event is correspondingly bigger.

A tip is to make suricata write to a file, then wait a bit and then copy a whole event from {"timestamp": (...) to the end, "}}}}}" to a file, and check the size of the file- and then adjust the buffer space accordingly. 

So to make the [[inputs.suricata]] plugin pick up stats events on this particular firewall, I had to increase the bufferspace to 131072 to avoid truncation. I do not know the implications of this, if there are any.

And I do not know yet what to do with all these stats events from suricata :)

https://github.com/opnsense/plugins/issues/2239#issuecomment-912531932