OPNsense Forum

English Forums => Intrusion Detection and Prevention => Topic started by: nines on February 22, 2018, 09:48:36 pm

Title: Suricata Drop Log
Post by: nines on February 22, 2018, 09:48:36 pm
I'm currently working an implementing ELK stack as centralized logging solution. Unfortunately not all drop messages are beeing sent to logstash.
I tried to search the drop messages in /var/log/suricata.log and in /var/log/syslog but there are only the rules I can already see in logstash ...
The reason I know there are more is that the gui shows a bunch of more drops in the same time period. How can this be possible? Where can I find them and what (obviously different) format is used to send them to a syslog server?

Thanks for clarification
André
Title: Re: Suricata Drop Log
Post by: nines on February 25, 2018, 07:23:28 pm
I can concretize the behaviour a bit. All IPS drops/alerts are in /var/log/suricata/eve.json as expected, however this is not the log nor the content beeing passed to syslog.
The logs beeing passed to syslog obviously are /var/log/suricata.log and /var/log/syslog (I'm not quiet sure if both).

Can someone confirm this and explain the reason please
Title: Re: Suricata Drop Log
Post by: franco on February 27, 2018, 08:35:38 pm
Incidentially, there is a working syslog + drop log for 18.1.3's development version, but this won't merge until 18.1.4. Syslog was a bit on the weak side due to all the eve log handling.


Cheers,
Franco
Title: Re: Suricata Drop Log
Post by: nines on March 03, 2018, 12:12:03 pm
thank you so much for clarifying this to me. I've running a vm and could easily test the dev version.
Is it (and how) possible to savely switch to dev and back if needed?
Title: Re: Suricata Drop Log
Post by: franco on March 05, 2018, 08:20:16 am
Yep, it's possible to move to the dev version from the GUI under System: Firmware: Settings and a save + update. The code will be included there once 18.1.3 is out.

But it's the development version after all. Proceed with the correct expectations...

If you want to try it, go to the Log File and clear it once. Worst case it also requires a service restart.


Cheers,
Franco
Title: Re: Suricata Drop Log
Post by: nines on March 05, 2018, 08:37:33 pm
surprisingly (at least for me) 18.1.3. was released today. I can't find any changes regarding syslog and ips in the changelog so I asume its not fixed as of now?
Title: Re: Suricata Drop Log
Post by: franco on March 06, 2018, 07:47:12 am
What I said was:

1. Update to 18.1.3.
2. Go to System: Firmware: Settings and select "Development". Hit save and update.
3. Go to Services: Intrusion Detection: Log File. Clear the log file.
4. Go to Services: Intrusion Detection: Administration. Enable Syslog if not enabled, hit save.
5. Drops should now be logged in the syslog file.


Cheers,
Franco
Title: Re: Suricata Drop Log
Post by: nines on March 06, 2018, 01:11:31 pm
did exactly what you describe but that unfortunately didnt resolve the issue. Behaviour is exactly the same, the events from the gui alert tab are 1:1 like the eve.json file but syslog and suricata.log is missing many entries.
Title: Re: Suricata Drop Log
Post by: franco on March 06, 2018, 02:20:05 pm
Drops were not shown in syslog previously. Can you clarify "missing many entries" please?


Cheers,
Franco
Title: Re: Suricata Drop Log
Post by: nines on March 06, 2018, 02:25:37 pm
of course - see the timestamps in comparison

eve.json

Code: [Select]
{"timestamp":"2018-03-06T12:07:40.571052+0100","flow_id":28971542295690,"in_iface":"vmx0","event_type":"drop","src_ip":"192.168.254.250","src_port":57828,"dest_ip":"13.107.4.50","dest_port":80,"proto":"TCP","drop"
:{"len":305,"tos":2,"ttl":127,"ipid":20877,"tcpseq":1321765378,"tcpack":3104282439,"tcpwin":8212,"syn":false,"ack":true,"psh":true,"rst":false,"urg":false,"fin":false,"tcpres":0,"tcpurgp":0},"alert":{"action":"blo
cked","gid":1,"signature_id":2020573,"rev":2,"signature":"ET CURRENT_EVENTS INFO .exe download with no referer (noalert)","category":"Potentially Bad Traffic","severity":2}}
{"timestamp":"2018-03-06T12:07:40.841859+0100","flow_id":786608068284868,"in_iface":"vmx1+","event_type":"drop","src_ip":"192.168.200.253","src_port":45951,"dest_ip":"13.107.4.50","dest_port":80,"proto":"TCP","dro
p":{"len":383,"tos":0,"ttl":64,"ipid":0,"tcpseq":3535207936,"tcpack":641069320,"tcpwin":517,"syn":false,"ack":true,"psh":true,"rst":false,"urg":false,"fin":false,"tcpres":0,"tcpurgp":0},"alert":{"action":"blocked"
,"gid":1,"signature_id":2020573,"rev":2,"signature":"ET CURRENT_EVENTS INFO .exe download with no referer (noalert)","category":"Potentially Bad Traffic","severity":2}}
{"timestamp":"2018-03-06T12:07:41.274817+0100","flow_id":28971542295690,"in_iface":"vmx0","event_type":"drop","src_ip":"192.168.254.250","src_port":57828,"dest_ip":"13.107.4.50","dest_port":80,"proto":"TCP","drop"
:{"len":356,"tos":2,"ttl":127,"ipid":20879,"tcpseq":1321765643,"tcpack":3104283110,"tcpwin":8209,"syn":false,"ack":true,"psh":true,"rst":false,"urg":false,"fin":false,"tcpres":0,"tcpurgp":0},"alert":{"action":"blo
cked","gid":1,"signature_id":2020573,"rev":2,"signature":"ET CURRENT_EVENTS INFO .exe download with no referer (noalert)","category":"Potentially Bad Traffic","severity":2}}
{"timestamp":"2018-03-06T12:07:41.295349+0100","flow_id":786608068284868,"in_iface":"vmx1+","event_type":"drop","src_ip":"192.168.200.253","src_port":45951,"dest_ip":"13.107.4.50","dest_port":80,"proto":"TCP","dro
p":
{"len":434,"tos":0,"ttl":64,"ipid":0,"tcpseq":3535208267,"tcpack":641069863,"tcpwin":517,"syn":false,"ack":true,"psh":true,"rst":false,"urg":false,"fin":false,"tcpres":0,"tcpurgp":0},"alert":{"action":"blocked"
,"gid":1,"signature_id":2020573,"rev":2,"signature":"ET CURRENT_EVENTS INFO .exe download with no referer (noalert)","category":"Potentially Bad Traffic","severity":2}}
{"timestamp":"2018-03-06T12:18:42.785185+0100","flow_id":2400770460837,"in_iface":"vmx0+","event_type":"alert","src_ip":"87.78.182.200","src_port":80,"dest_ip":"192.168.254.6","dest_port":44013,"proto":"TCP","aler
t":{"action":"allowed","gid":1,"signature_id":2260002,"rev":1,"signature":"SURICATA Applayer Detect protocol only one direction","category":"Generic Protocol Command Decode","severity":3},"http":{"length":1448},"a
pp_proto":"http"}

suricata.log

Code: [Select]
Mar  6 12:01:06 OPNsense suricata[2522]: [100327] <Notice> -- all 4 packet processing threads, 4 management threads initialized, engine started.
Mar  6 12:18:42 OPNsense suricata[2522]: [1:2260002:1] SURICATA Applayer Detect protocol only one direction [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 87.78.182.200:80 -> 192.168.254.6:4
4013
Title: Re: Suricata Drop Log
Post by: franco on March 06, 2018, 02:54:06 pm
Did you clear the log file? I'm seeing:

Mar 6 13:40:35   suricata[95097]: [1:2210044:2] SURICATA STREAM Packet with invalid timestamp [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 23.57.24.16:443 -> 192.168.178.20:37470
Mar 6 13:40:02   suricata[95097]: [1:2210007:2] SURICATA STREAM 3way handshake SYNACK with wrong ack [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 17.252.27.246:443 -> 192.168.178.20:3538


Cheers,
Franco
Title: Re: Suricata Drop Log
Post by: nines on March 06, 2018, 03:13:31 pm
Yes I did. Why do you have doubts?
To be honest I cleared it multiple times but have tested the behavior some hours ago


Gesendet von iPhone mit Tapatalk
Title: Re: Suricata Drop Log
Post by: franco on March 06, 2018, 03:15:17 pm
I'm not seeing your install. I keep asking because that's the only way to make sure. :)
Title: Re: Suricata Drop Log
Post by: nines on March 06, 2018, 03:17:04 pm
Sorry, wasn't meant to be harsh.
What do you mean by "your install" exactly?

//edit: gui tells me
Versions OPNsense 18.7.a_146-amd64
FreeBSD 11.1-RELEASE-p6
LibreSSL 2.6.4

but uname -a says
FreeBSD OPNsense.unimatrix01.local 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6  6621d681e(stable/18.1)  amd64

is that something to worry about?


Again, appreciating your help!


Gesendet von iPhone mit Tapatalk
Title: Re: Suricata Drop Log
Post by: franco on March 07, 2018, 05:25:41 pm
Looks good from the version perspective. Do you see any drops in the syslog now? If yes I'm unsure what to check for next.


Cheers,
Franco
Title: Re: Suricata Drop Log
Post by: nines on March 07, 2018, 05:41:42 pm
Yes I have drops in syslog, but I have to point out that I already had drops before the update. I cant tell for sure if there are more or drops as of the version I'm running now but what I can tell for sure is that the content from eve.json and suricata.log is definetely not the same (in terms of the blocked rules beeing logged)

this is a example from eve.json not showing up in suricata.log (one of many)

{"timestamp":"2018-03-07T12:07:48.969919+0100","flow_id":610197949434437,"in_iface":"vmx0","event_type":"drop","src_ip":"192.168.254.250","src_port":49354,"dest_ip":"8.249.59.254","dest_port":80,"proto":"TCP","dro
p":{"len":287,"tos":2,"ttl":127,"ipid":17292,"tcpseq":1736993020,"tcpack":1537099686,"tcpwin":8212,"syn":false,"ack":true,"psh":true,"rst":false,"urg":false,"fin":false,"tcpres":0,"tcpurgp":0},"alert":{"action":"b
locked","gid":1,"signature_id":2020573,"rev":2,"signature":"ET CURRENT_EVENTS INFO .exe download with no referer (noalert)","category":"Potentially Bad Traffic","severity":2}}

I have no idea why some logs are beeing sent to syslog and others are not. My personal feeling is that all the rules which are finally allowed but logged are visible in suricata.log but many of the actual drop rules are not.

Do you need additional information/output/files/tests. I surely can provide it, but I'm unsure what exactly would help.

The reason this is important for me is that I'm trying to send the IPS drop log to a central logstash instance for visualization und aggregation and as you can imagine that makes only sense if the logs beeing sent are complete :)

Thanks so far!
André
Title: Re: Suricata Drop Log
Post by: trigger_hippie on March 27, 2018, 01:20:24 am
Some events are not being pushed to syslog from eve.json. Could not find out why, but found a solution to push everything from eve.json to elasticsearch (as i see, you are using it as well).

Install filebeat on opnsense host and change filebeat.yml to point to logstash using this guide:
https://extelligenceblog.it/2017/07/11/elastic-stack-suricata-idps-and-pfsense-firewall-part-1/
 (https://extelligenceblog.it/2017/07/11/elastic-stack-suricata-idps-and-pfsense-firewall-part-1/)

Configure logstash:
beats {
    type => "Suricata"
    port => 5044
    codec => json
  }
filter {
         if [type] =~ /^Suricata/ {
                mutate {
                  add_tag => ["Suricata"]
                  remove_tag => ["beats_input_codec_json_applied"]
                  replace => {"service" => "Suricata"}
                }
        }
}
output {
if [type] == "Suricata" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "suricata-%{+YYYY.MM.dd}"
    }
  }
}

Hope it helps.
Cheers
Title: Re: Suricata Drop Log
Post by: nines on July 29, 2018, 02:42:18 pm
Some events are not being pushed to syslog from eve.json. Could not find out why, but found a solution to push everything from eve.json to elasticsearch (as i see, you are using it as well).

Install filebeat on opnsense host and change filebeat.yml to point to logstash using this guide:
https://extelligenceblog.it/2017/07/11/elastic-stack-suricata-idps-and-pfsense-firewall-part-1/
 (https://extelligenceblog.it/2017/07/11/elastic-stack-suricata-idps-and-pfsense-firewall-part-1/)

Configure logstash:
beats {
    type => "Suricata"
    port => 5044
    codec => json
  }
filter {
         if [type] =~ /^Suricata/ {
                mutate {
                  add_tag => ["Suricata"]
                  remove_tag => ["beats_input_codec_json_applied"]
                  replace => {"service" => "Suricata"}
                }
        }
}
output {
if [type] == "Suricata" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "suricata-%{+YYYY.MM.dd}"
    }
  }
}

Hope it helps.
Cheers

did you manage to configure autostart on boot of filebeat?
Title: Re: Suricata Drop Log
Post by: trigger_hippie on July 30, 2018, 01:01:24 pm
No, i did not manage to configure it to start automatically...
Title: Re: Suricata Drop Log
Post by: xmichielx on October 12, 2018, 07:05:57 pm
Also created https://github.com/opnsense/core/issues/2809 for this issue not sure if it is a bug or a works as expected.