log standard.. or how to decipher logs

[r0ckky]

hello all,

Friday question here

i am exporting the logs into logstash and i need some help deciphering the log structure

for example:

Code: [Select]

<134>Nov 20 15:35:55 OPNsense.localdomain filterlog: 14,,,0,igb0,match,block,in,4,0x0,,119,17591,0,DF,6,tcp,52,<redacted IP >,<redacted IP >,63652,7680,0,S,3174183196,,64240,,mss;nop;wscale;nop;nop;sackOK
im ok right up to this point

Code: [Select]

,0,S,3174183196,,64240,,mss;nop;wscale;nop;nop;sackOK
is there some sort of guide or technical opensense doc that details what each of these fields represents ?
what the numbers represent what mss, nop etc mean in regards to the firewall log output?


i tried the logstash-filter-opnsensefilter (https://github.com/fabianfrz/opnsense-logstash-config) but it dosnt install cleanly on the latest logstash version, and whilst i dont have any errors on the logstash conf files, it refuses to utilise the logstash plugin which might be that its now out of date.

i built a logstash grok for a specific log event

Code: [Select]

%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{DATA:syslog_program} %{NUMBER:rulenr},,,%{WORD:rid},%{WORD:interface},%{WORD:reason},%{WORD:action},%{WORD:dir},%{WORD:version},%{WORD:tos},,%{NUMBER:ttl},%{NUMBER:id},%{NUMBER:offset},%{WORD:ipflags},%{NUMBER:protonumber},%{WORD:protocol},%{NUMBER:length},%{IP:src_ip},%{IP:dst_ip},%{WORD}=%{NUMBER:datalen}
this works against : 

Code: [Select]

<134>Nov 20 18:11:24 OPNsense.localdomain filterlog: 82,,,0,igb2,match,pass,out,4,0x0,,63,6038,0,DF,1,icmp,36,<redacted IP >,<redacted IP >,datalength=16
which is a simple icmp event.. seperates into nice seperate fields, no parse failures... wonderful... but i have all sorts of data type flowing through the firewall so im wondering if anyone else is having or had similar issues and how you got around it.

Many thanks

R

[franco]

Hi there,

Take a look at: https://github.com/opnsense/ports/blob/master/opnsense/filterlog/files/description.txt


Cheers,
Franco

[r0ckky]

you sir are a fricken legend!!!!!!!!!

[fabian]

It is easy to parse using my plugin:

https://github.com/fabianfrz/logstash-filter-opnsensefilter

[r0ckky]

i really hoped it would be, but it has problems finding the correct gem on the rubygems site..

ill pull the ruby stuff from your git hub and try pushing it in manually

ive been creating my own conf for the opensense but id like to gives yours a go as it looks much cleaner...

do you know if your filter is going to be ok with opensense 20.7 ?