Graylog Grok Pattern for New OPNsense Filter and Suricata Log Format

[secdoc]

All for those that are planning and or dealing with updating to the latest release. There are changes to the log output for Filter and Suricata Logs (syslog output) as compared to the previous versions:

Old and New Logging Formats for FilterLog and Suricata

FilterLog:

Code: [Select]

Old Format: <134>Jul 16 06:36:52 xxxx.acme.tech filterlog[40156]: 85,,,9f96d956119c25145fc2ce221237f3a5,bridge0,match,pass,out,4,0x0,,64,63281,0,DF,17,udp,52,10.13.37.156,8.8.8.8,36444,53,32
New Format:

Code: [Select]

<134>1 2024-08-05T12:05:54+00:00 xxxx.acme.tech filterlog 54802 - [meta sequenceId="2763892"] 85,,,9f96d956119c25145fc2ce221237f3a5,bridge0,match,pass,out,4,0x0,,63,11771,0,DF,6,tcp,60,10.13.37.156,193.0.6.135,43802,43,0,S,1404651214,,64240,,mss;sackOK;TS;nop;wscale
Suricata: Old Format:

Code: [Select]

<173>Jul 26 13:17:32 xxxx.acme.tech suricata[48264]: [1:2017928:4] ET POLICY check.torproject.org IP lookup/Tor Usage check over TLS with SNI [Classification: Device Retrieving External IP Address Detected] [Priority: 2] {TCP} 192.168.200.69:1247 -> 116.202.120.181:443
New Format:

Code: [Select]

<173>1 2024-08-05T23:17:15+00:00 xxxx.acme.tech suricata 23296 - [meta sequenceId="1335381"] [1:2024364:4] ET SCAN Possible Nmap User-Agent Observed [Classification: Web Application Attack] [Priority: 1] {TCP} 192.168.2.141:56840 -> 192.168.88.2:80
Key Changes:

Date Format: The old format used a more condensed representation (e.g., "Jul 26 13:17:32"), while the new format adopts the ISO 8601 standard (e.g., "2024-08-05T23:17:15+00:00"), including timezone information.
Meta Sequence ID: The new format introduces a [meta sequenceId="XXXXXXX"] field, which can be useful for tracking log sequence and detecting missing logs.
Structure: The overall structure of the log entries has been modified to align with more standardized logging practices, potentially improving compatibility with log analysis tools.

While many already be aware, they may have noticed potential impacts to logging and SIEM solutions.  If you use Graylog, I have create a Github Repository that has Grok Patterns to deal with the updated formatting.

If you need a quick work around, here is the link:
https://github.com/secdoc/OPNsense-24.7-Graylog-Grok-Patterns/tree/main