logging format

[kamaradski]

Hi all,

I'm a long-time user of the *sense products dating all the way back to a dusty m0nowall server in my basement at the time. And I have recently switched to opnsense which is a large breeze of fresh air I must say, if not a storm of fresh air:).

Anyway, with the opnsense fork, I think we now have an environment where we can maybe more openly discuss ideas for the future and are no longer stuck with the roadmap vision of only a few. And this is why i think I shall come out of hiding and make my first post.

One thing I always found to be a large obstacle in the usage of pfsense was that the logs have their very specific native format. This in itself is fine since they take little space on the embedded servers where most of these instances run, i would not want to change that in any way.

However, it would be great that if the log-shipper is used to ship these logs to a logsever of sorts, we would have the option to send these logs in JSON format. Basically enriching the logs in a field:value format that can be parsed by the majority of log-servers out there. I have tried and failed so many times to make the logs useful in native ELK or Nagios logserver, etc...

As a family man and having a busy job that takes way more hours of my life than what I would like, I just don't have the time or energy to write complicated grok-filters or something. There are a few solutions available on the internet but these often require special plugins that one might not be able to install in the solution one is running, or they are so outdated that they stopped working years ago.

I would like to kickstart a discussion on this topic by opening this thread and maybe we can find a good solution that will benefit all of us that use different types of log-servers out there. Or maybe there is already a good solution and I have just overlooked it.

Opinions and ideas are very welcome.

[vermontfarmer]

I would LOVE JSON log format in OPNSense.  I keep checking in every now and then.  I REALLY do not want to do a customer parser......  JSON is self-parsing.

[badsmoke]

after a long search i still haven't found a reasonable way to output the logs as json, or have i missed something?

well anyway, i have now built a parser which i think does the job very well, feel free to try it out


https://github.com/badsmoke/LogSense2Loki