Increasing firewall log size hangs GUI

[rabievdm]

Hi,

Just checking on sizes...
I noticed that my logs rotate to quickly with default settings so I increased them from the default 512K (empty) to 200MB (209715200), saved the setting and reset the logs.
Everything appeared ok, but when I tried viewing the logs via the WEBGUI it would not return any results in the live view and then trying any other web interaction would fail and the web server would be unresponsive which required a restart of the services. I tried reducing the size, but had to bring the logs way down to 10MB for it to remain stable.

So 2 questions:
1) Anyone else seen this issue? (I'm on 19.7.7 and 19.7.
2) Anyone else increasing log file sizes? The defaults seems way too little and there doesn't seem to be an option to rotate logs. What do others do to maintain log history, send to remote?

Regards

[fabian]

From your message I expect that you mean the firewall log. For that case I would recommend you to use the ELK stack.

To start, use my config file for logstash:
https://github.com/fabianfrz/opnsense-logstash-config
This will also allow you to create charts and filter the logs effectively.

The default log viewer will likely crash. This is why my nginx log viewer handles logs differently (I had a bug report of a log in a size of 40MB which now works fine in Firefox and Chrome)

[franco]

400 times log write size roughly equals 400 times log read size on page load.


Cheers,
Franco

[rabievdm]

Thanks guys, I do appreciate the response.

I'm not too keen on deploying an ELK stack, but might just opt to push the syslog to a remote box just so I have a grep'able copy that goes back more than a couple hours

@Fabian, it would be nice to have an on box feature to archive selected logs to x numbers archives logs for auditing purposes

[fabian]

Thanks guys, I do appreciate the response.

I'm not too keen on deploying an ELK stack, but might just opt to push the syslog to a remote box just so I have a grep'able copy that goes back more than a couple hours

@Fabian, it would be nice to have an on box feature to archive selected logs to x numbers archives logs for auditing purposes

The ELK stack acts as a remote syslog server.

Elastic Search is a document storage which holds the data and effectively access them.
Logstash is a tool that processes the input data (syslog) and forwards the processed data to the output (Elastic Search).
Kibana is a UI for log data in Elastic Search  which allows to write queries and create charts like browser statistics.

[rabievdm]

Hi Fabian,

Again thanks for the response.

What I mean was I'm familiar with the ELK stack and the components, just that I'm not keen on deploying more resources and maintain them for odd occasion that I want to look back a little more than currently offered out of box. Hence the worst case off box syslog option

But again appreciate the feedback.

R