Significant increase in CPU and disk IO after update

[RutgerDiehard]

Following the recent update, disk and CPU usage has increased significantly which appears to be caused by elasticsearch and the Java process.

I've run a health check on the settings database and run a check on the database indexes which both completed successfully. I also increased the memory disk size to 500MB. I've even completely reset the reporting database but nothing has helped.

If I restart the Zenarmor filter engine, the system behaves itself for a while but then CPU load increases and disk IO - specifically write operations - goes through the roof.

The screenshots show IO after the latest update was applied which was around 10:00am mark and the Java process hitting the disk.

This does seem to be update related as the system has been rock solid with very little load up to this point.

[RutgerDiehard]

Java process screenshot

[RutgerDiehard]

Another screenshot showing Elasticsearch as the culprit

[RutgerDiehard]

I've spent some time looking at this and have come across something that's concerning. Checking the other Netdata graphs at the time the high disk writes occur, I can see similar patterns for IPv4 Bandwidth usage.

[RutgerDiehard]

This is also reflected in Handshake issues

[RutgerDiehard]

And SYN Cookies

[RutgerDiehard]

But what is confusing is that this is not reflected in the interface stats. Does this mean it's OPNsense only and not something on the network?

I've not seen anything like this before.

[sy]

Hi,

Can you share a report via Have Feedback option in the bottom right corner of UI to check the Zenarmor Engine logs.

[RutgerDiehard]

Thanks sy, report sent including logs, configuration and OPNsense configuration.

[RutgerDiehard]

Update:

Zenarmor Support has replied:

" We have identified some issues with IPDRStreamer and addressed them in version 1.18.4, which is scheduled for release next week."

I have been given access to the beta version to test and so far, it's fixed the high IO and IPv4 issues. CPU is raised slightly from pre-update levels, but I'll wait and see if the official release has this.

Many thanks to the quick response from Zenarmor Support and access to the fix.