CPU spikes from Elasticsearch since upgrading to Zenarmor 2.7.1. Anyone else?

[mlenje]

Since upgrading to Zenarmor 2.7.1 on Oct 26, I've had two short CPU spikes on my OPNsense box. I've never had one before the upgrade.

When: Oct 27 at 02:15 and 22:15

What: each lasted about 4 minutes, with CPU at 90%+ and the load average around 8–9. It recovered on its own.

Source: the Elasticsearch java process (local Elasticsearch 8.11.3)

Ruled out: network traffic and cron jobs

Setup: OPNsense 26.7, i7-3770 with 32 GB RAM, local Elasticsearch reporting database.

I'm monitoring to see if it repeats.

Has anyone else seen something similar since moving to 2.7.1?

[xpking]

I upgraded to newest zenarmor 8 hours ago and experienced the cpu spike twice.
Also, the restart zenarmor button caused opnsense to reboot.
Please check if you also experience this. Thanks.

[mlenje]

Thanks, that's helpful. It sounds like we're seeing the same thing.

My timeline after upgrading at about 22:15 on Oct 26:

First spike at 02:15, about 4 hours after the upgrade
Nothing overnight Oct 27→28
Second spike at 22:15, about 24 hours after the upgrade, which is when the Zenarmor agent restarted.

Elasticsearch itself wasn't restarted by the upgrade, so the trigger seems to be a task inside Zenarmor that sends a heavy job to Elasticsearch.

Could you share the exact times of your spikes relative to your upgrade?

I haven't tried the Restart Zenarmor button yet. I'm capturing Elasticsearch hot threads and tasks during tonight's expected spike (24h after last night's spike), and I'll post the results.

I'll test the restart button after that, when a reboot won't disrupt anything.

[Greg_E]

Not on mine, seeing about 9% cpu when someone is doing work on the web.

[mlenje]

Update: I exported CPU history from Reporting > Health > System > Processor, and the spikes follow a fixed schedule:

Every 5 hours at :15, counted from when Zenarmor restarted during the upgrade (for me, 22:15 on Oct 26). Today's runs were 03:15, 08:15, 13:15 and 18:15.
Every 4th run is heavier, about once every 20 hours, with CPU at 90%+ for about 4 minutes. The lighter runs peak around 55–60% and didn't trigger my alerts.
Before the upgrade: no such pattern in 11 weeks of history.

The load is application CPU, not network or disk. During one heavy run, Elasticsearch was the top process in Diagnostics > Activity. I going to try to capture Elasticsearch's active tasks during the next predicted runs to identify the job itself and will post the results.

If you check your own Processor graph (5-minute resolution), do you see the same 5-hour rhythm starting from your upgrade time?