CPU spikes from Elasticsearch since upgrading to Zenarmor 2.7.1. Anyone else?

[mlenje]

Since upgrading to Zenarmor 2.7.1 on Oct 26, I've had two short CPU spikes on my OPNsense box. I've never had one before the upgrade.

When: Oct 27 at 02:15 and 22:15

What: each lasted about 4 minutes, with CPU at 90%+ and the load average around 8–9. It recovered on its own.

Source: the Elasticsearch java process (local Elasticsearch 8.11.3)

Ruled out: network traffic and cron jobs

Setup: OPNsense 26.7, i7-3770 with 32 GB RAM, local Elasticsearch reporting database.

I'm monitoring to see if it repeats.

Has anyone else seen something similar since moving to 2.7.1?

[xpking]

I upgraded to newest zenarmor 8 hours ago and experienced the cpu spike twice.
Also, the restart zenarmor button caused opnsense to reboot.
Please check if you also experience this. Thanks.

[mlenje]

Thanks, that's helpful. It sounds like we're seeing the same thing.

My timeline after upgrading at about 22:15 on Oct 26:

First spike at 02:15, about 4 hours after the upgrade
Nothing overnight Oct 27→28
Second spike at 22:15, about 24 hours after the upgrade, which is when the Zenarmor agent restarted.

Elasticsearch itself wasn't restarted by the upgrade, so the trigger seems to be a task inside Zenarmor that sends a heavy job to Elasticsearch.

Could you share the exact times of your spikes relative to your upgrade?

I haven't tried the Restart Zenarmor button yet. I'm capturing Elasticsearch hot threads and tasks during tonight's expected spike (24h after last night's spike), and I'll post the results.

I'll test the restart button after that, when a reboot won't disrupt anything.

[Greg_E]

Not on mine, seeing about 9% cpu when someone is doing work on the web.

[mlenje]

Update: I exported CPU history from Reporting > Health > System > Processor, and the spikes follow a fixed schedule:

Every 5 hours at :15, counted from when Zenarmor restarted during the upgrade (for me, 22:15 on Oct 26). Today's runs were 03:15, 08:15, 13:15 and 18:15.
Every 4th run is heavier, about once every 20 hours, with CPU at 90%+ for about 4 minutes. The lighter runs peak around 55–60% and didn't trigger my alerts.
Before the upgrade: no such pattern in 11 weeks of history.

The load is application CPU, not network or disk. During one heavy run, Elasticsearch was the top process in Diagnostics > Activity. I going to try to capture Elasticsearch's active tasks during the next predicted runs to identify the job itself and will post the results.

If you check your own Processor graph (5-minute resolution), do you see the same 5-hour rhythm starting from your upgrade time?

[sy]

Hi,

Can you check the /usr/local/zenarmor/log/active/Periodical.log for these timestamps?

[xpking]

I received CPU high (>80%) alert on xx:20 in every 5 hours.
Periodical.log seems like being cleared automatically. I only see a few min of logs when I checked.

[mlenje]

Thanks both. xpking's :20 every 5 hours matches my :15 every 5 hours, offset by our different upgrade times.

Periodical.log

On my box this file only ever holds the last minute (about 2 KB). Two samples, Oct 27 07:59 and Oct 29 11:56 (neither at a spike time), look identical:

checking mongodb...
checking mongodb done
dasboard-cache...
CLI dashboard-cache: Caching ...
CLI dashboard-cache: Cached ...
dashboard cache done
elastic-index-check...
elastic-index-check done
done
[2026-10-29T11:56:01][INFO][TEMP] Starting Zenarmor CRON for 2026-10-29 11:56:01

That's followed by Zenarmor's usual 18 internal Cron Check lines, with only scheduled-service-control, userenrich and aliases actually running (all OK). The mongodb / dashboard-cache / elastic-index-check block is always at the top with no timestamps, so it looks like a separate job that rewrites the file each time.

What Elasticsearch is doing during a spike

I logged Elasticsearch's task list and per-thread top every 30 seconds during the 23:15 run last night.

Before (23:13:37): idle. Load was 0.20, and Elasticsearch was running only its own health and monitoring tasks.

From 23:14:39, this search appears:

indices:data/read/search
indices[zenarmor_<id>_conn_all]
"query":{"range":{"start_time":{"gte":1790640000000,"lt":1793232000000}}}
"aggregations":{"daily":{"composite":{"size":1000,
  "sources":[{"identity":{"terms":{"field":"ip_src_saddr.keyword"}}},
             {"day":{"date_histogram":{"field":"start_time","calendar_interval":"day"}}}],
  "after":{"identity":"<ip>","day":...}}}}
The range is Sep 29 to Oct 29 (UTC), exactly 30 days.
It's a composite aggregation by source IP × day, paged 1,000 buckets at a time.
The same query appears in 7 consecutive snapshots (about 3½ minutes), each with a different after key advancing through the source IPs. So it looks like one continuous pass through the whole result set.

At the peak (23:16:10): all 15 of the busiest threads in top -H were Elasticsearch's java, with 13 of them at about 30% each. No other process appeared.

Summary: every 5 hours at :15 since the 2.7.1 upgrade, a 30-day per-device daily aggregation on conn_all runs for about 4 minutes. Every 4th run is heavier (90%+ CPU). There was no such pattern in 11 weeks of CPU history before the upgrade.

Which task sends this query? Is it dashboard-cache? And are the interval or the 30-day window configurable? I'm capturing Periodical.log and the process connecting to port 9200 during my next predicted run at 14:15 today and will post the results.

[mlenje]

Update: it's the heartbeat, and 2.7.1 made it ~500x slower and runs it twice

I captured today's predicted heavy run at 14:15 (Monit: CPU 95.4% at 14:15:48). This answers the questions from my last post.

Periodical.log during the spike
I saved every version of the file from 14:13 to 14:23. Nothing unusual runs at the spike: scheduled-service-control, userenrich and aliases run every minute; check-remote-elastic and periodic-license-control run every 5 minutes (14:15 and 14:20); check-health ran at 14:14. So dashboard-cache and the per-minute scheduler aren't the cause. There are also no Elasticsearch transforms configured.

The cause: the ipdrstreamer-v2 heartbeat
grep 'heartbeat: collected' /usr/local/zenarmor/log/active/ipdrstreamer-v2*.log shows:

Before 2.7.1 (cli.log, triggered_by=agent, every 4h):

Code Select Expand

2026-10-25T22:51:54 heartbeat: collected took=405ms
2026-10-26T02:51:54 heartbeat: collected took=354ms
2026-10-26T06:51:54 heartbeat: collected took=416ms

After 2.7.1, two collectors:

Code Select Expand

# ipdrstreamer's own scheduler, every 5h from its start (22:14:38 on Oct 26)
2026-10-28T23:18:10 heartbeat: collected took=3m32.228s
2026-10-29T04:18:02 heartbeat: collected took=3m24.436s
2026-10-29T09:18:05 heartbeat: collected took=3m26.881s
2026-10-29T14:19:27 heartbeat: collected took=4m49.409s

# agent-triggered (cli.log), fired only 4 times since the upgrade
2026-10-27T02:19:07 heartbeat: collected triggered_by=agent took=3m59.286s
2026-10-27T22:19:34 heartbeat: collected triggered_by=agent took=4m26.416s
2026-10-28T18:20:01 heartbeat: collected triggered_by=agent took=4m53.923s
2026-10-29T14:19:55 heartbeat: collected triggered_by=agent took=4m47.296s

What this shows:
- The heartbeat collection went from ~0.4 seconds to 3.5-5 minutes in 2.7.1. During it, Elasticsearch runs the 30-day composite aggregation on conn_all (source IP x day, paged 1,000 buckets) from my last post, saturating 4+ cores.
- 2.7.1 adds a second collector inside ipdrstreamer that runs every 5 hours from ipdrstreamer's start time. That's the :15 / :20 cycle xpking and I see.
- The heavy runs are both collectors running in parallel: today the scheduler started at 14:14:38 and the agent-triggered one at 14:15:08. The agent's 4-hourly check only fires when 4+ hours have passed since the last heartbeat, which with the 5-hourly scheduler lines up once every 20 hours. All four agent-triggered runs match my four 90%+ CPU alerts exactly.

Questions for the Zenarmor team (@sy):
1. What changed in the 2.7.1 heartbeat that takes it from ~0.4 s to ~4 min? It now aggregates 30 days of conn data by source IP and day each time.
2. Is the second (5-hourly) collector in ipdrstreamer intended, and should it share a lock with the agent-triggered one so they can't run at the same time?
3. Can the heartbeat be made incremental, or its interval/lookback configured on Home licenses?

Setup: OPNsense 26.7, Zenarmor 2.7.1, local Elasticsearch 8.11.3, i7-3770 / 32 GB.