Very high SSD writes with Zenarmor + SQLite

[bodenlos]

I noticed unusually high SSD wear on an OPNsense system and started tracing the write activity.

A controlled A/B test using the NVMe SMART [tt]Data Units Written[/tt] counter showed a very large difference depending on whether the Zenarmor engine was running.

Zenarmor OFF, 16 minutes


212,834,990 -> 212,837,079

= 2,089 data units 
≈ 1.07 GB written in 16 minutes 
≈ 96 GB/day extrapolated

Zenarmor ON, 16 minutes


212,837,079 -> 212,845,277

= 8,198 data units 
≈ 4.20 GB written in 16 minutes 
≈ 378 GB/day extrapolated

So in this test, enabling Zenarmor increased total SSD writes by roughly 4x, corresponding to about 282 GB/day additional writes.

Zenarmor is using the local SQLite reporting backend. While the engine is running, the SQLite databases and WAL files under:

/usr/local/datastore/sqlite

are continuously updated, especially files such as:

conn_all.sqlite(-wal)
dns_all.sqlite(-wal)
tls_all.sqlite(-wal)
alert_all.sqlite(-wal)
http_all.sqlite(-wal)

What makes this concerning is SSD endurance.

The SSD in this system is rated for 320 TBW. At around 378 GB/day, the rated write endurance would be reached after only about 2.3 years of continuous 24/7 operation.

The additional writes attributable to Zenarmor alone are roughly 282 GB/day, which corresponds to more than 100 TB/year.

I do not actually need historical Zenarmor reporting and would prefer to keep only the filtering/security engine active.

Environment:

OPNsense: OPNsense 26.7.3_8-amd64 FreeBSD 15.1-RELEASE-p3
Zenarmor Engine: 2.6.2
Zenarmor Database: 2.0.26071708
Reporting backend: Local SQLite


Has anyone else on OPNsense measured similar write volumes with Zenarmor?

I am especially interested in whether:

• this level of write activity is considered normal,
• there are OPNsense/FreeBSD-specific factors that significantly amplify these writes,
• there is a supported way to disable or strongly reduce Zenarmor reporting persistence while keeping the engine active,
• or whether this could indicate an issue in the SQLite/reporting pipeline.

[sy]

Hi,

Thank you for the detials. We got your request as a ticket as well and investigating it. We will get back to you as soon as possible.

[Bunch]

Since no one at home right now, the loading of my router is low.
My test is as below:

Zenarmor ON, 15 minutes
127,049,983>127,051,329
=1346 data units
≈688.6MB written in 15 minutes
≈66GB/day

Zenarmor OFF, 15 minutes
127,052,544>127,052,839
=295 data units
≈150.9MB written in 15 minutes
≈14GB/day

Environment:
Base OS: Proxmox 9.2.11
Kernel Version: Linux 7.0.14-15-pve
SSD (hardware): Samsung SSD 970 EVO Plus 500GB
OPNsense VM: (Disk setting only) LVM-Thin, 32GB, Cache=Default (no Cache), Discard=on, IO thread=on, SSD emulation=on
OPNsense: OPNsense 26.7.3_11-amd64 FreeBSD 15.1-RELEASE-p3
Zenarmor Engine: 2.6.2
Zenarmor Database: 2.0.26071708
Reporting backend: Local SQLite 3.53.4,1

[bodenlos]

...

Environment:
Base OS: Proxmox 9.2.11
Kernel Version: Linux 7.0.14-15-pve
SSD (hardware): Samsung SSD 970 EVO Plus 500GB
OPNsense VM: (Disk setting only) LVM-Thin, 32GB, Cache=Default (no Cache), Discard=on, IO thread=on, SSD emulation=on
OPNsense: OPNsense 26.7.3_11-amd64 FreeBSD 15.1-RELEASE-p3
Zenarmor Engine: 2.6.2
Zenarmor Database: 2.0.26071708
Reporting backend: Local SQLite 3.53.4,1

->> What filesystems are you using on the Proxmox host and inside the OPNsense VM?

[Bunch]

Proxmox host: ext4
VM storage: LVM-thin
OPNSense: ZFS

[RamSense]

I can add another real-world data point to this.

I have been investigating unexpectedly high SSD wear on a Deciso DEC850v2 running OPNsense with Zenarmor and the local SQLite reporting backend. The original SSD became heavily worn after roughly two years of operation and I have now had to replace it.

My network is not particularly large: around 40 active devices in a normal home/family environment.

I opened a support case with Zenarmor on August 21 and supplied them with the measurements and logs I had collected. During that investigation, Zenarmor support confirmed that around 45 session records per second for approximately 40 devices is considered normal, and also explained that a single website visit can generate another 10–15 records because connections/sessions and DNS requests are recorded for reporting.

The preserved IPDR logs from my old installation show continuous activity in conn_all.sqlite, together with periodic cleanup/incremental-vacuum operations. The database was operating in SQLite WAL mode. Unfortunately, I did not record the instantaneous size of conn_all.sqlite-wal before the old SSD was removed, which Zenarmor support has subsequently asked about.

I am not claiming that Zenarmor alone has been conclusively proven to have caused the SSD failure. However, it is a very serious suspected contributor. The important point for me is that the high record rate is apparently considered normal Zenarmor workload, rather than the result of some abnormal client on my network.

Since replacing the SSD, I rebuilt OPNsense without reinstalling Zenarmor. So far, the excessive SSD write behaviour that triggered this investigation has not returned. In other words, removing Zenarmor from the new installation has, at least for now, clearly removed the abnormal write-load problem we were seeing.

Because of that, I currently have no intention of reinstalling Zenarmor until there is a convincing technical explanation or solution for the local write behaviour.

At this point I have also not received any reimbursement for the SSD that had to be replaced, nor have I received a concrete technical fix or mitigation from Zenarmor. The case is still open and I am still waiting for a substantive response from their technical/development team.

Their suggested workaround has been remote Elasticsearch, but to me that does not fully answer the underlying question: why does the normal local Zenarmor reporting workload generate enough sustained write activity that SSD endurance becomes a concern on an official Deciso appliance?

I have retained the old IPDR logs, SSD wear data and other evidence. I would therefore be very interested in further measurements from others in this thread, especially controlled Zenarmor ON/OFF comparisons and findings around SQLite WAL/checkpointing, cleanup and write amplification.

Your A/B measurements are particularly interesting because they seem consistent with the direction of what I observed, although my measurements were collected differently.

[bodenlos]

Thanks for sharing your findings. I can add some further real-world data points that seem to support the same pattern:

I am seeing the same behaviour on three additional production systems with essentially the same system design. The absolute write volumes differ somewhat, but the result is consistent: when the Zenarmor engine is running, SSD host writes increase by approximately 4–5x compared with the baseline without Zenarmor.

Based on the measured write rates and the respective SSD endurance ratings (TBW), the specified endurances would be reached after approximately 2–4 years of operation, depending on the SSD and system.

For comparison, on my own system with the 320 TBW rating of my Kingston NV3, that corresponds to roughly 9.1 years at the baseline write rate, compared with only 2.3 years with Zenarmor enabled. (TBH: are endurance projections, not predictions of actual SSD failure.)
What makes this particularly concerning is that these are Proxmox hosts running several other servers and services, which already generate a substantial amount of disk I/O. Nevertheless, the additional write load associated with Zenarmor exceeds the combined baseline workload by several times. Essentially for the purpose of persisting reporting and connection metadata!

At this point, the consistency across multiple systems makes me strongly suspect a fundamental design or implementation issue in how much data Zenarmor writes and how those writes are performed. Whether the main contributor is the record volume, SQLite WAL/checkpointing, cleanup operations, write amplification, or a combination of these still needs to be investigated. But the resulting sustained write load seems difficult to justify for a reporting feature, particularly on appliances with limited SSD endurance.

I would therefore really like to see Zenarmor investigate the root cause and provide a technical explanation and mitigation for the local reporting backend. Ideally, there should also be an option to disable persistent reporting entirely while keeping the filtering/security engine active, for users who do not need historical reporting data.