Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - bodenlos

#1
Thanks for sharing your findings. I can add some further real-world data points that seem to support the same pattern:

I am seeing the same behaviour on three additional production systems with essentially the same system design. The absolute write volumes differ somewhat, but the result is consistent: when the Zenarmor engine is running, SSD host writes increase by approximately 4–5x compared with the baseline without Zenarmor.

Based on the measured write rates and the respective SSD endurance ratings (TBW), the specified endurances would be reached after approximately 2–4 years of operation, depending on the SSD and system.

For comparison, on my own system with the 320 TBW rating of my Kingston NV3, that corresponds to roughly 9.1 years at the baseline write rate, compared with only 2.3 years with Zenarmor enabled. (TBH: are endurance projections, not predictions of actual SSD failure.)
What makes this particularly concerning is that these are Proxmox hosts running several other servers and services, which already generate a substantial amount of disk I/O. Nevertheless, the additional write load associated with Zenarmor exceeds the combined baseline workload by several times. Essentially for the purpose of persisting reporting and connection metadata!

At this point, the consistency across multiple systems makes me strongly suspect a fundamental design or implementation issue in how much data Zenarmor writes and how those writes are performed. Whether the main contributor is the record volume, SQLite WAL/checkpointing, cleanup operations, write amplification, or a combination of these still needs to be investigated. But the resulting sustained write load seems difficult to justify for a reporting feature, particularly on appliances with limited SSD endurance.

I would therefore really like to see Zenarmor investigate the root cause and provide a technical explanation and mitigation for the local reporting backend. Ideally, there should also be an option to disable persistent reporting entirely while keeping the filtering/security engine active, for users who do not need historical reporting data.
#2
Quote from: Bunch on September 07, 2026, 10:53:06 AM...

Environment:
Base OS: Proxmox 9.2.11
Kernel Version: Linux 7.0.14-15-pve
SSD (hardware): Samsung SSD 970 EVO Plus 500GB
OPNsense VM: (Disk setting only) LVM-Thin, 32GB, Cache=Default (no Cache), Discard=on, IO thread=on, SSD emulation=on
OPNsense: OPNsense 26.7.3_11-amd64 FreeBSD 15.1-RELEASE-p3
Zenarmor Engine: 2.6.2
Zenarmor Database: 2.0.26071708
Reporting backend: Local SQLite 3.53.4,1

->> What filesystems are you using on the Proxmox host and inside the OPNsense VM?
#3
I noticed unusually high SSD wear on an OPNsense system and started tracing the write activity.

A controlled A/B test using the NVMe SMART [tt]Data Units Written[/tt] counter showed a very large difference depending on whether the Zenarmor engine was running.

Zenarmor OFF, 16 minutes


212,834,990 -> 212,837,079

= 2,089 data units 
≈ 1.07 GB written in 16 minutes 
≈ 96 GB/day extrapolated

Zenarmor ON, 16 minutes


212,837,079 -> 212,845,277

= 8,198 data units 
≈ 4.20 GB written in 16 minutes 
≈ 378 GB/day extrapolated

So in this test, enabling Zenarmor increased total SSD writes by roughly 4x, corresponding to about 282 GB/day additional writes.

Zenarmor is using the local SQLite reporting backend. While the engine is running, the SQLite databases and WAL files under:

/usr/local/datastore/sqlite

are continuously updated, especially files such as:

conn_all.sqlite(-wal)
dns_all.sqlite(-wal)
tls_all.sqlite(-wal)
alert_all.sqlite(-wal)
http_all.sqlite(-wal)


What makes this concerning is SSD endurance.

The SSD in this system is rated for 320 TBW. At around 378 GB/day, the rated write endurance would be reached after only about 2.3 years of continuous 24/7 operation.

The additional writes attributable to Zenarmor alone are roughly 282 GB/day, which corresponds to more than 100 TB/year.

I do not actually need historical Zenarmor reporting and would prefer to keep only the filtering/security engine active.

Environment:

OPNsense: OPNsense 26.7.3_8-amd64 FreeBSD 15.1-RELEASE-p3
Zenarmor Engine: 2.6.2
Zenarmor Database: 2.0.26071708
Reporting backend: Local SQLite


Has anyone else on OPNsense measured similar write volumes with Zenarmor?

I am especially interested in whether:

  • this level of write activity is considered normal,
  • there are OPNsense/FreeBSD-specific factors that significantly amplify these writes,
  • there is a supported way to disable or strongly reduce Zenarmor reporting persistence while keeping the engine active,
  • or whether this could indicate an issue in the SQLite/reporting pipeline.