Print Page - Exessive SSD wear on Kingston m.2 NVMe

We have been running an installation of OPNsense with the latest version for a few months and have noticed alarming levels of SSD writes that we cannot explain.

There is very little logging happening. The file system has very little storage used and we have zenarmor and netflow enabled but not much more. Most firewall rules are set to not log traffic.

Here are two screenshots of SSD attributes about 30 days apart. The writes have increased by 2,11TB in that time.
https://imgur.com/a/SUzqITC

Ramdisks are not in use and only 13GB of storage is used currently

https://imgur.com/a/bnnq66A

When running the iostat command in shell, it does not show high writes:

Code: [Select]

iostat -x 1 10
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0      46      1.7    852.3     0     0     1     0    0   0
md43           0       0      0.0      1.8     0     0     0     0    0   0
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0     424      0.0   9191.4     0     0     1     0    0   1
md43           0       2      0.0      8.0     0     0     0     0    0   0
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0       6      0.0    103.9     0     0     1     0    0   0
md43           0       0      0.0      0.0     0     0     0     0    0   0
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0       4      0.0     24.0     0     0     1     1    0   0
md43           0       0      0.0      0.0     0     0     0     0    0   0
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0       7      0.0    100.0     0     0     1     0    0   1
md43           0       0      0.0      0.0     0     0     0     0    0   0
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0      45      0.0   2585.2     0     0     1     0    0   1
md43           0       0      0.0      0.0     0     0     0     0    0   0
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0     251      0.0   8193.3     0     0     4     0    0   6
md43           0       1      0.0      4.0     0     0     0     0    0   0
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0       4      0.0     32.0     0     9    14    12    0   6
md43           0       0      0.0      0.0     0     0     0     0    0   0
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0       4      0.0     56.0     0     0    12     6    0   5
md43           0       0      0.0      0.0     0     0     0     0    0   0
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0       0      0.0      0.0     0     0     0     0    0   0
md43           0       0      0.0      0.0     0     0     0     0    0   0

What could be causing this?

It has been 11 days and the writes are at 4.37TB. That makes it 79GB/day of writes that do not seem to take up any disk space.

https://imgur.com/a/ZLZvAet

Still no explanation as to what is causing this.

https://imgur.com/a/Rk1d1FY

If you use Netflow reporting, you will have /var/log/flowd*.log which are rotated and thus they do not take up much space, but get written A LOT.

There is a setting to have /var/log in the RAM disk to avoid this - you need to reboot to activate it.
With that setting, all logs will get lost on every reboot.

Also, the free space not diminishing is a tell-tale sign of a database that is being written to. In my case, it was Zenarmor, which uses an Sqlite database that was constantly being written to.

Quote from: meyergru on June 05, 2023, 03:04:02 pm

If you use Netflow reporting, you will have /var/log/flowd*.log which are rotated and thus they do not take up much space, but get written A LOT.

There is a setting to have /var/log in the RAM disk to avoid this - you need to reboot to activate it.
With that setting, all logs will get lost on every reboot.

Also, the free space not diminishing is a tell-tale sign of a database that is being written to. In my case, it was Zenarmor, which uses an Sqlite database that was constantly being written to.

Having logs written to a RAM disk that will be flushed in the event of a power outage is obviously not ideal.

At the moment we are at 5.64 TB written to the drive and I am wondering if the scale of writes can be concidered to be in the range of normal? At this rate the reliability of the SSD drive is in question since it is only rated for 160TB of writes within the warranty and possibly more before wearing out completely.

If there are features like this that will wear out the storage medium at this pace, I would like to have only these logs that are perhaps not very important to be held on a ram disk.

I have now disabled the collection of flowd logs and will see if this has an effect on the issue.

There have been a lot of discussions about this, see https://forum.opnsense.org/index.php?topic=29164.0

OPNsense Forum

English Forums => Hardware and Performance => Topic started by: mtchetch on May 25, 2023, 09:11:11 am