hostwatch db grows rapidly

[astrandb]

I upgraded to 26.1 a couple of hours ago and suddenly got a warning that the disk is close to full.
It is caused by hostwatch db growing rapidly:

Code Select Expand

root@xxx:~ # ll -h /var/db/hostwatch/
total 6392960
-rw-r--r--  1 hostd hostd  4.0M Jan 30 10:25 hosts.db
-rw-r--r--  1 hostd hostd   12M Jan 30 10:25 hosts.db-shm
-rw-r--r--  1 hostd hostd  6.1G Jan 30 10:25 hosts.db-wal
I have rebooted but still same filesize

[Patrick M. Hausen]

Disable hostwatch for the time being.

Interfaces: Neighbors: Automatic discovery

[franco]

You can also try the latest test version as we already found the auto-vacuum doesn't always trigger:

# opnsense-revert -z hostwatch
# service hostwatch restart


Cheers,
Franco

[bazineta]

Same issue observed with 26.1. Resolved by the latest test version, here's the before and after:

total 2286624
-rw-r--r--  1 hostd hostd  4.2M Jan 30 04:47 hosts.db
-rw-r--r--  1 hostd hostd  4.3M Jan 30 04:47 hosts.db-shm
-rw-r--r--  1 hostd hostd  2.2G Jan 30 04:47 hosts.db-wal
root@www:~ # ll -h /var/db/hostwatch/
total 139904
-rw-r--r--  1 hostd hostd  4.2M Jan 30 04:47 hosts.db
-rw-r--r--  1 hostd hostd  4.3M Jan 30 04:47 hosts.db-shm
-rw-r--r--  1 hostd hostd  128M Jan 30 04:47 hosts.db-wal
root@www:~ # ll -h /var/db/hostwatch/

[astrandb]

You can also try the latest test version as we already found the auto-vacuum doesn't always trigger:

# opnsense-revert -z hostwatch
# service hostwatch restart


Cheers,
Franco

Thank you. The test version worked fine. Same sizes as above.

[franco]

With the help of your feedback we decided to hotfix the .11 as well in 2-3 hours.


Cheers,
Franco

[joeyboon]

Thank you for this post. I had three instances that almost ran out of diskspace. Was able to fix it because of this in a couple of minutes.

[franco]

Hotfix is available now for everyone. Make sure to restart hostwatch to ensure the correct version is running since there is no reboot.


Cheers,
Franco

[HaMaMaHa]

Hi there,

I even rebooted after going to OPNsense 26.1_4-amd64, but hosts.db-wal grows incredible fast (for a homelab) - see below:

root@OPNsense:/var/db/hostwatch # ls -lha
total 4015
drwxr-xr-x   2 hostd hostd    5B Feb  2 16:12 .
drwxr-xr-x  21 root  wheel   28B Feb  2 16:11 ..
-rw-r-----   1 hostd hostd  4.0M Feb  2 16:12 hosts.db
-rw-r-----   1 hostd hostd   32K Feb  2 16:12 hosts.db-shm
-rw-r-----   1 hostd hostd  5.4M Feb  2 16:13 hosts.db-wal


root@OPNsense:/var/db/hostwatch # ls -lha
total 4503
drwxr-xr-x   2 hostd hostd    5B Feb  2 16:12 .
drwxr-xr-x  21 root  wheel   28B Feb  2 16:11 ..
-rw-r-----   1 hostd hostd  4.0M Feb  2 16:12 hosts.db
-rw-r-----   1 hostd hostd   32K Feb  2 16:12 hosts.db-shm
-rw-r-----   1 hostd hostd   14M Feb  2 16:17 hosts.db-wal

Any idea what could be wrong? Thanks in advance!
KR
Harald

[franco]

If it doesn't grow beyond tens of megabytes it's ok.

We'll refine this further to minimise database writes in the near future which should also make the journal smaller.


Cheers,
Franco

[dinguz]

It's currently 150 MB on my home router, mostly caused by IPv6 addresses which appear to change frequently over time. I also see log entries going back to the initial activation. Is there any expiry or cleanup mechanism in place for HostWatch data (database and/or logs)?