[SOLVED] hostwatch at 100% CPU

[meyergru]

I now installed 1.0.6 and re-enabled the service again. It works and CPU is not at 100%, but there is a hefty number of disk writes in bursts every minute and also, the SQLite db journal file is deleted after each transaction data batch has been committed. Using rollback capability in such a way incurs a huge penalty on writes, especially on ZFS.

Just a guess here: But I think that SQL rollback capability is not needed for this purpose and when I glanced at the hostwatch code, I found that the journaling mode is WAL, where other modes (like MEMORY or even OFF) might be more appropriate.


On a side note: I had some strange effects with the old version of the service - namely, that my own ping-based discovery tool suddenly had entries for every IP in the subnet active. Don't ask, IDK why or how this happened. I just disabled the service for the time being.

[franco]

https://github.com/opnsense/hostwatch/commit/482b45ce is on the way but not in 1.0.6.

For specific issues it may make sense to raise a ticket, but multiple versions are in flight now so it would be better to wait for the final one that's going into 26.1 to make reports on.


Cheers,
Franco

[LHoust]

I tried hostwatch-1.0.6.pkg, although I am still experiencing HDD Thrashing...

Restored hostwatch 1.0.5 from 25.7.11_2 and therefore I will run with Automatic Discovery disabled for now...

[LHoust]

I tried hostwatch-1.0.6.pkg, although I am still experiencing HDD Thrashing...

Restored hostwatch 1.0.5 from 25.7.11_2 and therefore I will run with Automatic Discovery disabled for now...

"I ran iostat -w 1 da0 and the data is conclusive. While throughput is low (~0.15 MB/s), the tps (Transactions Per Second) is constantly spiking between 20 and 30 with Hostwatch 1.0.6.

For a mechanical HDD, this high frequency of tiny 6KB writes forces constant head seeking. This confirms the issue isn't the amount of data being logged, but the frequency of the SQLite commits. I'll be keeping discovery OFF until a version with Batch/Lazy Writing is released."

[LisaMT]

My system crashed today.  Thanks for all the notes on here.  I disabled the hostwatch and logs and it's running again. 
There should be some indicator on the dashboard that monitors disk usage better.

[Cljackhammer]

I think that it might be beneficial to remove this capability. The issue with the disk writes is a major problem and introduces instability. I had it enabled and the disk writes are higher than my production elastic search instance by a factor 700%

[iMx]

I think there's definitely an argument for it to be disabled by default.

... I'm not sure I need to put unnecessary wear on an SSD for this.

I'd have thought that most Business Edition customers will disable it and they bring in the money!

[OPNenthu]

Here's an updated version of hostwatch we're also consider shipping in a hotfix based on user feedback:

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/hostwatch-1.0.6.pkg

[...]

I rebooted my 25.7.11_2 install that was already running this v1.0.6 patch for the last ~2 days and on startup I noticed this in the console:

Code Select Expand

Starting hostwatch.
Starting acme_http_challenge.
>>> Invoking start script 'syslog'
>>> Invoking start script 'carp'
>>> Invoking start script 'cron'
Starting Cron: OK


>>> Invoking start script 'hostwatch'
hostwatch already running?  (pid=56724).
>>> Error in start script '90-hostwatch'

Hostwatch is up and running, though.  Looks OK.

And the contents of that file:

Code Select Expand

root@firewall:~ # cat /usr/local/etc/rc.syshook.d/start/90-hostwatch
#!/bin/sh

# work around mysterious first-time-only segfault
/usr/local/etc/rc.d/hostwatch start


[franco]

@OPNenthu https://github.com/opnsense/hostwatch/issues/7

@iMx might be room for disabling it on Nano images and adding a wizard option for it. In any case we'll add a migration note. Disabling this manually is not an inconvenience IMO.


Cheers,
Franco