Hostwatch - high disk writes

[GreenMatter]

After upgrading to 25.7.11 hostwatch (v. 1.0.2) causes high (60M) disk writes and increased CPU utilisation. 
Is there any fix for it?

[franco]

https://github.com/opnsense/changelog/blob/efe03ef435b5abfff641262fd69e02efd926be5a/community/25.7/25.7.11#L10-L12

Interfaces: Neighbors: Automatic Discovery.


Cheers,
Franco

[GreenMatter]

https://github.com/opnsense/changelog/blob/efe03ef435b5abfff641262fd69e02efd926be5a/community/25.7/25.7.11#L10-L12

Interfaces: Neighbors: Automatic Discovery.


Cheers,
Franco

Thanks, I've seen it. But it still causing really high disk writes. For a time being I stopped this service...

[franco]

Well, it's either enabled or not. There may be a bug that doesn't stop it but I haven't seen it. Worst case a reboot would take care of it (when properly disabled).


Cheers,
Franco

[GreenMatter]

Well, it's either enabled or not. There may be a bug that doesn't stop it but I haven't seen it. Worst case a reboot would take care of it (when properly disabled).


Cheers,
Franco

Does hostwatch suppose to create such disk writes?

[OPNenthu]

Mine is using only ~56k so far since upgrade, but my home network is small.

Code Select Expand

root@firewall:/var/log/hostwatch # ls -l
total 56
-rw-------  1 root wheel 56388 Jan 16 14:35 hostwatch_20260116.log
lrwxr-x---  1 root wheel    41 Jan 16 15:01 latest.log -> /var/log/hostwatch/hostwatch_20260116.log


Is your hostwatch log being flooded with error messages, or is most of your log filled with host discoveries?

[jjelliott]

Looking at iostat in the console, I am seeing high disk writes, too.

Code Select Expand

root@OPNsense:~ # iostat -x
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
ada0           1     107     42.0   2570.7     1     1     0     1    0   8
You can see the instantaneous rate by issuing

Code Select Expand

iostat -x 2.  When I disable Automatic Discovery, the instantaneous writes drop back to near zero.

This is on a small home network.  I have disabled the feature for now.

[franco]

It's supposed to log hardware address movements, but if it sees them constantly that is probably undesirable as logging. The issue is clear and we'll find a solution for it soon.


Cheers,
Franco

[pfry]

Out of curiosity, what is considered a "movement", and what sort of errors would it log? Just trying to get a handle on the high writes. I don't see any notable change in write frequency on my own system, and it's the wacky four-bridge setup, where "Interfaces: Neighbors: Automatic Discovery" (by default) picks up every MAC on multiple interfaces. (I fired it up with default settings just to see if I could trigger the issue, as I don't use it normally; actual ARP mapping does not normally move, as I do not normally re-plug machines and I have static ARP entries to tame my ISP's unlimited proxy.)

[pfry]

Interesting, thanks. I'll have to do some reading at some point, as hostwatch's criteria for a "duplicate" are not entirely clear to me. Apparently it does not monitor (third-party) ARP traffic passing through the device (on the bridges), as I have one device that will trigger the ARP proxy and receive multiple replies. (It runs Linux, which is apparently immune to the ARP proxy, so I don't use static entries on it. Something else I've been meaning to look into. One of these days.)

[OPNenthu]

Interesting, thanks. I'll have to do some reading at some point, as hostwatch's criteria for a "duplicate" are not entirely clear to me

Sorry, it wasn't a response to your post.  I had posted a report of the issue and some logs after I ran into it, but it ended up being duplicate of what's already reported in GH.  I decided to remove my noise.

[franco]

> Out of curiosity, what is considered a "movement"

We're recording the last MAC address for any IPv4 and IPv6 we see. If the MAC changes that's considered a "movement". In some environments this happens very rapidly and thus the service constantly registers the changes.


Cheers,
Franco

[GreenMatter]

That's how increase (clearly you can see when hostwatch was running) in writes looks like; my instance runs in Proxmox as VM...

[Arno Thuber]

Hi,

last night I've been bitten the second time since installind 25.7.11_1 by the same behaviour:
Unbound exited due to disk full: hostwatch.log grew to 218GB and made df -h report me -17GB free.
Simultaneously average CPU load wento from 0.3 to 1.7 caused by two processes:

Code Select Expand

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
15589 root          6  23    0    75M    28M kqread   3 518:13 102.21% syslog-ng
30715 hostd        18  20    0    99M    11M uwait    2 542:56 100.41% hostwatch
And what did hostwatch contain multiple times per second?

Code Select Expand

<11>1 2026-01-17T00:00:00+00:00 OPNsense hostwatch 30715 - [meta sequenceId="1"]   2026-01-17T00:00:00.000015Z ERROR hostwatch: Error reading from capture: libpcap error: The interface disappeared
Other than hostwatch (which I just disabled for the time being) my OPNsense is running smooth as ever.

Ah, I just found out about https://github.com/opnsense/hostwatch/issues/8.

[GreenMatter]

I'll leave here following:

Code Select Expand

top -S -m io -o total

last pid: 25126;  load averages:  0.49,  0.44,  0.46                                                                                                                           up 0+01:26:59  20:11:30
143 processes: 3 running, 138 sleeping, 2 waiting
CPU:  9.4% user,  0.0% nice,  4.7% system,  0.9% interrupt, 84.9% idle
Mem: 1154M Active, 1634M Inact, 1585M Wired, 880M Buf, 7477M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
34081 hostd       2494     13      0   2496      0   2496  99.96% hostwatch
   16 root         500      0      0      1      0      1   0.04% bufdaemon
85376 root           0      0      0      0      0      0   0.00% php-cgi
53184 root          10      1      0      0      0      0   0.00% php
    1 root           0      0      0      0      0      0   0.00% init
74881 root           0      0      0      0      0      0   0.00% php-cgi
79105 root           0      0      0      0      0      0   0.00% csh
35073 root           0      0      0      0      0      0   0.00% php-cgi
68801 root           0      0      0      0      0      0   0.00% php-cgi
    2 root          33      0      0      0      0      0   0.00% clock
 5314 root           2      2      0      0      0      0   0.00% ng_queue
 1474 squid          0      0      0      0      0      0   0.00% security_file_certg

And also this:

Code Select Expand

fsck_ffs -n /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** SU+J Recovering /dev/gpt/rootfs

USE JOURNAL? no

Skipping journal, falling through to full fsck
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
UNALLOCATED  I=1122920  OWNER=hostd MODE=100644
SIZE=21032 MTIME=Jan 16 19:04 2026
FILE=/var/db/hostwatch/hosts.db-journal

UNEXPECTED SOFT UPDATE INCONSISTENCY

Second command's result worries me a bit, as I've just reinstalled Opnsense on fresh (and partitioned by installer) VM disk and right away I've got this kind of errors???