Hostwatch - high disk writes

[GreenMatter]

After upgrading to 25.7.11 hostwatch (v. 1.0.2) causes high (60M) disk writes and increased CPU utilisation. 
Is there any fix for it?

[franco]

https://github.com/opnsense/changelog/blob/efe03ef435b5abfff641262fd69e02efd926be5a/community/25.7/25.7.11#L10-L12

Interfaces: Neighbors: Automatic Discovery.


Cheers,
Franco

[GreenMatter]

https://github.com/opnsense/changelog/blob/efe03ef435b5abfff641262fd69e02efd926be5a/community/25.7/25.7.11#L10-L12

Interfaces: Neighbors: Automatic Discovery.


Cheers,
Franco

Thanks, I've seen it. But it still causing really high disk writes. For a time being I stopped this service...

[franco]

Well, it's either enabled or not. There may be a bug that doesn't stop it but I haven't seen it. Worst case a reboot would take care of it (when properly disabled).


Cheers,
Franco

[GreenMatter]

Well, it's either enabled or not. There may be a bug that doesn't stop it but I haven't seen it. Worst case a reboot would take care of it (when properly disabled).


Cheers,
Franco

Does hostwatch suppose to create such disk writes?

[OPNenthu]

Mine is using only ~56k so far since upgrade, but my home network is small.

Code Select Expand

root@firewall:/var/log/hostwatch # ls -l
total 56
-rw-------  1 root wheel 56388 Jan 16 14:35 hostwatch_20260116.log
lrwxr-x---  1 root wheel    41 Jan 16 15:01 latest.log -> /var/log/hostwatch/hostwatch_20260116.log


Is your hostwatch log being flooded with error messages, or is most of your log filled with host discoveries?

[jjelliott]

Looking at iostat in the console, I am seeing high disk writes, too.

Code Select Expand

root@OPNsense:~ # iostat -x
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
ada0           1     107     42.0   2570.7     1     1     0     1    0   8
You can see the instantaneous rate by issuing

Code Select Expand

iostat -x 2.  When I disable Automatic Discovery, the instantaneous writes drop back to near zero.

This is on a small home network.  I have disabled the feature for now.

[franco]

It's supposed to log hardware address movements, but if it sees them constantly that is probably undesirable as logging. The issue is clear and we'll find a solution for it soon.


Cheers,
Franco

[pfry]

Out of curiosity, what is considered a "movement", and what sort of errors would it log? Just trying to get a handle on the high writes. I don't see any notable change in write frequency on my own system, and it's the wacky four-bridge setup, where "Interfaces: Neighbors: Automatic Discovery" (by default) picks up every MAC on multiple interfaces. (I fired it up with default settings just to see if I could trigger the issue, as I don't use it normally; actual ARP mapping does not normally move, as I do not normally re-plug machines and I have static ARP entries to tame my ISP's unlimited proxy.)

[pfry]

Interesting, thanks. I'll have to do some reading at some point, as hostwatch's criteria for a "duplicate" are not entirely clear to me. Apparently it does not monitor (third-party) ARP traffic passing through the device (on the bridges), as I have one device that will trigger the ARP proxy and receive multiple replies. (It runs Linux, which is apparently immune to the ARP proxy, so I don't use static entries on it. Something else I've been meaning to look into. One of these days.)

[OPNenthu]

Interesting, thanks. I'll have to do some reading at some point, as hostwatch's criteria for a "duplicate" are not entirely clear to me

Sorry, it wasn't a response to your post.  I had posted a report of the issue and some logs after I ran into it, but it ended up being duplicate of what's already reported in GH.  I decided to remove my noise.

[franco]

> Out of curiosity, what is considered a "movement"

We're recording the last MAC address for any IPv4 and IPv6 we see. If the MAC changes that's considered a "movement". In some environments this happens very rapidly and thus the service constantly registers the changes.


Cheers,
Franco

[GreenMatter]

That's how increase (clearly you can see when hostwatch was running) in writes looks like; my instance runs in Proxmox as VM...

[Arno Thuber]

Hi,

last night I've been bitten the second time since installind 25.7.11_1 by the same behaviour:
Unbound exited due to disk full: hostwatch.log grew to 218GB and made df -h report me -17GB free.
Simultaneously average CPU load wento from 0.3 to 1.7 caused by two processes:

Code Select Expand

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
15589 root          6  23    0    75M    28M kqread   3 518:13 102.21% syslog-ng
30715 hostd        18  20    0    99M    11M uwait    2 542:56 100.41% hostwatch
And what did hostwatch contain multiple times per second?

Code Select Expand

<11>1 2026-01-17T00:00:00+00:00 OPNsense hostwatch 30715 - [meta sequenceId="1"]   2026-01-17T00:00:00.000015Z ERROR hostwatch: Error reading from capture: libpcap error: The interface disappeared
Other than hostwatch (which I just disabled for the time being) my OPNsense is running smooth as ever.

Ah, I just found out about https://github.com/opnsense/hostwatch/issues/8.