After upgrading to 25.7.11 hostwatch (v. 1.0.2) causes high (60M) disk writes and increased CPU utilisation.
Is there any fix for it?
https://github.com/opnsense/changelog/blob/efe03ef435b5abfff641262fd69e02efd926be5a/community/25.7/25.7.11#L10-L12
Interfaces: Neighbors: Automatic Discovery.
Cheers,
Franco
Quote from: franco on January 16, 2026, 09:02:37 PMhttps://github.com/opnsense/changelog/blob/efe03ef435b5abfff641262fd69e02efd926be5a/community/25.7/25.7.11#L10-L12
Interfaces: Neighbors: Automatic Discovery.
Cheers,
Franco
Thanks, I've seen it. But it still causing really high disk writes. For a time being I stopped this service...
Well, it's either enabled or not. There may be a bug that doesn't stop it but I haven't seen it. Worst case a reboot would take care of it (when properly disabled).
Cheers,
Franco
Quote from: franco on January 16, 2026, 09:26:15 PMWell, it's either enabled or not. There may be a bug that doesn't stop it but I haven't seen it. Worst case a reboot would take care of it (when properly disabled).
Cheers,
Franco
Does hostwatch suppose to create such disk writes?
Mine is using only ~56k so far since upgrade, but my home network is small.
root@firewall:/var/log/hostwatch # ls -l
total 56
-rw------- 1 root wheel 56388 Jan 16 14:35 hostwatch_20260116.log
lrwxr-x--- 1 root wheel 41 Jan 16 15:01 latest.log -> /var/log/hostwatch/hostwatch_20260116.log
Is your hostwatch log being flooded with error messages, or is most of your log filled with host discoveries?
Looking at iostat in the console, I am seeing high disk writes, too.
root@OPNsense:~ # iostat -x
extended device statistics
device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b
ada0 1 107 42.0 2570.7 1 1 0 1 0 8
You can see the instantaneous rate by issuing iostat -x 2. When I disable Automatic Discovery, the instantaneous writes drop back to near zero.
This is on a small home network. I have disabled the feature for now.
It's supposed to log hardware address movements, but if it sees them constantly that is probably undesirable as logging. The issue is clear and we'll find a solution for it soon.
Cheers,
Franco
Out of curiosity, what is considered a "movement", and what sort of errors would it log? Just trying to get a handle on the high writes. I don't see any notable change in write frequency on my own system, and it's the wacky four-bridge setup, where "Interfaces: Neighbors: Automatic Discovery" (by default) picks up every MAC on multiple interfaces. (I fired it up with default settings just to see if I could trigger the issue, as I don't use it normally; actual ARP mapping does not normally move, as I do not normally re-plug machines and I have static ARP entries to tame my ISP's unlimited proxy.)
Interesting, thanks. I'll have to do some reading at some point, as hostwatch's criteria for a "duplicate" are not entirely clear to me. Apparently it does not monitor (third-party) ARP traffic passing through the device (on the bridges), as I have one device that will trigger the ARP proxy and receive multiple replies. (It runs Linux, which is apparently immune to the ARP proxy, so I don't use static entries on it. Something else I've been meaning to look into. One of these days.)
Quote from: pfry on January 17, 2026, 04:43:37 AMInteresting, thanks. I'll have to do some reading at some point, as hostwatch's criteria for a "duplicate" are not entirely clear to me
Sorry, it wasn't a response to your post. I had posted a report of the issue and some logs after I ran into it, but it ended up being duplicate of what's already reported in GH. I decided to remove my noise.
> Out of curiosity, what is considered a "movement"
We're recording the last MAC address for any IPv4 and IPv6 we see. If the MAC changes that's considered a "movement". In some environments this happens very rapidly and thus the service constantly registers the changes.
Cheers,
Franco
That's how increase (clearly you can see when hostwatch was running) in writes looks like; my instance runs in Proxmox as VM...
Hi,
last night I've been bitten the second time since installind 25.7.11_1 by the same behaviour:
Unbound exited due to disk full: hostwatch.log grew to 218GB and made df -h report me -17GB free.
Simultaneously average CPU load wento from 0.3 to 1.7 caused by two processes:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
15589 root 6 23 0 75M 28M kqread 3 518:13 102.21% syslog-ng
30715 hostd 18 20 0 99M 11M uwait 2 542:56 100.41% hostwatch
And what did hostwatch contain multiple times per second?
<11>1 2026-01-17T00:00:00+00:00 OPNsense hostwatch 30715 - [meta sequenceId="1"] 2026-01-17T00:00:00.000015Z ERROR hostwatch: Error reading from capture: libpcap error: The interface disappeared
Other than hostwatch (which I just disabled for the time being) my OPNsense is running smooth as ever.
Ah, I just found out about https://github.com/opnsense/hostwatch/issues/8 (https://github.com/opnsense/hostwatch/issues/8).
I'll leave here following:
top -S -m io -o total
last pid: 25126; load averages: 0.49, 0.44, 0.46 up 0+01:26:59 20:11:30
143 processes: 3 running, 138 sleeping, 2 waiting
CPU: 9.4% user, 0.0% nice, 4.7% system, 0.9% interrupt, 84.9% idle
Mem: 1154M Active, 1634M Inact, 1585M Wired, 880M Buf, 7477M Free
Swap: 8192M Total, 8192M Free
PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND
34081 hostd 2494 13 0 2496 0 2496 99.96% hostwatch
16 root 500 0 0 1 0 1 0.04% bufdaemon
85376 root 0 0 0 0 0 0 0.00% php-cgi
53184 root 10 1 0 0 0 0 0.00% php
1 root 0 0 0 0 0 0 0.00% init
74881 root 0 0 0 0 0 0 0.00% php-cgi
79105 root 0 0 0 0 0 0 0.00% csh
35073 root 0 0 0 0 0 0 0.00% php-cgi
68801 root 0 0 0 0 0 0 0.00% php-cgi
2 root 33 0 0 0 0 0 0.00% clock
5314 root 2 2 0 0 0 0 0.00% ng_queue
1474 squid 0 0 0 0 0 0 0.00% security_file_certg
And also this:
fsck_ffs -n /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** SU+J Recovering /dev/gpt/rootfs
USE JOURNAL? no
Skipping journal, falling through to full fsck
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
UNALLOCATED I=1122920 OWNER=hostd MODE=100644
SIZE=21032 MTIME=Jan 16 19:04 2026
FILE=/var/db/hostwatch/hosts.db-journal
UNEXPECTED SOFT UPDATE INCONSISTENCY
Second command's result worries me a bit, as I've just reinstalled Opnsense on fresh (and partitioned by installer) VM disk and right away I've got this kind of errors???
These typical UFS issues mainly are from unclean shutdowns regardless of where the corruption occurs.
If you have a process that is writing while the power goes off there will be an error for it. And if the process is writing all the time the chances are pretty high it's going to catch that one.
Cheers,
Franco
Quote from: franco on January 17, 2026, 07:48:43 AMWe're recording the last MAC address for any IPv4 and IPv6 we see. If the MAC changes that's considered a "movement". In some environments this happens very rapidly and thus the service constantly registers the changes.
OK, thanks. This is just for virtual interface setup/teardown? Interesting. The behavior sounds... less than ideal. Not something I (and apparently others) would expect.
(The overall issue reminds me to always control information rate, unless a spew is intended.)
Quote from: franco on January 17, 2026, 02:31:13 PMThese typical UFS issues mainly are from unclean shutdowns regardless of where the corruption occurs.
If you have a process that is writing while the power goes off there will be an error for it. And if the process is writing all the time the chances are pretty high it's going to catch that one.
It is freshly installed system (UFS, because of VM's disk is on ZFS) with restored config and updated to 25.7.11. It didn't experience any unclean shutdowns... BTW, I couldn't finalized installation (was stuck at "preparing target system") if I imported config during installation. I had to install Opnsense, configure lan interface and restore config using webgui...
How to disable loggin of hostwatch or can it be disable?
Interfaces: Neighbors: Automatic Discovery
In my opinion, the plugin is useless; it should work like this:
https://github.com/netalertx/NetAlertX
Your opinion does not see the full picture, the hostwatch daemon was mostly designed to discover IPv6 addresses, so that for example an IPv6 only captive portal is possible in the future.
Any consumer that needs a full picture of IPv6 addresses now has a fast sqlite database to work with.
The other things that happen with the data can still be fleshed out over time.
I also had higher memory consumption with an increasing swap file size as well. Noticed that hostwatch was enable on all interfaces, as such it was logging IPs on the WAN interface as well. From my understanding this is targeted at keeping track of IPs on your local network, so shouldn't the WAN interface be excluded by default?
Odd thing is I disabled the service, selected all interfaces except WAN and re-enabled the service. However now the service refuses to start and nothing in the hostwatch logs says why. The system log reports: [meta sequenceId="2"] <6>[132593] pid 22972 (hostwatch), jid 0, uid 0: exited on signal 6 (no core dump - bad address)
So for now I just left it disabled since it seems to provide relatively little value.
@xpendable
https://github.com/opnsense/hostwatch/issues/13
Quote from: Monviech (Cedrik) on January 17, 2026, 06:41:23 PM[...] the hostwatch daemon was mostly designed to discover IPv6 addresses, so that for example an IPv6 only captive portal is possible in the future.
Any consumer that needs a full picture of IPv6 addresses now has a fast sqlite database to work with.
The other things that happen with the data can still be fleshed out over time.
Ok, it just clicked.
Can this also track IPv6 temporary addresses? If it will become possible to create a host alias that updates dynamically from ND data, or even update DNS so that firewall/unbound logs resolve to a host using privacy addresses...
Tempering my excitement :)
Yes it can track temporary addresses because of this:
https://github.com/opnsense/hostwatch/issues/2
Nice! 👍
I don't follow GitHub that closely and totally missed this.
Updated to 25.7.11_2.
Unfortunately there's only slight difference in i/o demand created by hostwatch.
top -S -m io -o total
last pid: 8428; load averages: 0.39, 0.41, 0.37 up 2+17:07:59 11:52:30
144 processes: 2 running, 140 sleeping, 2 waiting
CPU: 5.1% user, 0.0% nice, 2.9% system, 0.5% interrupt, 91.5% idle
Mem: 343M Active, 6483M Inact, 1559M Wired, 670M Buf, 3881M Free
Swap: 8192M Total, 8192M Free
PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND
92104 hostd 2707 37 0 2640 0 2640 99.21% hostwatch
7034 root 53 54 0 19 0 19 0.71% python3.11
16 root 548 1 0 2 0 2 0.08% bufdaemon
1 root 0 0 0 0 0 0 0.00% init
97153 unbound 47 1 0 0 0 0 0.00% unbound
2 root 56 0 0 0 0 0 0.00% clock
5314 root 12 6 0 0 0 0 0.00% ng_queue
1474 squid 0 0 0 0 0 0 0.00% security_file_certg
3 root 0 0 0 0 0 0 0.00% crypto
10115 root 0 0 0 0 0 0 0.00% ge
iostat -x 2
extended device statistics
device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b
da0 0 105 7.7 4048.1 0 1 0 1 0 0
da1 0 0 0.0 0.0 0 0 0 0 0 0
Plus I wasn't able to start hostwatch when I handpicked interfaces, works only when "All" is selected. When trying to start it in cli:
service hostwatch restart
hostwatch not running? (check /var/run/hostwatch/hostwatch.pid).
Starting hostwatch.
thread 'main' (116664) panicked at src/main.rs:53:79:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Abort trap
/usr/local/etc/rc.d/hostwatch: WARNING: failed to start hostwatch
It seems all issues have now been recorded in https://github.com/opnsense/hostwatch/issues and don't need to be reposted.
Let's give it a bit of slack to be resolved.
Cheers,
Franco