[SOLVED] hostwatch at 100% CPU

[LHoust]

I was also observing hostwatch running at nearly 100% CPU...

I Proof my OPNsense Updates within a VirtualBox VM running on an Ubuntu 24.04.3 Host, where my /home Partition sits on a HDD...

With 25.7.11_2 I was also observing a Serious Level of HDD Thrashing, until I disabled Automatic Discovery!!!

[passeri]

Just to mention, on one test box and two operational boxes, all bare metal Intel and AMD, hostwatch trots along quietly with no untoward CPU spikes or log writes. Three principal subnets (no vlans), all IPv4, around 25 devices.

[LHoust]

I was also observing hostwatch running at nearly 100% CPU...

I Proof my OPNsense Updates within a VirtualBox VM running on an Ubuntu 24.04.3 Host, where my /home Partition sits on a HDD...

With 25.7.11_2 I was also observing a Serious Level of HDD Thrashing, until I disabled Automatic Discovery!!!

For Proofing Updates, I also run OPNsense 25.7 within a Workstation VMPlayer VM: Host is Windows 11 (C: Drive is a SSD)...

Everthing seems "Quite" HERE??

[iMx]

I updated from 25.1.x -> 25.7.11_2-amd64 and whilst I didn't see the logs/disk usage growing (due to the _2 hot fix), having automatic discovery did lead to increased writes for me.

... not massive, but unnecessary in my view.  Screenshot attached - you can see when I disabled it, just after 16:00.  Auto Neighbour Discovery is unnecessary for my usage.

Personal preference would be that this is disabled by default, but it seems like I'll just need to remember to disable it on new builds/installs now!

For anyone that is curious, you can use iostat 1 (UFS) or zpool iostat -v 1 (zfs)

[EHRETic]

Hi there,

My 5 cents, also had this issue with 25.7.11_2 : it filed up one my FW hard disk in less than an hour.

What's relevant in my case (or weird) : I have a pair of FWs in HA mode and this morning, I did a rule update that I synced with the passive node and only the passive node started to fill up the HD after I did the sync.

I think I updated to  25.7.11_2 the day of publishing and I saw no problem until today.

Weird no? 😉

PS: what should I clean to get some space back?

[Patrick M. Hausen]

PS: what should I clean to get some space back?

Code Select Expand

/var/log/hostwatch/*


[EHRETic]

Code Select Expand

/var/log/hostwatch/*


Thanks a lot for this super fast answer, this helped a lot! 😊

[LHoust]

I was also observing hostwatch running at nearly 100% CPU...

I Proof my OPNsense Updates within a VirtualBox VM running on an Ubuntu 24.04.3 Host, where my /home Partition sits on a HDD...

With 25.7.11_2 I was also observing a Serious Level of HDD Thrashing, until I disabled Automatic Discovery!!!

For Proofing Updates, I also run OPNsense 25.7 within a Workstation VMPlayer VM: Host is Windows 11 (C: Drive is a SSD)...

Everthing seems "Quite" HERE??

"Update on the 50 MB/s Thrashing: I have isolated the trigger. The issue occurs only when the VirtualBox WAN adapter is set to 'Bridged Adapter' on a host with multiple virtual interfaces (Tailscale, VMware hooks). When switched to 'NAT', the thrashing stops.

This suggests Hostwatch is attempting to perform neighbor discovery on the host's 'shadow' virtual interfaces visible through the bridge, failing, and entering a high-frequency SQLite write loop. Since VirtualBox defaults to NAT, most users are likely bypassing this bug by accident."

[franco]

Here's an updated version of hostwatch we're also consider shipping in a hotfix based on user feedback:

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/hostwatch-1.0.6.pkg

Apply once from the GUI under Interfaces: Neighbors: Automatic Discovery to restart with the new binary.

To go back to the latest shipped version just issue this command:

# opnsense-revert -r 25.7.11_2 hostwatch

And reapply again from the GUI.


Cheers,
Franco

[drclaw]

Ok I just applied the hotfix above to my OPNsense 25.7.11_1-amd64 VM running on Proxmox - my VM was impacted by this and filled up ~4gb of logs in /var/log/hostwatch in the last few days.

I have re-enabled Neighbours: Automatic Discovery - things seem ok, but I will keep an eye on it.

[Patrick M. Hausen]

Running 1.0.6 now. Limiting interfaces works. No problems so far.

[OPNenthu]

Installed 1.0.6 also.

The previous trigger on my system that caused it to spike the CPU was when I toggled some WG instances.  Did the same just now and it didn't cause hostwatch to go haywire.  Just some errors logged, but otherwise CPU is idling.

Code Select Expand

2026-01-22T09:32:32.553239Z ERROR hostwatch: Error reading from capture: libpcap error: The interface disappeared, exit wg2
2026-01-22T09:32:32.533231Z ERROR hostwatch: Error reading from capture: libpcap error: The interface disappeared, exit wg1
2026-01-22T09:32:32.513240Z ERROR hostwatch: Error reading from capture: libpcap error: The interface disappeared, exit wg0
2026-01-22T09:23:31.644128Z WARN hostwatch: Failed to initialize capture for device: pfsync0
2026-01-22T09:23:31.643592Z WARN hostwatch: Failed to initialize capture for device: usbus0

I won't comment on disk writes because I don't know what my baseline is.  I'll just say that I see some writes (nda0), but the hostwatch log is <8 kB.

Code Select Expand

root@firewall:/var/log/hostwatch # iostat -x
                        extended device statistics 
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b 
mmcsd0         0       0      0.2      0.0     0     0     0     0    0   0
mmcsd0bo       0       0      0.0      0.0     0     0     0     0    0   0
mmcsd0bo       0       0      0.0      0.0     0     0     0     0    0   0
nda0           2       4     38.6    109.3     0     0     2     0    0   0
pass0          0       0      0.0      0.0     0     0     0     0    0   0


Code Select Expand

root@firewall:/var/log/hostwatch # ls -l
total 8
-rw-------  1 root wheel 7941 Jan 22 04:39 hostwatch_20260122.log

Couple questions:

- Where is the sqlite db?
- There was previously a 'latest.log' symlink in this directory which is now gone.  Can we not rely on that symlink being there for Monit test purposes?

[OPNenthu]

The symlink appeared again after some time :)

Code Select Expand

root@firewall:/var/log/hostwatch # ls -l
total 8
-rw-------  1 root wheel 7941 Jan 22 04:39 hostwatch_20260122.log
lrwxr-x---  1 root wheel   41 Jan 22 05:01 latest.log -> /var/log/hostwatch/hostwatch_20260122.log


[franco]

Thanks for testing! <3

sqlite db is /var/db/hostwatch/hosts.db

latest symlink is added by a cron job or

# configctl syslog archive


Cheers,
Franco

[iMx]

Installed the latest version: hostwatch-1.0.6.pkg

Writes, for me, seem to be more or less the same.  Not clear whether this is just 'how things will be' with this service enabled. 

But, for completeness, screenshot added again.  Set to 'All' interfaces which is:

Code Select Expand

Initialized 21 packet device_captures
If I filter out the various VPNs, WAN - cable modem on the WAN, so can be noisy on the front end - it's then 'Initialized 11 packet device_captures', writes roughly half - as should probably be expected.

... with only a handful of interfaces, or only a handful enabled for this service, the writes are probably negligible. 

But the writes do seem to be constant when monitoring with 'zpool iostat -v 1' - for me, whilst it is an Enterprise SSD, I think I can live without the convenience this service is designed to bring.

Not seeing any signs of logs etc growing in size, nor CPU spikes.