OPNsense Forum

English Forums => 25.7, 25.10 Series => Topic started by: tessus on January 17, 2026, 03:54:15 PM

Title: [SOLVED] hostwatch at 100% CPU
Post by: tessus on January 17, 2026, 03:54:15 PM
I've updated to 25.7.11_1 (from 25.7.10) 1 or 2 days ago. A few minutes ago the hostwatch process went haywire and stayed at 100%. I also saw that syslog-ng was at about 80%. Then I killed the hostwatch process and everything went back to normal.

What is this process and what does it do? Is it a new service and what could it make to jump to 100% CPU all of a sudden?

Solution (https://forum.opnsense.org/index.php?msg=257185)
Title: Re: hostwatch at 100% CPU
Post by: tessus on January 17, 2026, 03:59:02 PM
Ok, in the release notes I can see:

This release brings the new host discovery service which resolves and remembers MAC addresses for IPv4 and IPv6 hosts in your connected networks and provides this data for the firewall MAC aliases and captive portal clients. It is now enabled by default, but you can choose to opt out by disabling the automatic discovery option.

Well, my questions still remain. What could it make to jump to 100% CPU all of a sudden? Also, I am not sure what this actually does. MAC addresses are "remembered" in the ARP cache, so why do I need this service? What is going to be worse (perf, functionality, ...) when I do not use this service?
Title: Re: hostwatch at 100% CPU
Post by: Patrick M. Hausen on January 17, 2026, 04:01:33 PM
Nothing is going to be worse, just disable it.

Interfaces: Neighbors: Automatic Discovery

It fills in a missing feature people coming from consumer routers like Fritzbox got used to and frequently demanded: show an overview of all devices in my network.
Title: Re: hostwatch at 100% CPU
Post by: bycarlsjr on January 17, 2026, 07:05:25 PM
Quote from: Patrick M. Hausen on January 17, 2026, 04:01:33 PMNothing is going to be worse, just disable it.

Interfaces: Neighbors: Automatic Discovery

It fills in a missing feature people coming from consumer routers like Fritzbox got used to and frequently demanded: show an overview of all devices in my network.

More useless garbage that we didn't ask for..... Why can't this be a plugin that those folks can install separately and not brick our routers.... I have a 16Gig hostwatch log this morning, lose gui, forced to restart to recover...  Definitely not a professional group here....
Title: Re: hostwatch at 100% CPU
Post by: s1l3nce on January 17, 2026, 07:43:37 PM
Quote from: bycarlsjr on January 17, 2026, 07:05:25 PM
Quote from: Patrick M. Hausen on January 17, 2026, 04:01:33 PMNothing is going to be worse, just disable it.

Interfaces: Neighbors: Automatic Discovery

It fills in a missing feature people coming from consumer routers like Fritzbox got used to and frequently demanded: show an overview of all devices in my network.

More useless garbage that we didn't ask for..... Why can't this be a plugin that those folks can install separately and not brick our routers.... I have a 16Gig hostwatch log this morning, lose gui, forced to restart to recover...  Definitely not a professional group here....

Yep, that new feature broke my WebUI because it filled up the storage completely ( /var/log/hostwatch/hostwatch_20260116.log was more than 100 Gigs).

People reporting high CPU usage with this update is probably related to this also.
Here is the explanation -> https://github.com/opnsense/hostwatch/issues/8
Title: Re: hostwatch at 100% CPU
Post by: cinergi on January 18, 2026, 12:26:36 AM
I noticed that since upgrading, the disk access LED on my Opnsense box flashes much more often than before the upgrade - like every 2 seconds or so.  I disabled the Host Discovery feature and it stopped immediately.  I'm not sure what this feature needs to access the disk so much for (my log file was only 40KB after running for a few hours), but this seems like a great way to wear out an SSD.  I agree this feature should be disabled by default; those who need it can enable it themselves.
Title: Re: hostwatch at 100% CPU
Post by: tessus on January 18, 2026, 12:59:08 AM
Quote from: Patrick M. Hausen on January 17, 2026, 04:01:33 PMNothing is going to be worse, just disable it.

Interfaces: Neighbors: Automatic Discovery

It fills in a missing feature people coming from consumer routers like Fritzbox got used to and frequently demanded: show an overview of all devices in my network.

Thank you very much. This explains it perfectly. Not the 100% CPU usage, but what this thing actually is. I checked the docs, but I couldn't find anything related to the new UI form. e.g. there should also be an "exclude interfaces" field. I have over 20 interfaces (mostly VLANs), and 4 WAN interfaces. I certainly don't need discovery on the WAN interfaces, but the current UI requires me to select all internal interfaces. This is rather tedious.

Your explanation (answer) is exactly what should be in the documentation. ;-)

Title: Re: hostwatch at 100% CPU
Post by: crlt on January 18, 2026, 03:58:59 PM
Quote from: bycarlsjr on January 17, 2026, 07:05:25 PM
Quote from: Patrick M. Hausen on January 17, 2026, 04:01:33 PMNothing is going to be worse, just disable it.

Interfaces: Neighbors: Automatic Discovery

It fills in a missing feature people coming from consumer routers like Fritzbox got used to and frequently demanded: show an overview of all devices in my network.

More useless garbage that we didn't ask for..... Why can't this be a plugin that those folks can install separately and not brick our routers.... I have a 16Gig hostwatch log this morning, lose gui, forced to restart to recover...  Definitely not a professional group here....

I don't think that's fair to say as it was a popular request. I believe that it's not a plugin because it's developed by the opnsense team and you can simply disable it. With all that said it probably could have shipped disabled by default.

I manage a few personal firewalls across a few locations and I always read the change log and forums before updating so I knew to look out for this potential issue. Perhaps you should consider doing that in the future.
Title: Re: hostwatch at 100% CPU
Post by: bycarlsjr on January 18, 2026, 05:10:27 PM
Quote from: crlt on January 18, 2026, 03:58:59 PM
Quote from: bycarlsjr on January 17, 2026, 07:05:25 PM
Quote from: Patrick M. Hausen on January 17, 2026, 04:01:33 PMNothing is going to be worse, just disable it.

Interfaces: Neighbors: Automatic Discovery

It fills in a missing feature people coming from consumer routers like Fritzbox got used to and frequently demanded: show an overview of all devices in my network.

More useless garbage that we didn't ask for..... Why can't this be a plugin that those folks can install separately and not brick our routers.... I have a 16Gig hostwatch log this morning, lose gui, forced to restart to recover...  Definitely not a professional group here....

I don't think that's fair to say as it was a popular request. I believe that it's not a plugin because it's developed by the opnsense team and you can simply disable it. With all that said it probably could have shipped disabled by default.

I manage a few personal firewalls across a few locations and I always read the change log and forums before updating so I knew to look out for this potential issue. Perhaps you should consider doing that in the future.

No, it's completely fair to say. Anything that potentially trades stability for features should not be allowed to be enabled as a default in a mainline release, ever. For that point, no new features should be enabled by default. Bugs happen, I get that, but with 26 around the corner who releases new features on possibly the last release of a given train!

Title: Re: hostwatch at 100% CPU
Post by: aperezva on January 18, 2026, 05:33:07 PM
Any recomendation to update or not? Finally this hostwatch situation is a issue or normal behaviour? 

Its not normal to see that access to disk increase in this way.

No make sense ti add a new service when more of us will disable it. It would be better to have the option to enable it.
Title: Re: hostwatch at 100% CPU
Post by: zakaron on January 18, 2026, 05:57:39 PM
Something definitely went wrong somewhere. Just curious, what size environment is this installed in? I just installed the update in my home network yesterday and it has discovered 43 unique MACs. I've left the default settings alone on it. The log from yesterday is 25KB and only 223B from today.


When running "top" from the CLI, I did notice the hostwatch process near the top of the list with typical usage of 0.08% to 0.15% with occasional spikes to 0.35%
Still less that a whole percent, but still more than most processes.

EDIT: I should have read the post further down regarding similar issues. It has more info in there:  https://forum.opnsense.org/index.php?topic=50405.0 (https://forum.opnsense.org/index.php?topic=50405.0)
Title: Re: hostwatch at 100% CPU
Post by: jp0469 on January 19, 2026, 01:16:09 AM
Quote from: bycarlsjr on January 17, 2026, 07:05:25 PM
Quote from: Patrick M. Hausen on January 17, 2026, 04:01:33 PMNothing is going to be worse, just disable it.

Interfaces: Neighbors: Automatic Discovery

It fills in a missing feature people coming from consumer routers like Fritzbox got used to and frequently demanded: show an overview of all devices in my network.

More useless garbage that we didn't ask for..... Why can't this be a plugin that those folks can install separately and not brick our routers.... I have a 16Gig hostwatch log this morning, lose gui, forced to restart to recover...  Definitely not a professional group here....
You should definitely demand a refund. Be sure to draw attention to your post count so the devs know who they're dealing with.
Title: Re: hostwatch at 100% CPU
Post by: Panther2698 on January 19, 2026, 05:19:39 AM
The fix was to disable automatic discovery as Partrick mentioned. The patch was definitely not professional. This was a very easy mistake to miss but completely avoidable, hopefully future patches dont brick and/or break our routers.... (Don't enable random features by default)
Title: Re: hostwatch at 100% CPU
Post by: tessus on January 19, 2026, 07:55:17 AM
Ok, I think this topic is done/solved.

I suspect that the service has a bug somewhere which causes the CPU to run amok. It is a pretty new service after all and it is easy to deactivate it.

Personally I don't need the service, but I understand that people have requested this feature. Who knows, maybe I will activate it again in the future. I am certain the devs will fix the current issues over time.

Patrick's explanation (https://forum.opnsense.org/index.php?msg=257002) what the service does exactly was very helpful.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: Patrick M. Hausen on January 19, 2026, 09:21:01 AM
@franco I just noticed the hotfix. What about making the "changed ethernet address" messages configurable? We already have Interfaces: Settings: ARP Handling. I have that enabled although I do not have multiple interfaces in the same broadcast domain. Apple TVs do weird things with power saving states and proxy ARP ;-)
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: franco on January 19, 2026, 09:38:27 AM
Interfaces: Settings: ARP Handling is dead, long live the two tunables :)

But yes something similar needs to be done. We'll have a coordination meeting later about it and try to work through the reported items.


Cheers,
Franco
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: troplin on January 19, 2026, 11:24:33 AM
Maybe I just don't understand it completely but those movement messages seem odd to me.

In my log I see the same message over and over again:
INFO hostwatch: changed ethernet address host 00:1d:63:63:eb:35 moved from 64:62:66:22:44:8c to fe80::6662:66ff:fe22:448c at igc1
The message makes no sense to me, how can it move from a MAC address to an IPv6 address? Could there be a confusion of variables in the code?

Also the fact that it is the same message over and over again kind of makes me a bit suspicious that there might be an error in the code where the previous and current address are compared, or that the state isn't updated correctly.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: Taomyn on January 19, 2026, 11:31:06 AM
I woke up this morning to my firewall on 100% cpu after upgrading to 25.7.11_1 last night, and a reboot seemed to fix it but then I came across this thread and also the _2 hotfix which I have now applied. I have tried to configure it to limit to only my LAN interface, but anything other than All keeps the service running. In the logs I see:
2026-01-19T11:18:57Noticekernel<6>[18592] pid 77296 (hostwatch), jid 0, uid 0: exited on signal 6 (no core dump - bad address)
2026-01-19T11:18:56Noticeroot/usr/local/etc/rc.d/hostwatch: WARNING: failed to start hostwatch

For now I have just disabled it.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: franco on January 19, 2026, 12:24:56 PM
@troplin that's fixed, but the message has been disabled for now to avoid excessive logging

@Taomyn it's in the list of things to fix this week


Cheers,
Franco
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: bycarlsjr on January 19, 2026, 03:11:15 PM
Quote from: jp0469 on January 19, 2026, 01:16:09 AMYou should definitely demand a refund. Be sure to draw attention to your post count so the devs know who they're dealing with.

I'm glad you looked at your own..... don't be an ass
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: s1l3nce on January 20, 2026, 03:12:34 AM
I've just applied the latest patch, rebooted and for a while it was all good. After an hour of usage, suddenly I notice very high CPU usage on the hostwatch service.

(https://i.imgur.com/UskuFcu.png)

I'm just stopping this service until they figure it out because it's clearly giving a lot of issues at the moment.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: amarek on January 20, 2026, 08:14:11 AM
THX for this thread, this service was eating all my memory. after disabling it the usage was immediately at 28%, what a great solution to roll this out for all as fix implemented and started service............
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: aperezva on January 20, 2026, 08:27:56 AM
@franco, What´s your recomendation, I´m in 25.7.10, wait till all the issues wil be solved? Wait 26.1?

Thanks for your efforts and support.

BR

Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: d0shie on January 20, 2026, 09:14:42 AM
Quote from: amarek on January 20, 2026, 08:14:11 AMTHX for this thread, this service was eating all my memory. after disabling it the usage was immediately at 28%, what a great solution to roll this out for all as fix implemented and started service............
I was away from home and thankfully only the firewall's Web UI became non-functional, so I could still do remote SSH and diagnose the problem. For me the new service silently ate up 52GB of space for logging alone in less than 2 days and somewhat stalled the system as a result. I even read the changelog and noticed it but didn't think much at the time.
So, it's one of those blunders with an unexpectedly high impact, yes, but it's rare. And they did promptly push out hotfixes to remedy the issue on reasonably short notice.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: troplin on January 20, 2026, 09:39:00 AM
@franco Even if the log message has been fixed (and is now disabled), it still makes no sense:

Firstly,
00:1d:63:63:eb:35 is the MAC address of my dishwasher and
64:62:66:22:44:8c is the LAN interface of the OPNsense box itself. The IPv6 address is the link-local address of the OPNsense box.
So why would hostwatch think that the LLA of the OPNsense box itself has been previously used by my dishwasher?

Secondly, the message is always just ,,host X moved from A to B", shouldn't the database be updated to reflect that after the fist time? There are no messages the opposite way, i.e. ,,host X moved from B to A".

I still believe that the logging issue is just a symptom of the actual problem, e.g. you're somehow comparing the wrong addresses.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: troplin on January 20, 2026, 10:02:48 AM
@franco maybe this line?
https://github.com/opnsense/hostwatch/blob/3000f8f6611c098a7e7d01eaa0253b31c6af9ca3/src/database.rs#L141

Shouldn't that be
when real_ether_address = excluded.real_ether_addressInstead of
when ether_address = excluded.real_ether_address?

EDIT:
Also, this condition is always true:
https://github.com/opnsense/hostwatch/blob/3000f8f6611c098a7e7d01eaa0253b31c6af9ca3/src/lib.rs#L196
Because in the SQL statement prev_ethernet_address is only updated when ethernet_address actually changes.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: Monviech (Cedrik) on January 20, 2026, 10:18:50 AM
@troplin best track coding things in the repository in an issue.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: troplin on January 20, 2026, 10:23:15 AM
Quote from: Monviech (Cedrik) on January 20, 2026, 10:18:50 AM@troplin best track coding things in the repository in an issue.

Ok, can do that. Is there already one that fits or should I create a new one?
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: Monviech (Cedrik) on January 20, 2026, 10:23:52 AM
Create a new one and point out the issue that you think exist, that way it can be evaluated and potentially fixed. Thank you.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: troplin on January 20, 2026, 10:51:18 AM
https://github.com/opnsense/hostwatch/issues/21

Hope that works for you. I'm sick (and bored) in my bed and typing on my phone, not the ideal environment for programming stuff...
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: LHoust on January 21, 2026, 04:37:25 AM
I was also observing hostwatch running at nearly 100% CPU...

I Proof my OPNsense Updates within a VirtualBox VM running on an Ubuntu 24.04.3 Host, where my /home Partition sits on a HDD...

With 25.7.11_2 I was also observing a Serious Level of HDD Thrashing, until I disabled Automatic Discovery!!!
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: passeri on January 21, 2026, 05:25:27 AM
Just to mention, on one test box and two operational boxes, all bare metal Intel and AMD, hostwatch trots along quietly with no untoward CPU spikes or log writes. Three principal subnets (no vlans), all IPv4, around 25 devices.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: LHoust on January 21, 2026, 06:26:53 AM
Quote from: LHoust on January 21, 2026, 04:37:25 AMI was also observing hostwatch running at nearly 100% CPU...

I Proof my OPNsense Updates within a VirtualBox VM running on an Ubuntu 24.04.3 Host, where my /home Partition sits on a HDD...

With 25.7.11_2 I was also observing a Serious Level of HDD Thrashing, until I disabled Automatic Discovery!!!

For Proofing Updates, I also run OPNsense 25.7 within a Workstation VMPlayer VM: Host is Windows 11 (C: Drive is a SSD)...

Everthing seems "Quite" HERE??
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: iMx on January 21, 2026, 09:40:05 AM
I updated from 25.1.x -> 25.7.11_2-amd64 and whilst I didn't see the logs/disk usage growing (due to the _2 hot fix), having automatic discovery did lead to increased writes for me.

... not massive, but unnecessary in my view.  Screenshot attached - you can see when I disabled it, just after 16:00.  Auto Neighbour Discovery is unnecessary for my usage.

Personal preference would be that this is disabled by default, but it seems like I'll just need to remember to disable it on new builds/installs now!

For anyone that is curious, you can use iostat 1 (UFS) or zpool iostat -v 1 (zfs)
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: EHRETic on January 21, 2026, 02:57:08 PM
Hi there,

My 5 cents, also had this issue with 25.7.11_2 : it filed up one my FW hard disk in less than an hour.

What's relevant in my case (or weird) : I have a pair of FWs in HA mode and this morning, I did a rule update that I synced with the passive node and only the passive node started to fill up the HD after I did the sync.

I think I updated to  25.7.11_2 the day of publishing and I saw no problem until today.

Weird no? 😉

PS: what should I clean to get some space back?
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: Patrick M. Hausen on January 21, 2026, 02:58:19 PM
Quote from: EHRETic on January 21, 2026, 02:57:08 PMPS: what should I clean to get some space back?

/var/log/hostwatch/*
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: EHRETic on January 21, 2026, 03:24:15 PM
Quote from: Patrick M. Hausen on January 21, 2026, 02:58:19 PM/var/log/hostwatch/*

Thanks a lot for this super fast answer, this helped a lot! 😊
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: LHoust on January 21, 2026, 03:29:06 PM
Quote from: LHoust on January 21, 2026, 06:26:53 AM
Quote from: LHoust on January 21, 2026, 04:37:25 AMI was also observing hostwatch running at nearly 100% CPU...

I Proof my OPNsense Updates within a VirtualBox VM running on an Ubuntu 24.04.3 Host, where my /home Partition sits on a HDD...

With 25.7.11_2 I was also observing a Serious Level of HDD Thrashing, until I disabled Automatic Discovery!!!

For Proofing Updates, I also run OPNsense 25.7 within a Workstation VMPlayer VM: Host is Windows 11 (C: Drive is a SSD)...

Everthing seems "Quite" HERE??

"Update on the 50 MB/s Thrashing: I have isolated the trigger. The issue occurs only when the VirtualBox WAN adapter is set to 'Bridged Adapter' on a host with multiple virtual interfaces (Tailscale, VMware hooks). When switched to 'NAT', the thrashing stops.

This suggests Hostwatch is attempting to perform neighbor discovery on the host's 'shadow' virtual interfaces visible through the bridge, failing, and entering a high-frequency SQLite write loop. Since VirtualBox defaults to NAT, most users are likely bypassing this bug by accident."
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: franco on January 22, 2026, 07:31:32 AM
Here's an updated version of hostwatch we're also consider shipping in a hotfix based on user feedback:

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/hostwatch-1.0.6.pkg

Apply once from the GUI under Interfaces: Neighbors: Automatic Discovery to restart with the new binary.

To go back to the latest shipped version just issue this command:

# opnsense-revert -r 25.7.11_2 hostwatch

And reapply again from the GUI.


Cheers,
Franco
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: drclaw on January 22, 2026, 10:30:56 AM
Ok I just applied the hotfix above to my OPNsense 25.7.11_1-amd64 VM running on Proxmox - my VM was impacted by this and filled up ~4gb of logs in /var/log/hostwatch in the last few days.

I have re-enabled Neighbours: Automatic Discovery - things seem ok, but I will keep an eye on it.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: Patrick M. Hausen on January 22, 2026, 10:41:28 AM
Running 1.0.6 now. Limiting interfaces works. No problems so far.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: OPNenthu on January 22, 2026, 10:53:53 AM
Installed 1.0.6 also.

The previous trigger on my system that caused it to spike the CPU was when I toggled some WG instances.  Did the same just now and it didn't cause hostwatch to go haywire.  Just some errors logged, but otherwise CPU is idling.

2026-01-22T09:32:32.553239Z ERROR hostwatch: Error reading from capture: libpcap error: The interface disappeared, exit wg2
2026-01-22T09:32:32.533231Z ERROR hostwatch: Error reading from capture: libpcap error: The interface disappeared, exit wg1
2026-01-22T09:32:32.513240Z ERROR hostwatch: Error reading from capture: libpcap error: The interface disappeared, exit wg0
2026-01-22T09:23:31.644128Z WARN hostwatch: Failed to initialize capture for device: pfsync0
2026-01-22T09:23:31.643592Z WARN hostwatch: Failed to initialize capture for device: usbus0

I won't comment on disk writes because I don't know what my baseline is.  I'll just say that I see some writes (nda0), but the hostwatch log is <8 kB.

root@firewall:/var/log/hostwatch # iostat -x
                        extended device statistics 
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b 
mmcsd0         0       0      0.2      0.0     0     0     0     0    0   0
mmcsd0bo       0       0      0.0      0.0     0     0     0     0    0   0
mmcsd0bo       0       0      0.0      0.0     0     0     0     0    0   0
nda0           2       4     38.6    109.3     0     0     2     0    0   0
pass0          0       0      0.0      0.0     0     0     0     0    0   0

root@firewall:/var/log/hostwatch # ls -l
total 8
-rw-------  1 root wheel 7941 Jan 22 04:39 hostwatch_20260122.log

Couple questions:

- Where is the sqlite db?
- There was previously a 'latest.log' symlink in this directory which is now gone.  Can we not rely on that symlink being there for Monit test purposes?

Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: OPNenthu on January 22, 2026, 11:06:16 AM
The symlink appeared again after some time :)

root@firewall:/var/log/hostwatch # ls -l
total 8
-rw-------  1 root wheel 7941 Jan 22 04:39 hostwatch_20260122.log
lrwxr-x---  1 root wheel   41 Jan 22 05:01 latest.log -> /var/log/hostwatch/hostwatch_20260122.log
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: franco on January 22, 2026, 11:19:26 AM
Thanks for testing! <3

sqlite db is /var/db/hostwatch/hosts.db

latest symlink is added by a cron job or

# configctl syslog archive


Cheers,
Franco
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: iMx on January 22, 2026, 12:03:51 PM
Installed the latest version: hostwatch-1.0.6.pkg

Writes, for me, seem to be more or less the same.  Not clear whether this is just 'how things will be' with this service enabled. 

But, for completeness, screenshot added again.  Set to 'All' interfaces which is:

Initialized 21 packet device_captures
If I filter out the various VPNs, WAN - cable modem on the WAN, so can be noisy on the front end - it's then 'Initialized 11 packet device_captures', writes roughly half - as should probably be expected.

... with only a handful of interfaces, or only a handful enabled for this service, the writes are probably negligible. 

But the writes do seem to be constant when monitoring with 'zpool iostat -v 1' - for me, whilst it is an Enterprise SSD, I think I can live without the convenience this service is designed to bring.

Not seeing any signs of logs etc growing in size, nor CPU spikes.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: meyergru on January 22, 2026, 12:59:59 PM
I now installed 1.0.6 and re-enabled the service again. It works and CPU is not at 100%, but there is a hefty number of disk writes in bursts every minute and also, the SQLite db journal file is deleted after each transaction data batch has been committed. Using rollback capability in such a way incurs a huge penalty on writes, especially on ZFS.

Just a guess here: But I think that SQL rollback capability is not needed for this purpose and when I glanced at the hostwatch code (https://github.com/opnsense/hostwatch/blob/master/src/database.rs#L74), I found that the journaling mode (https://www.sqlite.org/pragma.html#pragma_journal_mode) is WAL, where other modes (like MEMORY or even OFF) might be more appropriate.


On a side note: I had some strange effects with the old version of the service - namely, that my own ping-based discovery tool suddenly had entries for every IP in the subnet active. Don't ask, IDK why or how this happened. I just disabled the service for the time being.

Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: franco on January 22, 2026, 01:38:43 PM
https://github.com/opnsense/hostwatch/commit/482b45ce is on the way but not in 1.0.6.

For specific issues it may make sense to raise a ticket, but multiple versions are in flight now so it would be better to wait for the final one that's going into 26.1 to make reports on.


Cheers,
Franco
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: LHoust on January 22, 2026, 07:02:06 PM
I tried hostwatch-1.0.6.pkg, although I am still experiencing HDD Thrashing...

Restored hostwatch 1.0.5 from 25.7.11_2 and therefore I will run with Automatic Discovery disabled for now...
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: LHoust on January 22, 2026, 07:24:59 PM
Quote from: LHoust on January 22, 2026, 07:02:06 PMI tried hostwatch-1.0.6.pkg, although I am still experiencing HDD Thrashing...

Restored hostwatch 1.0.5 from 25.7.11_2 and therefore I will run with Automatic Discovery disabled for now...

"I ran iostat -w 1 da0 and the data is conclusive. While throughput is low (~0.15 MB/s), the tps (Transactions Per Second) is constantly spiking between 20 and 30 with Hostwatch 1.0.6.

For a mechanical HDD, this high frequency of tiny 6KB writes forces constant head seeking. This confirms the issue isn't the amount of data being logged, but the frequency of the SQLite commits. I'll be keeping discovery OFF until a version with Batch/Lazy Writing is released."
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: LisaMT on January 23, 2026, 02:47:30 AM
My system crashed today.  Thanks for all the notes on here.  I disabled the hostwatch and logs and it's running again. 
There should be some indicator on the dashboard that monitors disk usage better.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: Cljackhammer on January 23, 2026, 08:34:15 PM
I think that it might be beneficial to remove this capability. The issue with the disk writes is a major problem and introduces instability. I had it enabled and the disk writes are higher than my production elastic search instance by a factor 700%
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: iMx on January 23, 2026, 09:24:45 PM
I think there's definitely an argument for it to be disabled by default.

... I'm not sure I need to put unnecessary wear on an SSD for this.

I'd have thought that most Business Edition customers will disable it and they bring in the money!
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: OPNenthu on January 24, 2026, 10:42:56 AM
Quote from: franco on January 22, 2026, 07:31:32 AMHere's an updated version of hostwatch we're also consider shipping in a hotfix based on user feedback:

# pkg add -f https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/misc/hostwatch-1.0.6.pkg

[...]

I rebooted my 25.7.11_2 install that was already running this v1.0.6 patch for the last ~2 days and on startup I noticed this in the console:

Starting hostwatch.
Starting acme_http_challenge.
>>> Invoking start script 'syslog'
>>> Invoking start script 'carp'
>>> Invoking start script 'cron'
Starting Cron: OK


>>> Invoking start script 'hostwatch'
hostwatch already running?  (pid=56724).
>>> Error in start script '90-hostwatch'

Hostwatch is up and running, though.  Looks OK.

And the contents of that file:

root@firewall:~ # cat /usr/local/etc/rc.syshook.d/start/90-hostwatch
#!/bin/sh

# work around mysterious first-time-only segfault
/usr/local/etc/rc.d/hostwatch start
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: franco on January 24, 2026, 11:10:53 AM
@OPNenthu https://github.com/opnsense/hostwatch/issues/7

@iMx might be room for disabling it on Nano images and adding a wizard option for it. In any case we'll add a migration note. Disabling this manually is not an inconvenience IMO.


Cheers,
Franco
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: nero355 on January 25, 2026, 12:02:38 AM
Quote from: franco on January 24, 2026, 11:10:53 AMDisabling this manually is not an inconvenience IMO.
Considering the fact that it's a very "sensitive" piece of software at the moment I would like to suggest disabling it in all the images !!

Also please make sure it's not enabled after any of the updates or has some sub-process running/is running partially basically...
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: franco on January 25, 2026, 09:25:17 AM
No, that's not something we're considering at the moment.


Cheers,
Franco
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: myg63 on January 30, 2026, 10:35:17 AM
Hello guys,
I had this problem with lot of Disk IO writes because of hostwatch checking whatever I don't know.
I updated to 26.1 yesterday and it became worse!!!
hostwatch --version: 1.0.2

hostwatch fills the disk completely in a few minutes with files in /var/db/hostwatch

After stopping hostwatch and deleting the files and restarting hostwatch it fills the disk again.

-rw-r--r--  1 hostd hostd     4272128 Jan 30 10:17 hosts.db
-rw-r--r--  1 hostd hostd    23396352 Jan 30 10:21 hosts.db-shm
-rw-r--r--  1 hostd hostd 35686871496 Jan 30 10:00 hosts.db-wal

My environment:
proxmox host with wan coming thru a vlan (tagging is done in the proxmox host) and a virtual opnsense
with
2 physical adapters: LAN and WAN running on a proxmox network bridge connected to physical adapters on the host.
And there is a wireguard network with about 20 hosts connecting thru the WAN into a virtual net which is routed to the LAN.


I stopped the daemon

hopefully there will be a solution soon or any advice here?
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: Patrick M. Hausen on January 30, 2026, 10:57:58 AM
Do you need hostwatch?
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: franco on January 30, 2026, 11:30:31 AM
We've testing 1.0.11 now which fixes some auto-cleanups that were not cleaning up...

# opnsense-revert -z hostwatch
# service hostwatch restart


Cheers,
Franco
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: myg63 on January 30, 2026, 12:30:04 PM
I think I could solve it.
1. NO, I don't need hostwatch.
2. I checked the reason and this thread.
In the end, I installed the 1.0.9 version and removed watching WAN interfaces, only LAN
PROBLEM SOLVED!

Fun fact: when I switch into the shell and enter "hostwatch --version" it shows 1.0.2 even 1.0.9 packet is installed. May be there is some little work needed to remove this chance of misunderstanding when trying to check if correct version is installed.



Quote from: myg63 on January 30, 2026, 10:35:17 AMHello guys,
I had this problem with lot of Disk IO writes because of hostwatch checking whatever I don't know.
I updated to 26.1 yesterday and it became worse!!!
hostwatch --version: 1.0.2

hostwatch fills the disk completely in a few minutes with files in /var/db/hostwatch

After stopping hostwatch and deleting the files and restarting hostwatch it fills the disk again.

-rw-r--r--  1 hostd hostd     4272128 Jan 30 10:17 hosts.db
-rw-r--r--  1 hostd hostd    23396352 Jan 30 10:21 hosts.db-shm
-rw-r--r--  1 hostd hostd 35686871496 Jan 30 10:00 hosts.db-wal

My environment:
proxmox host with wan coming thru a vlan (tagging is done in the proxmox host) and a virtual opnsense
with
2 physical adapters: LAN and WAN running on a proxmox network bridge connected to physical adapters on the host.
And there is a wireguard network with about 20 hosts connecting thru the WAN into a virtual net which is routed to the LAN.


I stopped the daemon

hopefully there will be a solution soon or any advice here?
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: tessus on January 30, 2026, 12:36:08 PM
Quote from: myg63 on January 30, 2026, 12:30:04 PMwhen I switch into the shell and enter "hostwatch --version" it shows 1.0.2 even 1.0.9 packet is installed.

Yep, the manifest was not updated: see https://github.com/opnsense/hostwatch/blob/1.0.9/Cargo.toml

I believe the release workflow is not yet stabilized. I will open an issue to make a few suggestions.
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: franco on January 30, 2026, 12:54:33 PM
It was updated in .11 https://github.com/opnsense/hostwatch/commit/5f35418
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: 0xd on January 30, 2026, 07:37:32 PM
I wasn't sure if the update clears the database so here is an attempt to save someone some time:

Web UI: System->Settings->Administration:
Check:
Enable Secure Shell
Permit root user login
Permit password login
Apply

ssh root@<yourgateway>

root@OPNsense:/var/db/hostwatch # ls -lh
total 207598624
-rw-r--r--  1 hostd hostd  4.0M Jan 30 17:30 hosts.db
-rw-r--r--  1 hostd hostd  393M Jan 30 18:05 hosts.db-shm
-rw-r--r--  1 hostd hostd  198G Jan 30 18:05 hosts.db-wal

service hostwatch stop
rm -rf /var/db/hostwatch/*

Update to OPNsense 26.1_4:
exit
12) Update from console

Web UI: System->Settings->Administration:
Uncheck:
Enable Secure Shell
Permit root user login
Permit password login
Apply
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: franco on January 30, 2026, 08:29:47 PM
The .11 in 26.1_4 enforces the proper cleanup now. Just make sure to restart after update.


Cheers,
Franco
Title: Re: [SOLVED] hostwatch at 100% CPU
Post by: hugo on February 04, 2026, 08:02:42 PM
I'm not really clear if it's expected that 1.0.11 should fix hostwatch filling up disks from the sqlite DB records. I installed 26.1 fresh yesterday, patched up to 26.1_4, which I understand runs hostwatch 1.0.11. hostwatch still filled up my entire disk (~20 GB of disk space used) in under 6 hours.

Our provider (coax/DOCSIS) makes the WAN *extremely* chatty at ~1,200 received ARP requests per second. I've disabled hostwatch to avoid this causing me issues again. Blanket enabling hostwatch by default on host interfaces is really risky to users, imho, unless its disk usage can be constrained.