"
About once a day (10 times in 6 days) the packet-processor worker stops sending heartbeats for ~21 s, the watchdog sends SIGTERM, the worker needs another 15–36 s to die (ends with signal 3), and during that time the kernel logs netmap_transmit igc1 full and traffic through the interface stalls (users notice a 30–60 s internet hiccup). All 10 kills happened in minute :10–:14 of an hour. I can't find what the worker is doing when it hangs. Looking for ideas / known issues / a way to get a useful trace.
Environment
- OPNsense 26.7.4_1 (series 26.7, regular update repo 26.7/latest; FreeBSD 15.1-RELEASE-p3)
- Intel Celeron N5100 (4 cores), 7.8 GB RAM, Intel I226-V (igc), XGS-PON ~900/900 Mbit
- Zenarmor Home, stable release channel, latest version (no update pending): os-sensei 2.6.2, os-sensei-agent 2.6.1, engine "Release 2.6.2 built at Jul 17 2026"
- One instance: netmap@igc1 (LAN uplink to the switch, routed mode, 1 worker, no CPU pinning)
- dev.netmap.admode=2 (generic/emulated; I tried native and went back), buf_size=4096 (2048 gave drop packet size 2345 > 2048), buf_num=1000000, ring_num=1024, ring_size=36864
- Exclusions: 56 MAC addresses + 6 IP nets (incl. my UniFi management VLAN and UniFi OS server) To try to lower the load as a test, Watchdog enabled=true, timeout=20
- Use case: home network, Zenarmor only to block malicious sites, botnets/C2 and malware — no content/app-control policies
- Typical load is low: ~200 UDP pps outside 443 (measured), load average ~0.5–1.3, 0 kernel drops in a normal minute
Example (last one, times CEST)
20:12:00 worker0 log: last line (normal: DnsEnrichmentManager::setDNSCacheItem WARN ...)
20:12:28 ERR [WatchDogManager::checkHeartBeats] Watchdog: Worker
20:12:46–20:13:04 kernel: netmap_transmit igc1 full ... (2 lines/s, ~19 s)
20:13:04 CRIT [EastpectWorker::WaitWorkers] Child worker0 ... terminated with signal: 3
20:13:04 CRIT [EastpectWorker::WaitWorkers] Child worker0, new [native_pid: 82907] re-spawned.
20:13:04 kernel: pid 92412 (eastpect), jid 0, uid 0: exited on signal 3 (no core dump - denied by kern.coredump)
20:13:04–20:13:19 kernel: generic_netmap_unregister/dtor/attach ... "Emulated adapter for igc1" deactivated → created → activated
What I found
- 10 watchdog kills between 16 Sep and 21 Sep. All in minute :10–:14 (4× :10, 2× :11, 1× each :12/:13/:14, ...). Not every hour, roughly 1–2 per day.
- TERM → dead takes 1–36 s and always ends with signal 3 (the worker doesn't react to SIGTERM). Every slow one (≥10 s) produced netmap_transmit igc1 full bursts; fast ones sometimes.
- Heartbeat is lost 13–29 s before the kill. The last worker0 log lines before the silence are ordinary (DnsEnrichmentManager mismatch WARNs, TLS ClientHello parse, LibhtpParser "httpInfo is null" WARN) — no error, no crash.
- It is not a traffic spike: during a 26-drop burst I captured only 467 UDP packets in 13 s.
- Not correlated (checked against the logs, at chance level): cloud_agent SyncPolicyUpdate (runs every ~2 min) and scheduledServiceControl (~every 4 min). Zenarmor's own hourly check_license.py runs at :10:00 but I have no evidence that it is involved.
- Earlier I also had bursts of UDPConnectionFlow::connect ... UDP flow connect failed: Address already in use / EastpectInstance::onAcceptUDP New UDP flow connect failed (98% in minute :09–:12) from one server VM; excluding that host removed those errors, but the worker hangs remain.
Questions
1. Is this a known issue in 2.6.2 (or with emulated netmap on igc/I226-V)? What in the worker can block for 20+ s?
2. Is there a supported way to get a stack/trace of the hung worker (core dumps are denied by kern.coredump; what would you like me to enable)?
3. Why does the worker ignore SIGTERM for 15–36 s?
4. Would bypassEnable/bypassModeOn let traffic pass while the worker is hung/being killed? (Both are false now, I could not find documentation on the exact behaviour.)
5. Is it safe to change the watchdog timeout (20 s), or does that just make the stall longer?
I can provide main_*.log/worker0_*.log excerpts for any of the events. For now I have stopped the engine (configctl zenarmor service eastpect stop) to confirm the household hiccups disappear without it
Environment
- OPNsense 26.7.4_1 (series 26.7, regular update repo 26.7/latest; FreeBSD 15.1-RELEASE-p3)
- Intel Celeron N5100 (4 cores), 7.8 GB RAM, Intel I226-V (igc), XGS-PON ~900/900 Mbit
- Zenarmor Home, stable release channel, latest version (no update pending): os-sensei 2.6.2, os-sensei-agent 2.6.1, engine "Release 2.6.2 built at Jul 17 2026"
- One instance: netmap@igc1 (LAN uplink to the switch, routed mode, 1 worker, no CPU pinning)
- dev.netmap.admode=2 (generic/emulated; I tried native and went back), buf_size=4096 (2048 gave drop packet size 2345 > 2048), buf_num=1000000, ring_num=1024, ring_size=36864
- Exclusions: 56 MAC addresses + 6 IP nets (incl. my UniFi management VLAN and UniFi OS server) To try to lower the load as a test, Watchdog enabled=true, timeout=20
- Use case: home network, Zenarmor only to block malicious sites, botnets/C2 and malware — no content/app-control policies
- Typical load is low: ~200 UDP pps outside 443 (measured), load average ~0.5–1.3, 0 kernel drops in a normal minute
Example (last one, times CEST)
20:12:00 worker0 log: last line (normal: DnsEnrichmentManager::setDNSCacheItem WARN ...)
20:12:28 ERR [WatchDogManager::checkHeartBeats] Watchdog: Worker
- failed to send heartbeat for 21 seconds
20:12:46–20:13:04 kernel: netmap_transmit igc1 full ... (2 lines/s, ~19 s)
20:13:04 CRIT [EastpectWorker::WaitWorkers] Child worker0 ... terminated with signal: 3
20:13:04 CRIT [EastpectWorker::WaitWorkers] Child worker0, new [native_pid: 82907] re-spawned.
20:13:04 kernel: pid 92412 (eastpect), jid 0, uid 0: exited on signal 3 (no core dump - denied by kern.coredump)
20:13:04–20:13:19 kernel: generic_netmap_unregister/dtor/attach ... "Emulated adapter for igc1" deactivated → created → activated
What I found
- 10 watchdog kills between 16 Sep and 21 Sep. All in minute :10–:14 (4× :10, 2× :11, 1× each :12/:13/:14, ...). Not every hour, roughly 1–2 per day.
- TERM → dead takes 1–36 s and always ends with signal 3 (the worker doesn't react to SIGTERM). Every slow one (≥10 s) produced netmap_transmit igc1 full bursts; fast ones sometimes.
- Heartbeat is lost 13–29 s before the kill. The last worker0 log lines before the silence are ordinary (DnsEnrichmentManager mismatch WARNs, TLS ClientHello parse, LibhtpParser "httpInfo is null" WARN) — no error, no crash.
- It is not a traffic spike: during a 26-drop burst I captured only 467 UDP packets in 13 s.
- Not correlated (checked against the logs, at chance level): cloud_agent SyncPolicyUpdate (runs every ~2 min) and scheduledServiceControl (~every 4 min). Zenarmor's own hourly check_license.py runs at :10:00 but I have no evidence that it is involved.
- Earlier I also had bursts of UDPConnectionFlow::connect ... UDP flow connect failed: Address already in use / EastpectInstance::onAcceptUDP New UDP flow connect failed (98% in minute :09–:12) from one server VM; excluding that host removed those errors, but the worker hangs remain.
Questions
1. Is this a known issue in 2.6.2 (or with emulated netmap on igc/I226-V)? What in the worker can block for 20+ s?
2. Is there a supported way to get a stack/trace of the hung worker (core dumps are denied by kern.coredump; what would you like me to enable)?
3. Why does the worker ignore SIGTERM for 15–36 s?
4. Would bypassEnable/bypassModeOn let traffic pass while the worker is hung/being killed? (Both are false now, I could not find documentation on the exact behaviour.)
5. Is it safe to change the watchdog timeout (20 s), or does that just make the stall longer?
I can provide main_*.log/worker0_*.log excerpts for any of the events. For now I have stopped the engine (configctl zenarmor service eastpect stop) to confirm the household hiccups disappear without it
