26.7.9_1 kernel panic in mana(4) on Azure MANA hosts - boot loop Environment
Symptom Since updating to 26.7.9_1 (installed 16 Nov), any boot on a MANA host panics ~13s in, during "Setting up extended sysctls...", then reboots and repeats indefinitely. 2,779 consecutive panics across two outages (713 on 18 Nov, 2,066 between 19/11 and 20/11); 0 panics on any other kernel/VF combination.
Panic
What the serial console log shows
Possibly related
Workaround in place (not yet confirmed on a MANA host) Tunable hint.mana.0.disabled=1 (if_mana is a module, auto-loaded by devmatch on MANA hosts). Will update once it survives a MANA boot.
Questions
- OPNsense 26.7.9_1, FreeBSD 15.1-RELEASE-p6, kernel stable/26.7-n284285-1633399de9dd
- Azure VM Standard_D2s_v5, built from the FreeBSD Foundation freebsd-14_1 image via the opnazure (dmauser) bootstrap
- Two NICs (hn0 WAN / hn1 LAN). Accelerated Networking is disabled on both NICs in Azure, but the guest still receives a VF depending on host: mana0 on MANA hosts, mce0/mce1 (Mellanox) on others
- VM is deallocated nightly and started at 07:00, so host/VF type varies day to day
Symptom Since updating to 26.7.9_1 (installed 16 Nov), any boot on a MANA host panics ~13s in, during "Setting up extended sysctls...", then reboots and repeats indefinitely. 2,779 consecutive panics across two outages (713 on 18 Nov, 2,066 between 19/11 and 20/11); 0 panics on any other kernel/VF combination.
Panic
Code Select
Fatal trap 12: page fault while in kernel mode fault virtual address = 0x70 current process = sysctl ... mana_sysctl_tx_stat_agg_u64()
at mana_sysctl_tx_stat_agg_u64+0xd0 sysctl_root_handler_locked()
... userland_sysctl() ...
sys___sysctl() ... [full trace attached] What the serial console log shows
- n284285 booted normally on Mellanox hosts (mce0) on 16 and 17 Nov
- Every n284285 boot on a MANA host (mana0) panicked with the trace above. None succeeded
- Older kernels (n284159, p4, and n284176, p6) booted cleanly on MANA hosts many times, including a live mana0 detach/reattach
- A plain restart stays on the same host and keeps looping. Stop/deallocate + start moved the VM to a Mellanox host, and the same kernel booted fine
Possibly related
- 26.7.9 changelog: "system: invalidate sysctl cache on apply". The panic happens in the sysctl apply step at boot
- The 26.7.2 boot loops were attributed to reading counters on an interface without allocated counters. This looks like the same class of bug, but in mana(4)
Workaround in place (not yet confirmed on a MANA host) Tunable hint.mana.0.disabled=1 (if_mana is a module, auto-loaded by devmatch on MANA hosts). Will update once it survives a MANA boot.
Questions
- Is this a known regression between n284176 and n284285?
- Is the hint.mana.0.disabled workaround safe/recommended until a fix lands?
"