OPNsense kernel panics regularly; reports tons of nvme events; no SMART errors

[MattD76]

hi,
for a couple of days, my opnsense device fails regularly, most often at around midnight.
Current software: OPNsense 25.7.10; Zenarmor installed. Logs show loads of nvme0 async event occured messages, with kernel panic ultimately happening. If the nvme messages and the kernel panic are related, I am not sure, but at least it is suspicious. nvmecontrol does not show any errors; SMART utilities and tests do not report any errors either.

Does anyone have any idea? The hardware is relatively new, 12 months old approximately, it's one of those N100 integrated devices with four NIC's. Is it time to replace already, or just add a different drive?

Thanks in advance!


dmesg at around the last crash:

...
[959833] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[959834] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[959842] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[959847] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[959848] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[959848] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[959849] kernel trap 12 with interrupts disabled
[959849]
[959849]
[959849] Fatal trap 12: page fault while in kernel mode
[959849] cpuid = 0; apic id = 00
[959849] fault virtual address  = 0x0
[959849] fault code             = supervisor read instruction, page not present
[959849] instruction pointer    = 0x20:0x0
[959849] stack pointer          = 0x28:0xfffffe00af74fcf8
[959849] frame pointer          = 0x28:0x0
[959849] code segment           = base 0x0, limit 0xfffff, type 0x1b
[959849]                        = DPL 0, pres 1, long 1, def32 0, gran 1
[959849] processor eflags       = resume, IOPL = 0
[959849] current process                = 0 (netlink_socket (PID)
[959849] rdi: 000000005f40b000 rsi: fffff8003df69600 rdx: 8000000000000000
[959849] rcx: 8000000131c017fd  r8: fffff80115e46520  r9: 0000000000002819
[959849] rax: fffffe008d6411c0 rbx: 0000000000000000 rbp: 0000000000000000
[959849] r10: 0000000000001388 r11: 00000000b92ccfd3 r12: fffff80115e36740
[959849] r13: fffff80115e46000 r14: ffffffff81d825d0 r15: 0000000000000000
[959849] trap number            = 12
[959849] panic: page fault
[959849] cpuid = 0
[959849] time = 1767394903
[959849] KDB: stack backtrace:
[959849] db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00af74fa40
[959849] vpanic() at vpanic+0x161/frame 0xfffffe00af74fb70
[959849] panic() at panic+0x43/frame 0xfffffe00af74fbd0
[959849] trap_pfault() at trap_pfault+0x3da/frame 0xfffffe00af74fc20
[959849] calltrap() at calltrap+0x8/frame 0xfffffe00af74fc20
[959849] --- trap 0xc, rip = 0, rsp = 0xfffffe00af74fcf8, rbp = 0 ---
[959849] KDB: enter: panic
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.3-RELEASE-p7 stable/25.7-n271786-8d37907deeb8 SMP amd64
FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)
[1] VT(vga): resolution 640x480
[1] CPU: Intel(R) N100 (806.40-MHz K8-class CPU)
[1]   Origin="GenuineIntel"  Id=0xb06e0  Family=0x6  Model=0xbe  Stepping=0
[1]   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2
,SS,HTT,TM,PBE>
[1]   Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVB
E,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
[1]   AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
[1]   AMD Features2=0x121<LAHF,ABM,Prefetch>
[1]   Structured Extended Features=0x239ca7eb<FSGSBASE,TSCADJ,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,NFPUSG,PQE,RDSEED,ADX,SMAP,CLF
LUSHOPT,CLWB,PROCTRACE,SHA>
[1]   Structured Extended Features2=0x98c007bc<UMIP,PKU,OSPKE,WAITPKG,GFNI,VAES,VPCLMULQDQ,RDPID,MOVDIRI,MOVDIR64B>
[1]   Structured Extended Features3=0xfc184410<FSRM,MD_CLEAR,IBT,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
[1]   XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
[1]   IA32_ARCH_CAPS=0x180fd6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO,TAA_NO>
[1]   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
[1]   TSC: P-state invariant, performance statistics
[1] real memory  = 8589934592 (8192 MB)
[1] avail memory = 7969275904 (7600 MB)
[1] Event timer "LAPIC" quality 600
[1] ACPI APIC Table: <ALASKA A M I >
[1] WARNING: L3 data cache covers more APIC IDs than a package (7 > 3)
[1] FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
[1] FreeBSD/SMP: 1 package(s) x 4 core(s)
[1] random: registering fast source Intel Secure Key RNG
[1] random: fast provider: "Intel Secure Key RNG"
[1] random: unblocking device.
[1] ioapic0 <Version 2.0> irqs 0-119
[1] Launching APs: 1 3 2
[1] random: entropy device external interface
[1] wlan: mac acl policy registered
[1] kbd0 at kbdmux0
[1] WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0.
[1] efirtc0: <EFI Realtime Clock>
[1] efirtc0: registered as a time-of-day clock, resolution 1.000000s
[1] smbios0: <System Management BIOS> at iomem 0x73c77000-0x73c77017
[1] smbios0: Entry point: v3 (64-bit), Version: 3.6
[1] aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
[1] acpi0: <ALASKA A M I >
[1] Firmware Error (ACPI): Could not resolve symbol [\134_SB.PC00.TXHC.RHUB.SS01], AE_NOT_FOUND (20221020/dswload2-315)
[1] ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20221020/psobject-372)
[1] acpi0: Power Button (fixed)
[1] hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
[1] Timecounter "HPET" frequency 19200000 Hz quality 950
[1] Event timer "HPET" frequency 19200000 Hz quality 550
[1] Event timer "HPET1" frequency 19200000 Hz quality 440
[1] Event timer "HPET2" frequency 19200000 Hz quality 440
[1] Event timer "HPET3" frequency 19200000 Hz quality 440
[1] Event timer "HPET4" frequency 19200000 Hz quality 440
[1] atrtc1: <AT realtime clock> on acpi0
[1] atrtc1: Warning: Couldn't map I/O.
[1] atrtc1: registered as a time-of-day clock, resolution 1.000000s
[1] Event timer "RTC" frequency 32768 Hz quality 0
[1] attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
[1] Timecounter "i8254" frequency 1193182 Hz quality 0
[1] Event timer "i8254" frequency 1193182 Hz quality 100
[1] Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
[1] acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
[1] pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
[1] pci0: <ACPI PCI bus> on pcib0
[1] vgapci0: <VGA-compatible display> port 0x3000-0x303f mem 0x6000000000-0x6000ffffff,0x4000000000-0x400fffffff at device 2.0 on pci0
[1] vgapci0: Boot video device
[1] xhci0: <XHCI (generic) USB 3.0 controller> mem 0x6001100000-0x600110ffff at device 20.0 on pci0
[1] xhci0: 32 bytes context size, 64-bit DMA
[1] usbus0 on xhci0
[1] usbus0: 5.0Gbps Super Speed USB v3.0
[1] pci0: <memory, RAM> at device 20.2 (no driver attached)
[1] ahci0: <AHCI SATA controller> port 0x3090-0x3097,0x3080-0x3083,0x3060-0x307f mem 0x80d00000-0x80d01fff,0x80d03000-0x80d030ff,0x80d02000-0x80d027ff at device 23.0 on pci0
[1] ahci0: AHCI v1.31 with 1 6Gbps ports, Port Multiplier not supported
[1] ahcich0: <AHCI channel> at channel 0 on ahci0
[1] sdhci_pci0: <Generic SD HCI> mem 0x6001119000-0x6001119fff at device 26.0 on pci0
[1] sdhci_pci0: 1 slot(s) allocated
[1] mmc0: <MMC/SD bus> on sdhci_pci0
[1] pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
[1] pci1: <ACPI PCI bus> on pcib1
[1] igc0: <Intel(R) Ethernet Controller I226-V> mem 0x80a00000-0x80afffff,0x80b00000-0x80b03fff at device 0.0 on pci1
[1] igc0: EEPROM V2.14-0 eTrack 0x80000290
[1] igc0: Using 1024 TX descriptors and 1024 RX descriptors
[1] igc0: Using 4 RX queues 4 TX queues
[1] igc0: Using MSI-X interrupts with 5 vectors
[1] igc0: Ethernet address: 60:be:b4:16:b7:38
[1] igc0: netmap queues/slots: TX 4/1024, RX 4/1024
[1] pcib2: <ACPI PCI-PCI bridge> at device 28.3 on pci0
[1] pci2: <ACPI PCI bus> on pcib2
[1] igc1: <Intel(R) Ethernet Controller I226-V> mem 0x80800000-0x808fffff,0x80900000-0x80903fff at device 0.0 on pci2
[1] igc1: EEPROM V2.14-0 eTrack 0x80000290
[1] igc1: Using 1024 TX descriptors and 1024 RX descriptors
[1] igc1: Using 4 RX queues 4 TX queues
[1] igc1: Using MSI-X interrupts with 5 vectors
[1] igc1: Ethernet address: 60:be:b4:16:b7:39
[1] igc1: netmap queues/slots: TX 4/1024, RX 4/1024
[1] pcib3: <ACPI PCI-PCI bridge> at device 28.6 on pci0
[1] pci3: <ACPI PCI bus> on pcib3
[1] igc2: <Intel(R) Ethernet Controller I226-V> mem 0x80600000-0x806fffff,0x80700000-0x80703fff at device 0.0 on pci3
[1] igc2: EEPROM V2.14-0 eTrack 0x80000290
[1] igc2: Using 1024 TX descriptors and 1024 RX descriptors
[1] igc2: Using 4 RX queues 4 TX queues
[1] igc2: Using MSI-X interrupts with 5 vectors
[1] igc1: Ethernet address: 60:be:b4:16:b7:39
[1] igc1: netmap queues/slots: TX 4/1024, RX 4/1024
[1] pcib3: <ACPI PCI-PCI bridge> at device 28.6 on pci0
[1] pci3: <ACPI PCI bus> on pcib3
[1] igc2: <Intel(R) Ethernet Controller I226-V> mem 0x80600000-0x806fffff,0x80700000-0x80703fff at device 0.0 on pci3
[1] igc2: EEPROM V2.14-0 eTrack 0x80000290
[1] igc2: Using 1024 TX descriptors and 1024 RX descriptors
[1] igc2: Using 4 RX queues 4 TX queues
[1] igc2: Using MSI-X interrupts with 5 vectors
[1] igc2: Ethernet address: 60:be:b4:16:b7:3a
[1] igc2: netmap queues/slots: TX 4/1024, RX 4/1024
[1] pcib4: <ACPI PCI-PCI bridge> at device 29.0 on pci0
[1] pci4: <ACPI PCI bus> on pcib4
[1] igc3: <Intel(R) Ethernet Controller I226-V> mem 0x80400000-0x804fffff,0x80500000-0x80503fff at device 0.0 on pci4
[1] igc3: EEPROM V2.14-0 eTrack 0x80000290
[1] igc3: Using 1024 TX descriptors and 1024 RX descriptors
[1] igc3: Using 4 RX queues 4 TX queues
[1] igc3: Using MSI-X interrupts with 5 vectors
[1] igc3: Ethernet address: 60:be:b4:16:b7:3b
[1] igc3: netmap queues/slots: TX 4/1024, RX 4/1024
[1] pcib5: <ACPI PCI-PCI bridge> at device 29.3 on pci0
[1] pci5: <ACPI PCI bus> on pcib5
[1] nvme0: <Generic NVMe Device> mem 0x80c00000-0x80c03fff at device 0.0 on pci5
[1] isab0: <PCI-ISA bridge> at device 31.0 on pci0
[1] isa0: <ISA bus> on isab0
[1] hdac0: <Intel Alder Lake-N HDA Controller> mem 0x6001110000-0x6001113fff,0x6001000000-0x60010fffff at device 31.3 on pci0
[1] pci0: <serial bus> at device 31.5 (no driver attached)
[1] acpi_button0: <Sleep Button> on acpi0
[1] cpu0: <ACPI CPU> on acpi0
[1] acpi_button1: <Power Button> on acpi0
[1] acpi_tz0: <Thermal Zone> on acpi0
[1] ns8250: UART FCR is broken
[1] ns8250: UART FCR is broken
[1] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
[1] ns8250: UART FCR is broken
[1] uart0: console (115200,n,8,1)
[1] ns8250: UART FCR is broken
[1] ns8250: UART FCR is broken
[1] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
[1] acpi_syscontainer0: <System Container> on acpi0
[1] acpi_syscontainer1: <System Container> on acpi0
[1] atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
[1] atrtc0: Warning: Couldn't map I/O.
[1] atrtc0: registered as a time-of-day clock, resolution 1.000000s
[1] atrtc0: Can't map interrupt.
[1] hwpstate_intel0: <Intel Speed Shift> on cpu0
[1] cpufreq0: <CPU frequency control> on cpu0
[1] hwpstate_intel1: <Intel Speed Shift> on cpu1
[1] cpufreq1: <CPU frequency control> on cpu1
[1] hwpstate_intel2: <Intel Speed Shift> on cpu2
[1] cpufreq2: <CPU frequency control> on cpu2
[1] hwpstate_intel3: <Intel Speed Shift> on cpu3
[1] cpufreq3: <CPU frequency control> on cpu3
[1] Timecounter "TSC" frequency 806414197 Hz quality 1000
[1] Timecounters tick every 1.000 msec
[1] ugen0.1: <Intel XHCI root HUB> at usbus0
[2] uhub0 on usbus0
[2] uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
[2] ZFS filesystem version: 5
[2] ZFS storage pool version: features support (5000)
[2] mmc0: No compatible cards found on bus
[2] nvme0: Allocated 64MB host memory buffer
[2] nda0 at nvme0 bus 0 scbus1 target 0 lun 1
nda0: <ShiJi 128GB M.2-NVMe T0709A3 AA000000000000000062>
nda0: Serial Number AA000000000000000062
[2] nda0: nvme version 1.3
nda0: 122104MB (250069680 512 byte sectors)
[2] Trying to mount root from zfs:zroot/ROOT/default []...
[3] uhub0: 16 ports with 16 removable, self powered
[4] pid 31 (zpool) is attempting to use unsafe AIO requests - not logging anymore
[7] igc0: link state changed to UP
[7] igc1: link state changed to UP
[7] igc2: link state changed to UP
[7] igc3: link state changed to UP
[20] ichsmb0: <Intel Alder Lake SMBus controller> port 0xefa0-0xefbf mem 0x6001118000-0x60011180ff at device 31.4 on pci0
[20] smbus0: <System Management Bus> on ichsmb0
[21] acpi_wmi0: <ACPI-WMI mapping> on acpi0
[21] acpi_wmi0: cannot find EC device
[21] acpi_wmi0: Embedded MOF found
[21] ACPI: \134_SB.WFDE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
[21] acpi_wmi1: <ACPI-WMI mapping> on acpi0
[21] acpi_wmi1: cannot find EC device
[21] acpi_wmi1: Embedded MOF found
[21] ACPI: \134_SB.WFTE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
[23] lo0: link state changed to UP
[26] igc0: link state changed to DOWN
[26] vlan0: changing name to 'vlan0.1'
[26] igc1: link state changed to DOWN
[26] vlan1: changing name to 'vlan0.10'
[26] igc2: link state changed to DOWN
[26] vlan2: changing name to 'vlan0.100'
[26] vlan3: changing name to 'vlan0.20'
[26] vlan4: changing name to 'vlan0.30'
[26] vlan5: changing name to 'vlan0.40'
[27] igc3: link state changed to DOWN
[27] vlan6: changing name to 'vlan07'
[27] ng0: changing name to 'pppoe0'
[30] igc1: link state changed to UP
[30] vlan0.20: link state changed to UP
[30] vlan0.10: link state changed to UP
[30] vlan0.30: link state changed to UP
[30] igc2: link state changed to UP
[30] vlan0.100: link state changed to UP
[30] igc0: link state changed to UP
[30] vlan0.1: link state changed to UP
[30] vlan0.40: link state changed to UP
[33] igc3: link state changed to UP
[33] vlan07: link state changed to UP
[45] tun0: changing name to 'zen0'
[49] tun1: link state changed to UP
[49] tun1: changing name to 'tailscale0'
[59] 481.320610 [1167] generic_netmap_attach     Emulated adapter for vlan0.40 created (prev was NULL)
[59] 481.330346 [1072] generic_netmap_dtor       Emulated netmap adapter for vlan0.40 destroyed
[59] 481.339404 [1167] generic_netmap_attach     Emulated adapter for vlan0.40 created (prev was NULL)
[60] 481.791594 [ 319] generic_netmap_register   Emulated adapter for vlan0.40 activated
[60] 481.873288 [1167] generic_netmap_attach     Emulated adapter for vlan0.20 created (prev was NULL)
[60] 481.883426 [1072] generic_netmap_dtor       Emulated netmap adapter for vlan0.20 destroyed
[60] 481.892538 [1167] generic_netmap_attach     Emulated adapter for vlan0.20 created (prev was NULL)
[60] 481.902443 [ 319] generic_netmap_register   Emulated adapter for vlan0.20 activated
[60] 481.912969 [1167] generic_netmap_attach     Emulated adapter for vlan0.10 created (prev was NULL)
[60] 481.922689 [1072] generic_netmap_dtor       Emulated netmap adapter for vlan0.10 destroyed
[60] 481.931919 [1167] generic_netmap_attach     Emulated adapter for vlan0.10 created (prev was NULL)
[60] 481.941645 [ 319] generic_netmap_register   Emulated adapter for vlan0.10 activated
[60] 481.951301 [1167] generic_netmap_attach     Emulated adapter for igc2 created (prev was igc2)
[60] 481.960669 [1068] generic_netmap_dtor       Native netmap adapter for igc2 restored
[60] 481.969103 [1072] generic_netmap_dtor       Emulated netmap adapter for igc2 destroyed
[60] 481.977932 [1167] generic_netmap_attach     Emulated adapter for igc2 created (prev was igc2)
[60] 481.988265 [ 319] generic_netmap_register   Emulated adapter for igc2 activated
[61] 482.443239 [1167] generic_netmap_attach     Emulated adapter for vlan0.30 created (prev was NULL)
[61] 482.452961 [1072] generic_netmap_dtor       Emulated netmap adapter for vlan0.30 destroyed
[61] 482.462070 [1167] generic_netmap_attach     Emulated adapter for vlan0.30 created (prev was NULL)
[61] 482.486784 [ 319] generic_netmap_register   Emulated adapter for vlan0.30 activated
[61] 482.495174 [1167] generic_netmap_attach     Emulated adapter for igc3 created (prev was igc3)
[61] 482.504418 [1068] generic_netmap_dtor       Native netmap adapter for igc3 restored
[61] 482.512781 [1072] generic_netmap_dtor       Emulated netmap adapter for igc3 destroyed
[61] 482.521475 [1167] generic_netmap_attach     Emulated adapter for igc3 created (prev was igc3)
[61] 482.530847 [ 319] generic_netmap_register   Emulated adapter for igc3 activated
[77] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[78] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[80] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[10672] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[10674] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[10676] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[14198] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[14199] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[14201] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[14206] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[14214] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
....
[28648] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[28660] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[28663] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[28674] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[28685] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[28703] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[28707] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[28716] nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
[33604] nvme0: GET_LOG_PAGE (02) sqid:0 cid:10 nsid:ffffffff cdw10:03ff00c1 cdw11:00000000
[33604] nvme0: INVALID LOG PAGE (01/09) crd:0 m:0 dnr:0 p:1 sqid:0 cid:10 cdw0:0
[33607] nvme0: GET_LOG_PAGE (02) sqid:0 cid:10 nsid:ffffffff cdw10:03ff0000 cdw11:00000000
[33607] nvme0: INVALID LOG PAGE (01/09) crd:0 m:0 dnr:0 p:1 sqid:0 cid:10 cdw0:0
root@firewall:/var/log/system # nvmecontrol logpage -p 2 nvme0
SMART/Health Information Log
============================
Critical Warning State:         0x00
 Available spare:               0
 Temperature:                   0
 Device reliability:            0
 Read only:                     0
 Volatile memory backup:        0
Temperature:                    313 K, 39.85 C, 103.73 F
Available spare:                100
Available spare threshold:      10
Percentage used:                16
Data units (512,000 byte) read: 184301
Data units written:             40724337
Host read commands:             4080580
Host write commands:            343783484
Controller busy time (minutes): 3540
Power cycles:                   17
Power on hours:                 1856
Unsafe shutdowns:               5
Media errors:                   0
No. error info log entries:     0
Warning Temp Composite Time:    345
Error Temp Composite Time:      146
Temperature 1 Transition Count: 0
Temperature 2 Transition Count: 0
Total Time For Temperature 1:   0
Total Time For Temperature 2:   0


root@firewall:/var/log/system # smartctl -a /dev/nvme0
smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       ShiJi 128GB M.2-NVMe
Serial Number:                      AA000000000000000062
Firmware Version:                   T0709A3
PCI Vendor/Subsystem ID:            0x126f
IEEE OUI Identifier:                0x000000
Controller ID:                      1
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          128,035,676,160 [128 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            000000 0000000001
Local Time is:                      Sun Jan  4 10:13:31 2026 CET
Firmware Updates (0x12):            1 Slot, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x0015):     Comp DS_Mngmt Sav/Sel_Feat
Log Page Attributes (0x03):         S/H_per_NS Cmd_Eff_Lg
Maximum Data Transfer Size:         64 Pages
Warning  Comp. Temp. Threshold:     83 Celsius
Critical Comp. Temp. Threshold:     85 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     6.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        40 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    16%
Data Units Read:                    184,302 [94.3 GB]
Data Units Written:                 40,727,739 [20.8 TB]
Host Read Commands:                 4,080,612
Host Write Commands:                343,809,545
Controller Busy Time:               3,541
Power Cycles:                       17
Power On Hours:                     1,856
Unsafe Shutdowns:                   5
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    345
Critical Comp. Temperature Time:    146

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06, NSID 0xffffffff)
Self-test status: No self-test in progress
Num  Test_Description  Status                       Power_on_Hours  Failing_LBA  NSID Seg SCT Code
 0   Short             Completed without error                1856            -     -   -   -    -
 1   Short             Completed without error                1855            -     -   -   -    -

[meyergru]

See this, point 23. Try applying microcode updates and setting the tuneables.

[MattD76]

Thank you for these hints!
I updated the microcode (dmesg reports an update from version 0x10 to 0x1e) as well as setting the tunables. At first I only disabled PCID; the "nvme0: async event occured" continued to pop up, thus adjusted the rest of the tunables, too. The messages still continue to come in. But I will wait and see if the system panics again.

PS: it seems I was ignorant to see the read me - apologies.

[meyergru]

Success is absolutely not guaranteed by these measures,  especially considering that no-name NVME SSD. These errors might go unnoticed under other OSes, so the manufacturer probably does not bother. If you still experience problems, I would suggest changing the SSD for a better one with a high TBW.