System reboots after a kernel panic

[rlaace423]

Hi all,

I'm battling recurrent kernel panics on my OPNsense router and would really appreciate some pointers.

* Hardware
  - Mini‑PC – Topton fanless box with Intel N100 (Alder Lake‑N) (NICs – 4 × Intel I226‑V 2.5 GbE (igc driver))
  - I've placed a Fan(USB powered) on top of it.

* OS / Kernel

Code Select Expand

FreeBSD router.sings.local 14.2-RELEASE-p2 FreeBSD 14.2-RELEASE-p2 stable/25.1-n269753-d27208952e13 SMP amd64
* Symptom
System reboots every few hours (or day) after a kernel panic like below.

Code Select Expand

<13>1 2025-04-20T10:43:21+09:00 router.sings.local opnsense 88222 - [meta sequenceId="40"] /usr/local/etc/rc.newwanip: ROUTING: entering configure using wan
<13>1 2025-04-20T10:43:21+09:00 router.sings.local opnsense 88222 - [meta sequenceId="41"] /usr/local/etc/rc.newwanip: ROUTING: configuring inet default gateway on wan
<13>1 2025-04-20T10:43:21+09:00 router.sings.local opnsense 88222 - [meta sequenceId="42"] /usr/local/etc/rc.newwanip: ROUTING: keeping inet default route to 222.107.163.254
<13>1 2025-04-20T10:43:21+09:00 router.sings.local opnsense 88222 - [meta sequenceId="43"] /usr/local/etc/rc.newwanip: plugins_configure monitor (,[WAN_DHCP])
<13>1 2025-04-20T10:43:21+09:00 router.sings.local opnsense 88222 - [meta sequenceId="44"] /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,[WAN_DHCP]))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="45"] /usr/local/etc/rc.newwanip: plugins_configure vpn (,[wan],inet)
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="46"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,[wan]))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="47"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,[wan]))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="48"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : wireguard_configure_do())
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="49"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (,[wan],inet)
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="50"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dhcrelay_configure_if(,[wan],inet))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="51"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="52"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="53"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="54"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : openssh_configure_do(,[wan]))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="55"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,[wan]))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 84186 - [meta sequenceId="56"] /usr/local/etc/rc.linkup: plugins_configure newwanip:rfc2136 (,[wan])
<13>1 2025-04-20T10:43:23+09:00 router.sings.local opnsense 88222 - [meta sequenceId="57"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2025-04-20T10:43:23+09:00 router.sings.local opnsense 88222 - [meta sequenceId="58"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : webgui_configure_do(,[wan]))
<13>1 2025-04-20T10:43:23+09:00 router.sings.local opnsense 88222 - [meta sequenceId="59"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : wireguard_sync())
<13>1 2025-04-20T11:43:20+09:00 router.sings.local dhclient 73079 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T11:43:20+09:00 router.sings.local dhclient 73300 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T12:43:20+09:00 router.sings.local dhclient 30512 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T12:43:20+09:00 router.sings.local dhclient 31116 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T13:43:20+09:00 router.sings.local dhclient 92802 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T13:43:20+09:00 router.sings.local dhclient 93313 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T14:43:20+09:00 router.sings.local dhclient 55074 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T14:43:20+09:00 router.sings.local dhclient 55481 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T15:43:20+09:00 router.sings.local dhclient 16269 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T15:43:20+09:00 router.sings.local dhclient 16661 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T16:43:20+09:00 router.sings.local dhclient 74413 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T16:43:20+09:00 router.sings.local dhclient 74887 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T17:43:20+09:00 router.sings.local dhclient 34209 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T17:43:20+09:00 router.sings.local dhclient 34530 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T18:43:20+09:00 router.sings.local dhclient 91877 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T18:43:20+09:00 router.sings.local dhclient 92229 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<45>1 2025-04-20T19:31:05+09:00 router.sings.local syslog-ng 25552 - [meta sequenceId="1"] syslog-ng starting up; version='4.8.1'
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="2"] Fatal trap 12: page fault while in kernel mode
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="3"] cpuid = 0; apic id = 00
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="4"] fault virtual address       = 0x600000028
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="5"] fault code          = supervisor read data, page not present
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="6"] instruction pointer = 0x20:0xffffffff80f7e572
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="7"] stack pointer               = 0x28:0xfffffe00f74e37c0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="8"] frame pointer               = 0x28:0xfffffe00f74e37c0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="9"] code segment                = base 0x0, limit 0xfffff, type 0x1b
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="10"]                    = DPL 0, pres 1, long 1, def32 0, gran 1
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="11"] processor eflags   = interrupt enabled, resume, IOPL = 0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="12"] current process            = 68076 (python3.11)
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="13"] rdi: fffff802d6b3ddc0 rsi: 0000000000000a33 rdx: 0000000000000003
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="14"] rcx: 0000000000000000  r8: 0000000000000000  r9: fffff802d658c990
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="15"] rax: 0000000600000000 rbx: 0000000000000a33 rbp: fffffe00f74e37c0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="16"] r10: 0000000000000004 r11: 000000000000000d r12: 0000000000000a33
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="17"] r13: fffffe0008c9de38 r14: fffff801ba525210 r15: fffffe0008c9dea0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="18"] trap number                = 12
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="19"] panic: page fault
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="20"] cpuid = 0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="21"] time = 1745145000
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="22"] KDB: stack backtrace:
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="23"] db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f74e34b0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="24"] vpanic() at vpanic+0x131/frame 0xfffffe00f74e35e0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="25"] panic() at panic+0x43/frame 0xfffffe00f74e3640
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="26"] trap_fatal() at trap_fatal+0x40b/frame 0xfffffe00f74e36a0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="27"] trap_pfault() at trap_pfault+0x46/frame 0xfffffe00f74e36f0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="28"] calltrap() at calltrap+0x8/frame 0xfffffe00f74e36f0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="29"] --- trap 0xc, rip = 0xffffffff80f7e572, rsp = 0xfffffe00f74e37c0, rbp = 0xfffffe00f74e37c0 ---
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="30"] vm_radix_lookup() at vm_radix_lookup+0x32/frame 0xfffffe00f74e37c0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="31"] vm_object_collapse() at vm_object_collapse+0x1dd/frame 0xfffffe00f74e3840
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="32"] vm_object_deallocate() at vm_object_deallocate+0x317/frame 0xfffffe00f74e3880
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="33"] vm_map_process_deferred() at vm_map_process_deferred+0x92/frame 0xfffffe00f74e38a0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="34"] vm_map_remove() at vm_map_remove+0xf9/frame 0xfffffe00f74e38d0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="35"] exec_new_vmspace() at exec_new_vmspace+0x245/frame 0xfffffe00f74e3930
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="36"] exec_elf64_imgact() at exec_elf64_imgact+0x61b/frame 0xfffffe00f74e39f0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="37"] kern_execve() at kern_execve+0x7c5/frame 0xfffffe00f74e3d80
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="38"] sys_execve() at sys_execve+0x56/frame 0xfffffe00f74e3e00
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="39"] amd64_syscall() at amd64_syscall+0x10e/frame 0xfffffe00f74e3f30
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="40"] fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f74e3f30
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="41"] --- syscall (59, FreeBSD ELF64, execve), rip = 0x822fd971a, rsp = 0x820397658, rbp = 0x820397660 ---
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="42"] KDB: enter: panic
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="43"] ---<<BOOT>>---
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="44"] Copyright (c) 1992-2023 The FreeBSD Project.
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="45"] Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="46"]    The Regents of the University of California. All rights reserved.

My opnsense box is in closet. is it because of heat issue?
Belows are Reporting: Health for cpu temp at that time.
https://imgur.com/a/9ieOuJe

[rlaace423]

Is it syslog-ng bug? Does it use python 3.11?

[newsense]

This is a known bug that has been worked on before, so I would expect that something is different in your particular case.

If by Wednsday Franco doesn't see this thread open a Github issue in opnsense/src mentioning this thread.


And since you have the N100, do you have os-cpu-microcode-intel-1.1 installed ? If not do it from GUI or console/ssh

Code Select Expand

pkg install os-cpu-microcode-intel-1.1

[rlaace423]

This is a known bug that has been worked on before, so I would expect that something is different in your particular case.

If by Wednsday Franco doesn't see this thread open a Github issue in opnsense/src mentioning this thread.


And since you have the N100, do you have os-cpu-microcode-intel-1.1 installed ? If not do it from GUI or console/ssh

Code Select Expand

pkg install os-cpu-microcode-intel-1.1

I've already installed it. Then I'll wait until Wednesday and open a GitHub issue if needed.

[rlaace423]

This is a known bug that has been worked on before, so I would expect that something is different in your particular case.

If by Wednsday Franco doesn't see this thread open a Github issue in opnsense/src mentioning this thread.


And since you have the N100, do you have os-cpu-microcode-intel-1.1 installed ? If not do it from GUI or console/ssh

Code Select Expand

pkg install os-cpu-microcode-intel-1.1

I've posted github issue at opnsense/src but nobody responds to my issues. Should I post issue at opnsense/core?
Here's my issue : https://github.com/opnsense/src/issues/248

[newsense]

Just a guess, your timing might have been unfortunate and the issue landed during a short week

[Djokkum]

I was observing identical symptoms on similar hardware (Shuttle DL30N with Intel N100, 2x2.5GbE Intel i226-LM NICs). After a lot of troubleshooting, it seems the problem caused by RAM. Even though the two specimen of [Crucial CT16G48C40S5](https://www.crucial.com/memory/ddr5/CT16G48C40S5) were not necessarily faulty or out-of-spec for said device, switching to a different model of RAM that was on the list of supported modules for my device resolved the page fault issues. OPNSense has been running for 80 hours now. I never used to pay much mind to those compatibility lists, but it seems these little machines can be picky. Perhaps trying a different model of RAM is also something worth trying for you @rlaace423. Either way, neither OPNSense nor the FreeBSD kernel was the source kernel panics in my case.

[Seimus]

The china miniPCs are extremely picky when it comes to SSD and RAM. Due to lack of proper BIOS updates you are often locked to a certain type of parts. Specifically, to avoid Hynix ICs as those on mainly the early miniPCs had/have massive problems.

Regards,
S.