Hi all,
I'm battling recurrent kernel panics on my OPNsense router and would really appreciate some pointers.
* Hardware
- Mini‑PC – Topton fanless box with Intel N100 (Alder Lake‑N) (NICs – 4 × Intel I226‑V 2.5 GbE (igc driver))
- I've placed a Fan(USB powered) on top of it.
* OS / Kernel
FreeBSD router.sings.local 14.2-RELEASE-p2 FreeBSD 14.2-RELEASE-p2 stable/25.1-n269753-d27208952e13 SMP amd64
* Symptom
System reboots every few hours (or day) after a kernel panic like below.
<13>1 2025-04-20T10:43:21+09:00 router.sings.local opnsense 88222 - [meta sequenceId="40"] /usr/local/etc/rc.newwanip: ROUTING: entering configure using wan
<13>1 2025-04-20T10:43:21+09:00 router.sings.local opnsense 88222 - [meta sequenceId="41"] /usr/local/etc/rc.newwanip: ROUTING: configuring inet default gateway on wan
<13>1 2025-04-20T10:43:21+09:00 router.sings.local opnsense 88222 - [meta sequenceId="42"] /usr/local/etc/rc.newwanip: ROUTING: keeping inet default route to 222.107.163.254
<13>1 2025-04-20T10:43:21+09:00 router.sings.local opnsense 88222 - [meta sequenceId="43"] /usr/local/etc/rc.newwanip: plugins_configure monitor (,[WAN_DHCP])
<13>1 2025-04-20T10:43:21+09:00 router.sings.local opnsense 88222 - [meta sequenceId="44"] /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,[WAN_DHCP]))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="45"] /usr/local/etc/rc.newwanip: plugins_configure vpn (,[wan],inet)
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="46"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,[wan]))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="47"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,[wan]))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="48"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : wireguard_configure_do())
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="49"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (,[wan],inet)
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="50"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dhcrelay_configure_if(,[wan],inet))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="51"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="52"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="53"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="54"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : openssh_configure_do(,[wan]))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 88222 - [meta sequenceId="55"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,[wan]))
<13>1 2025-04-20T10:43:22+09:00 router.sings.local opnsense 84186 - [meta sequenceId="56"] /usr/local/etc/rc.linkup: plugins_configure newwanip:rfc2136 (,[wan])
<13>1 2025-04-20T10:43:23+09:00 router.sings.local opnsense 88222 - [meta sequenceId="57"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2025-04-20T10:43:23+09:00 router.sings.local opnsense 88222 - [meta sequenceId="58"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : webgui_configure_do(,[wan]))
<13>1 2025-04-20T10:43:23+09:00 router.sings.local opnsense 88222 - [meta sequenceId="59"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : wireguard_sync())
<13>1 2025-04-20T11:43:20+09:00 router.sings.local dhclient 73079 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T11:43:20+09:00 router.sings.local dhclient 73300 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T12:43:20+09:00 router.sings.local dhclient 30512 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T12:43:20+09:00 router.sings.local dhclient 31116 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T13:43:20+09:00 router.sings.local dhclient 92802 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T13:43:20+09:00 router.sings.local dhclient 93313 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T14:43:20+09:00 router.sings.local dhclient 55074 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T14:43:20+09:00 router.sings.local dhclient 55481 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T15:43:20+09:00 router.sings.local dhclient 16269 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T15:43:20+09:00 router.sings.local dhclient 16661 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T16:43:20+09:00 router.sings.local dhclient 74413 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T16:43:20+09:00 router.sings.local dhclient 74887 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T17:43:20+09:00 router.sings.local dhclient 34209 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T17:43:20+09:00 router.sings.local dhclient 34530 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<13>1 2025-04-20T18:43:20+09:00 router.sings.local dhclient 91877 - [meta sequenceId="1"] dhclient-script: Reason RENEW on igc0 executing
<13>1 2025-04-20T18:43:20+09:00 router.sings.local dhclient 92229 - [meta sequenceId="2"] dhclient-script: Creating resolv.conf
<45>1 2025-04-20T19:31:05+09:00 router.sings.local syslog-ng 25552 - [meta sequenceId="1"] syslog-ng starting up; version='4.8.1'
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="2"] Fatal trap 12: page fault while in kernel mode
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="3"] cpuid = 0; apic id = 00
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="4"] fault virtual address = 0x600000028
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="5"] fault code = supervisor read data, page not present
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="6"] instruction pointer = 0x20:0xffffffff80f7e572
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="7"] stack pointer = 0x28:0xfffffe00f74e37c0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="8"] frame pointer = 0x28:0xfffffe00f74e37c0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="9"] code segment = base 0x0, limit 0xfffff, type 0x1b
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="10"] = DPL 0, pres 1, long 1, def32 0, gran 1
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="11"] processor eflags = interrupt enabled, resume, IOPL = 0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="12"] current process = 68076 (python3.11)
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="13"] rdi: fffff802d6b3ddc0 rsi: 0000000000000a33 rdx: 0000000000000003
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="14"] rcx: 0000000000000000 r8: 0000000000000000 r9: fffff802d658c990
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="15"] rax: 0000000600000000 rbx: 0000000000000a33 rbp: fffffe00f74e37c0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="16"] r10: 0000000000000004 r11: 000000000000000d r12: 0000000000000a33
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="17"] r13: fffffe0008c9de38 r14: fffff801ba525210 r15: fffffe0008c9dea0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="18"] trap number = 12
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="19"] panic: page fault
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="20"] cpuid = 0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="21"] time = 1745145000
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="22"] KDB: stack backtrace:
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="23"] db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f74e34b0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="24"] vpanic() at vpanic+0x131/frame 0xfffffe00f74e35e0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="25"] panic() at panic+0x43/frame 0xfffffe00f74e3640
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="26"] trap_fatal() at trap_fatal+0x40b/frame 0xfffffe00f74e36a0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="27"] trap_pfault() at trap_pfault+0x46/frame 0xfffffe00f74e36f0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="28"] calltrap() at calltrap+0x8/frame 0xfffffe00f74e36f0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="29"] --- trap 0xc, rip = 0xffffffff80f7e572, rsp = 0xfffffe00f74e37c0, rbp = 0xfffffe00f74e37c0 ---
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="30"] vm_radix_lookup() at vm_radix_lookup+0x32/frame 0xfffffe00f74e37c0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="31"] vm_object_collapse() at vm_object_collapse+0x1dd/frame 0xfffffe00f74e3840
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="32"] vm_object_deallocate() at vm_object_deallocate+0x317/frame 0xfffffe00f74e3880
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="33"] vm_map_process_deferred() at vm_map_process_deferred+0x92/frame 0xfffffe00f74e38a0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="34"] vm_map_remove() at vm_map_remove+0xf9/frame 0xfffffe00f74e38d0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="35"] exec_new_vmspace() at exec_new_vmspace+0x245/frame 0xfffffe00f74e3930
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="36"] exec_elf64_imgact() at exec_elf64_imgact+0x61b/frame 0xfffffe00f74e39f0
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="37"] kern_execve() at kern_execve+0x7c5/frame 0xfffffe00f74e3d80
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="38"] sys_execve() at sys_execve+0x56/frame 0xfffffe00f74e3e00
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="39"] amd64_syscall() at amd64_syscall+0x10e/frame 0xfffffe00f74e3f30
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="40"] fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f74e3f30
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="41"] --- syscall (59, FreeBSD ELF64, execve), rip = 0x822fd971a, rsp = 0x820397658, rbp = 0x820397660 ---
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="42"] KDB: enter: panic
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="43"] ---<<BOOT>>---
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="44"] Copyright (c) 1992-2023 The FreeBSD Project.
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="45"] Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
<13>1 2025-04-20T19:31:06+09:00 router.sings.local kernel - - [meta sequenceId="46"] The Regents of the University of California. All rights reserved.
My opnsense box is in closet. is it because of heat issue?
Belows are Reporting: Health for cpu temp at that time.
https://imgur.com/a/9ieOuJe (https://imgur.com/a/9ieOuJe)
Is it syslog-ng bug? Does it use python 3.11?
This is a known bug that has been worked on before, so I would expect that something is different in your particular case.
If by Wednsday Franco doesn't see this thread open a Github issue in opnsense/src mentioning this thread.
And since you have the N100, do you have os-cpu-microcode-intel-1.1 installed ? If not do it from GUI or console/ssh
pkg install os-cpu-microcode-intel-1.1
Quote from: newsense on April 20, 2025, 10:50:05 PMThis is a known bug that has been worked on before, so I would expect that something is different in your particular case.
If by Wednsday Franco doesn't see this thread open a Github issue in opnsense/src mentioning this thread.
And since you have the N100, do you have os-cpu-microcode-intel-1.1 installed ? If not do it from GUI or console/ssh
pkg install os-cpu-microcode-intel-1.1
I've already installed it. Then I'll wait until Wednesday and open a GitHub issue if needed.
Quote from: newsense on April 20, 2025, 10:50:05 PMThis is a known bug that has been worked on before, so I would expect that something is different in your particular case.
If by Wednsday Franco doesn't see this thread open a Github issue in opnsense/src mentioning this thread.
And since you have the N100, do you have os-cpu-microcode-intel-1.1 installed ? If not do it from GUI or console/ssh
pkg install os-cpu-microcode-intel-1.1
I've posted github issue at opnsense/src but nobody responds to my issues. Should I post issue at opnsense/core?
Here's my issue : https://github.com/opnsense/src/issues/248
Just a guess, your timing might have been unfortunate and the issue landed during a short week
I was observing identical symptoms on similar hardware (Shuttle DL30N with Intel N100, 2x2.5GbE Intel i226-LM NICs). After a lot of troubleshooting, it seems the problem caused by RAM. Even though the two specimen of [Crucial CT16G48C40S5](https://www.crucial.com/memory/ddr5/CT16G48C40S5) were not necessarily faulty or out-of-spec for said device, switching to a different model of RAM that was on the list of supported modules for my device resolved the page fault issues. OPNSense has been running for 80 hours now. I never used to pay much mind to those compatibility lists, but it seems these little machines can be picky. Perhaps trying a different model of RAM is also something worth trying for you @rlaace423. Either way, neither OPNSense nor the FreeBSD kernel was the source kernel panics in my case.
The china miniPCs are extremely picky when it comes to SSD and RAM. Due to lack of proper BIOS updates you are often locked to a certain type of parts. Specifically, to avoid Hynix ICs as those on mainly the early miniPCs had/have massive problems.
Regards,
S.
Quote from: Djokkum on May 23, 2025, 02:20:55 PMI was observing identical symptoms on similar hardware (Shuttle DL30N with Intel N100, 2x2.5GbE Intel i226-LM NICs). After a lot of troubleshooting, it seems the problem caused by RAM. Even though the two specimen of [Crucial CT16G48C40S5](https://www.crucial.com/memory/ddr5/CT16G48C40S5) were not necessarily faulty or out-of-spec for said device, switching to a different model of RAM that was on the list of supported modules for my device resolved the page fault issues. OPNSense has been running for 80 hours now. I never used to pay much mind to those compatibility lists, but it seems these little machines can be picky. Perhaps trying a different model of RAM is also something worth trying for you @rlaace423. Either way, neither OPNSense nor the FreeBSD kernel was the source kernel panics in my case.
I disassembled my PC, and found I'm using exact same RAM stick.
I'll try another one, and see works well.
Thank you for your advice!
Quote from: Djokkum on May 23, 2025, 02:20:55 PMI was observing identical symptoms on similar hardware (Shuttle DL30N with Intel N100, 2x2.5GbE Intel i226-LM NICs). After a lot of troubleshooting, it seems the problem caused by RAM. Even though the two specimen of [Crucial CT16G48C40S5](https://www.crucial.com/memory/ddr5/CT16G48C40S5) were not necessarily faulty or out-of-spec for said device, switching to a different model of RAM that was on the list of supported modules for my device resolved the page fault issues. OPNSense has been running for 80 hours now. I never used to pay much mind to those compatibility lists, but it seems these little machines can be picky. Perhaps trying a different model of RAM is also something worth trying for you @rlaace423. Either way, neither OPNSense nor the FreeBSD kernel was the source kernel panics in my case.
I've tested another reliable RAM stick(SK Hynix) and no kernel panic happend for 1 or 2 weeks.
I think this is it. Thank you!