Topics

I'm tracking some MCE issues and I want to rule out the culprit being FreeBSD 13.0-STABLE or 13.1-RELEASE (occurs on both).

What is the best way to downgrade from 22.1.x to 21.7.x or to 21.7.8_1 or latest in the 21.7 series that still has the older FreeBSD 12.x kernel.

I mainly want to go back to FreeBSD 12.x or the hardened BSD kernel that shipped with 21.7.x but reverting the packages and userland is fine too, my usage is very minimal just OpenVPN and basic NAT.

Can I use?

Code Select Expand

opnsense-update -kr 21.7

I have read the opnsense-tools man pages and docs but it's not entirely clear to me.

For any reference I'm trying to workaround this error, it may turn out being a hardware issue but I want to try the 12.x FreeBSD kernels first before replacing.

Code Select Expand

MCA: Bank 2, Status 0xb6000000000a010a
MCA: Global Cap 0x0000000000000806, Status 0x0000000000000004
MCA: Vendor "GenuineIntel", ID 0x406c4, APIC ID 4
MCA: CPU 1 UNCOR EN PCC GCACHE L2 ERR error
MCA: Address 0xe50e020
panic: Unrecoverable machine check exception
cpuid = 1
time = 1653803162
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0010c11e40
vpanic() at vpanic+0x17f/frame 0xfffffe0010c11e90
panic() at panic+0x43/frame 0xfffffe0010c11ef0
mca_intr() at mca_intr+0xbb/frame 0xfffffe0010c11f20
mchk_calltrap() at mchk_calltrap+0x8/frame 0xfffffe0010c11f20
--- trap 0x1c, rip = 0x8003eb189, rsp = 0x7fffffffe560, rbp = 0x7fffffffe570 ---
KDB: enter: panic

(older related thread trying to figure out what happened here:  https://forum.opnsense.org/index.php?topic=28422.msg137981#msg137981)

Hello, long-time lurker.

I am running OPNSense 22.1.7_1 on a brand new Protectli FW2B and I'm getting a crash every 12-24 hours or so (panic string: page fault).

I'm only running an OpenVPN client which I've associated with my LAN network and it works great.

System Info

Code Select Expand

FreeBSD 13.0-STABLE stable/22.1-n248071-cafeb6ce414 SMP amd64
OPNsense 22.1.7_1 3cc3877c1
Plugins os-ddclient-1.5 os-dmidecode-1.1_1 os-nut-1.8.1 os-theme-cicada-1.29 os-theme-rebellion-1.8.8
Time Thu, 19 May 2022 10:23:02 +0000
OpenSSL 1.1.1o  3 May 2022
PHP 7.4.29

I get the following crash dumps:

Code Select Expand

KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0010c11e40
vpanic() at vpanic+0x17f/frame 0xfffffe0010c11e90
panic() at panic+0x43/frame 0xfffffe0010c11ef0
mca_intr() at mca_intr+0xbb/frame 0xfffffe0010c11f20
mchk_calltrap() at mchk_calltrap+0x8/frame 0xfffffe0010c11f20
--- trap 0x1c, rip = 0xffffffff8120c430, rsp = 0xfffffe000f5abd48, rbp = 0xfffffe000f5abd70 ---
native_lapic_eoi() at native_lapic_eoi/frame 0xfffffe000f5abd70
Xtimerint() at Xtimerint+0xb1/frame 0xfffffe000f5abd70
--- interrupt, rip = 0xffffffff8110d8e5, rsp = 0xfffffe000f5abe40, rbp = 0xfffffe000f5abe50 ---
cpu_idle() at cpu_idle+0xe5/frame 0xfffffe000f5abe50
sched_idletd() at sched_idletd+0x4e1/frame 0xfffffe000f5abef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe000f5abf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000f5abf30
--- trap 0xf618070, rip = 0xffffffff80c2b91f, rsp = 0, rbp = 0xffffffff8131d1ea ---
mi_startup() at mi_startup+0xdf/frame 0xffffffff8131d1ea
KDB: enter: panic
---<>---

Code Select Expand

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 04
fault virtual address = 0xffffffff80cf6250
fault code = supervisor read instruction, page not present
instruction pointer = 0x20:0xffffffff80cf6250
stack pointer         = 0x28:0xfffffe0010c11df8
frame pointer         = 0x28:0xfffffe0010c11e30
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 11 (idle: cpu1)
trap number = 12
panic: page fault
cpuid = 1
time = 1652910869
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0010c11bb0
vpanic() at vpanic+0x17f/frame 0xfffffe0010c11c00
panic() at panic+0x43/frame 0xfffffe0010c11c60
trap_fatal() at trap_fatal+0x385/frame 0xfffffe0010c11cc0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0010c11d20
calltrap() at calltrap+0x8/frame 0xfffffe0010c11d20
--- trap 0xc, rip = 0xffffffff80cf6250, rsp = 0xfffffe0010c11df8, rbp = 0xfffffe0010c11e30 ---
printf() at printf/frame 0xfffffe0010c11e30
mca_scan() at mca_scan+0x4f6/frame 0xfffffe0010c11ef0
mca_intr() at mca_intr+0x39/frame 0xfffffe0010c11f20
mchk_calltrap() at mchk_calltrap+0x8/frame 0xfffffe0010c11f20
--- trap 0x1c, rip = 0xffffffff80c2eae9, rsp = 0xfffffe000f5abda0, rbp = 0xfffffe000f5abdc0 ---
statclock() at statclock+0x159/frame 0xfffffe000f5abdc0
handleevents() at handleevents+0xf3/frame 0xfffffe000f5abe00
cpu_activeclock() at cpu_activeclock+0x70/frame 0xfffffe000f5abe30
cpu_idle() at cpu_idle+0xa8/frame 0xfffffe000f5abe50
sched_idletd() at sched_idletd+0x4e1/frame 0xfffffe000f5abef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe000f5abf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000f5abf30
--- trap 0xf618070, rip = 0xffffffff80c2b91f, rsp = 0, rbp = 0xffffffff8131d1ea ---
mi_startup() at mi_startup+0xdf/frame 0xffffffff8131d1ea
KDB: enter: panic

Attached are the full crash files.

Does this look like a hardware issue?  This unit is a few weeks old.