IPSec road warrior causing panic on 24.7

[rnoland]

This was working reliably on 24.1, but since upgrading to 24.7 as soon as a client connects the system panics.

MCA: Bank 6, Status 0xbb80000000000e0b
MCA: Global Cap 0x000000000f000c14, Status 0x0000000000000005
MCA: Vendor "GenuineIntel", ID 0x50654, APIC ID 8
MCA: CPU 8 UNCOR EN PCC S AR generic I/O error (pci0:178:0:0)
MCA: Misc 0xb2000000
panic: Unrecoverable machine check exception
cpuid = 8
time = 1722267831
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00d684dd60
vpanic() at vpanic+0x131/frame 0xfffffe00d684de90
panic() at panic+0x43/frame 0xfffffe00d684def0
mca_intr() at mca_intr+0xbb/frame 0xfffffe00d684df20
mchk_calltrap() at mchk_calltrap+0x8/frame 0xfffffe00d684df20
--- trap 0x1c, rip = 0xffffffff8106501b, rsp = 0xfffffe00d3b4cd70, rbp = 0xfffffe00d3b4cd70 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6b/frame 0xfffffe00d3b4cd70
acpi_cpu_idle() at acpi_cpu_idle+0x188/frame 0xfffffe00d3b4cdb0
cpu_idle_acpi() at cpu_idle_acpi+0x46/frame 0xfffffe00d3b4cdd0
cpu_idle() at cpu_idle+0x9d/frame 0xfffffe00d3b4cdf0
sched_idletd() at sched_idletd+0x576/frame 0xfffffe00d3b4cef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00d3b4cf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3b4cf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 11 tid 100011 ]
Stopped at      kdb_enter+0x33: movq    $0,0xfd9b22(%rip)
db:0:kdb.enter.default> textdump set
textdump set
db:0:kdb.enter.default>  capture on
db:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo>  show alllocks
No such command; use "help" to list available commands
db:1:lockinfo>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 8
dynamic pcpu = 0xfffffe0154edac40
curthread    = 0xfffff8010398a740: pid 11 tid 100011 critnest 2 "idle: cpu8"
curpcb       = 0xfffff8010398ac60
fpcurthread  = none
idlethread   = 0xfffff8010398a740: tid 100011 "idle: cpu8"
self         = 0xffffffff82e18000
curpmap      = 0xffffffff81b81670
tssp         = 0xffffffff82e18384
rsp0         = 0xfffffe00d3b4d000
kcr3         = 0x800000005dc47002
ucr3         = 0xffffffffffffffff
scr3         = 0x28bdb4ce3
gs32p        = 0xffffffff82e18404
ldt          = 0xffffffff82e18444
tss          = 0xffffffff82e18434
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 11 tid 100011 td 0xfffff8010398a740
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00d684de90
panic() at panic+0x43/frame 0xfffffe00d684def0
mca_intr() at mca_intr+0xbb/frame 0xfffffe00d684df20
mchk_calltrap() at mchk_calltrap+0x8/frame 0xfffffe00d684df20
--- trap 0x1c, rip = 0xffffffff8106501b, rsp = 0xfffffe00d3b4cd70, rbp = 0xfffffe00d3b4cd70 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6b/frame 0xfffffe00d3b4cd70
acpi_cpu_idle() at acpi_cpu_idle+0x188/frame 0xfffffe00d3b4cdb0
cpu_idle_acpi() at cpu_idle_acpi+0x46/frame 0xfffffe00d3b4cdd0
cpu_idle() at cpu_idle+0x9d/frame 0xfffffe00d3b4cdf0
sched_idletd() at sched_idletd+0x576/frame 0xfffffe00d3b4cef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00d3b4cf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3b4cf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

[franco]

panic: Unrecoverable machine check exception

and

acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6b/frame 0xfffffe00d3b4cd70

Are quite peculiar and not related to IPsec at all (although I don't doubt it could be the trigger).

Something obvious to try in ACPI cases is to update the BIOS of the box and let us know what hardware this is.


Thanks,
Franco

[rnoland]

This is a Dell VEP 4600.  I'm using the 8 core version.  I believe  that BIOS is the latest, but will double check.  Again, this was working reliably in 24.1 on the same hardware.  I agree that the backtrace doesn't seem incredibly helpful, however.
 
https://www.delltechnologies.com/asset/en-us/products/networking/technical-support/virtual-edge-platform-4600-spec-sheet.pdf

[franco]

This is clearly a change in behaviour from FreeBSD 13 to 14. It's not unusual to manifest in this way pertaining to a limited set of hardware. What is problematic is finding the relevant code changes, a workaround and eventual fix.


Cheers,
Franco

[franco]

Side note: this thing appears to have a lot of moving software parts... DiagOS, BMC, CPLD and BIOS oO

Couldn't find any hint of FreeBSD 14.1 running these out in the wild but therefore also no problems related to that version.


Cheers,
Franco

[blblblb]

Just in case I can grab a kernel panic log later and this might be related:
https://forum.opnsense.org/index.php?topic=42194.msg208066#msg208066