Reboots and kernel panic since upgrade to 24.1.6

[basx_ch]

Hallo zusammen
I'm using a Scope7-1510 box for a client, and since the last upgrade (to 24.1.6) the box reboots in irregular frequency. What I find from the logs is the following:
---
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 18
fault virtual address   = 0x28
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80ec943d
stack pointer           = 0x28:0xfffffe00858d6db0
frame pointer           = 0x28:0xfffffe00858d6dc0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 3 (crypto returns 0)
trap number             = 12
panic: page fault
cpuid = 3
time = 1715620516
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00858d6b70
vpanic() at vpanic+0x151/frame 0xfffffe00858d6bc0
panic() at panic+0x43/frame 0xfffffe00858d6c20
trap_fatal() at trap_fatal+0x387/frame 0xfffffe00858d6c80
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00858d6ce0
calltrap() at calltrap+0x8/frame 0xfffffe00858d6ce0
--- trap 0xc, rip = 0xffffffff80ec943d, rsp = 0xfffffe00858d6db0, rbp = 0xfffffe00858d6dc0 ---
key_freesav() at key_freesav+0x5d/frame 0xfffffe00858d6dc0
esp_input_cb() at esp_input_cb+0x75/frame 0xfffffe00858d6e80
crypto_ret_thread() at crypto_ret_thread+0x1e9/frame 0xfffffe00858d6ef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00858d6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00858d6f30
--- trap 0xc0af0f41, rip = 0x10b90000387b05, rsp = 0x84f8b48178b4831, rbp = 0xfb83507601fd8355 ---
KDB: enter: panic
---
We are using IPS and HA Proxy, as guessed as part of the problem in this old threads: https://forum.opnsense.org/index.php?topic=27211.0 and https://forum.opnsense.org/index.php?topic=28302.0 . We deactived IPS mode, but this did not solve the problem. But we realised, that stopping a large copy job that would go through an IPSec tunnel, slowed down the rebooting cadence (now every few hours instead of minutes). And yes, the box uses intel nics (X553 and I210)
Has anybody any similar experiences?
Thanks , Beat

[franco]

Definitely an in-kernel crypto issue, perhaps related to IPsec, but this could also be hardware related problem.

What was the last known good version? I suppose you could try and downgrade the kernel first to see if it makes a difference. 24.1.6 merged some crypto code fixes, but I doubt them being the issue a bit (but we can't be certain).


Cheers,
Franco