Kernel panic at 3am daily and during health checks

[dstrctdagain321]

Hi,

I've noticed a few different threads with folks having similar symptoms, but I'll add my issue here separately. I attached the log of the panic and can share more if needed. I would like some assistance because I'm afraid that the frequent panics will end up damaging my installation or hardware.

I have a Protectli VP2420 with the latest firmware. In fact, it's a brand new box. It's experiencing kernel panics daily at 3am. A reproducible kernel panic also occurs during a health check when checking packages. I assume they are related.

I am only using ACME and Chrony plugins, and Suratica. I am using a LAGG with VLANs. It is connected to a Unifi switch->Unifi Pro 6 (wireless), which is sending tagged/untagged traffic to Protectli.

On an unrelated note, I have another box with the same exact hardware and firmware, but different configuration (using Zenarmor and Suratica). I have been using this box for 6 months with no issues. It is not experiencing kernel panics. It is not using VLANs or LAGG.

Thank you for your help! 

[meyergru]

The fault occurs upon ZFS access, so assuming both devices are setup with ZFS, I would think that maybe there are memory problems. Maybe you got defective RAM?

You can use dmidecode to find which type of RAM is in use.

[dstrctdagain321]

Thank you!! 

It does not display anything I can tell being useful for us besides BIOS version. I attached screenshot.

But I know I am using x2 16 GB Kingston Impact DDR4-3200 SO-DIMM modules.

[meyergru]

You mean "dmidecode -t17" does not show anything?

Since your BIOS does not allow to set RAM timings and the Kingston Impact modules are DDR4-3200 CL20, it might be an incompatibility, because DDR4 is specced at DDR4-2400 CL16. I would try to use either base-spec DDR4 RAM or high quality modules which can operate at lower cache latencies.

For starters, you could try to swap the RAM between the two machines and see if that helps at all.

[dstrctdagain321]

Apologies, been busy but still experiencing issues here.

Output:

SMBIOS 3.3 present.

Handle 0x000A, DMI type 17, 40 bytes
Memory Device
   Array Handle: 0x0009
   Error Information Handle: Not Provided
   Total Width: 64 bits
   Data Width: 64 bits
   Size: 32 GB
   Form Factor: SODIMM
   Set: None
   Locator: Channel-1-DIMM-0
   Bank Locator: BANK 0
   Type: DDR4
   Type Detail: Unknown Synchronous
   Speed: 3200 MT/s
   Manufacturer: Kingston
   Serial Number: a6229aae
   Asset Tag: Channel-1-DIMM-0-AssetTag
   Part Number: KF3200C20S4/32GX
   Rank: 2
   Configured Memory Speed: 3200 MT/s
   Minimum Voltage: 1.2 V
   Maximum Voltage: 1.2 V
   Configured Voltage: 1.2 V

Same RAM as my other box. Coreboot is a newer version I believe, but will need to verify that when I get the chance. I will not be able to attempt a RAM swap anytime soon, but that could end up being a time-saving answer in the long run.

I'm wondering (if)/hoping it's a configuration issue. Anything else I can provide to help the troubleshoot?

Thanks for your help

[dstrctdagain321]

I had the chance to swap RAM with my other box, and the issue persists with different RAM on the same machine, which appears to rule out the RAM.

I noticed I did not attach the entire panic report, so maybe this will be of some use to somebody.

[meyergru]

If the coreboot version differs, maybe the microcode versions do, too. I have a tutorial here in the forum on how to update the CPU microcode. Some CPUs, especially Alder Lake, are known to be unstable unless they have current microcodes.

I would also try to swap devices, just to rule out hardware problems. You save backup and restore the configurations.

[dstrctdagain321]

I used your tutorial successfully, but no change in behavior.

Great news though: I updated to the latest 24.7_9 firmware offered to me, and the kernel panics have disappeared! I wish I knew exactly what in the update solved the issue, but it might just continue to be a mystery.