Install problem on NVMe (nvme0: async event occurred)

[Jwidess]

Hi all,
I have been struggling the last few hours trying to figure out how to suppress messages I am getting during the install on my Mini PC. During the install, after I boot normally into multi-user mode, the console is spammed with the following lines,

Code Select Expand

nvme0: async event occurred (type 0x1, info 0x01, page 0x02)They are spammed at such a rate I cannot get through the installer as even the end GUI when run with installer and opnsense is overwritten by the console messages.
The only other post I can find referencing a similar issue is here, but @MattD76 reported they were still getting the async messages. Please let me know if anyone has a solution to this problem; it would be much appreciated!
NOTE: I tested this with another model of NVMe drive (WD Blue) and this issue completely disappeared, so if I can't get this resolved, I will return my current drive and get another WD, but I'd prefer not to.

Computer: Generic Mini PC "MOGINSOK Mini PC 2.5Gbe Intel Celeron N5095 Quad Core, 4*Intel I225-V"
Drive with async errors: Silicon Power 500GB UD90 NVMe 4.0 Gen4 PCIe M.2 (SP500GBP44UD900)
Install: OPNsense-25.7-vga-amd64.img.bz2
Video Example: https://youtu.be/SFHt2blRSR0
EDIT: Actually, the 128GB A60 SP Drive (SP128GBP34A60M28) would cause my Mini PC to not even POST; the UD90 drive had the async errors.

[meyergru]

As it seems, there are now many more no-name NVMEs build into those chinese units, probably because of the price raise in SSD storage.

Under FreeBSD, they can show such effects, because their manufacturers do not care about FreeBSD (and FreeBSD does not care about circumventing NVME protocol problems).

I would change the NVME for a known brand with high TBW.

P.S.: That now made it to https://forum.opnsense.org/index.php?topic=42985, point 14.

[pfry]

A real Silicon Power unit should be OK, from a brand/manufacturer standpoint. The device itself is a tiny cacheless unit - very cheap, but likely no bargain.

[Jwidess]

As it seems, there are now many more no-name NVMEs build into those chinese units, probably because of the price raise in SSD storage.

Under FreeBSD, they can show such effects, because their manufacturers do not care about FreeBSD (and FreeBSD does not care about circumventing NVME protocol problems).

I would change the NVME for a known brand with high TBW.

P.S.: That now made it to https://forum.opnsense.org/index.php?topic=42985, point 14.

Thanks for the reply, however I've had good experiences with Silicon Power in the past and thought they where at least a decent drive brand, however trying to use another SP branded 128GB Model (P34A60 SP128GBP34A60M28) resulted in the mini PC not even posting, so maybe they're a mediocre brand after all.

And yeah @pfry they appear to be real SP drives, don't know who'd bother making clone SP drives lol.

I might go digging around in the kernel to understand how and why these async events are getting generated, but for now we'll get a different brand drive.

[meyergru]

It does not have to be the drive, see https://forum.opnsense.org/index.php?topic=42985.0, point 23.

[Jwidess]

It does not have to be the drive, see https://forum.opnsense.org/index.php?topic=42985.0, point 23.

Well when repeating the install with a WD branded 500GB NVMe drive, these async messages disappeared. Additionally I did follow point 23 from that link and applied the tweaks from here, but that changed nothing unfortunately. And I appreciate the help!

[Jwidess]

I decided to investigate these nvme0: async event occurred (type 0x1, info 0x01, page 0x02) messages further to figure out what was causing them and if there was a way to prevent/suppress them. I ended up modifying the FreeBSD kernel, specifically nvme_ctrlr.c under /sys/dev/nvme (alongside a few complementary files), to wrap this logging in a tunable parameter that can be specified in /boot/loader.conf to suppress these messages.
I built the vga iso, burned it to a USB, and started the install. Then I set the new param hw.nvme.log_async_events="0" and installation went perfectly. I've made a fork and branch of opnsense/src with my changes and added a release with the compressed iso attached to it if anyone else is interested. I plan to probably PR this into the FreeBSD src repo as I feel like this is an upstream change that would benefit all of FreeBSD, not just OPNsense. However, if anyone has advice on that, please let me know!

My 25.7.10 async-tunable Release Here: https://github.com/jwidess/src/releases/tag/25.7-async-tunable.1

[bsdimp]

So async events are problems with the drive, usually temperature. Log page 2 is the SMART page and it should say what it is.

But if it's a constant spew, then maybe we aren't clearing enough bits in the event masks. Turning off logging almost certainly is the wrong approach, since all those interrupts are boggong down the system...

What does logpage 2 say? nvmecontrol logpage -p2 nvme0

Warner

[Patrick M. Hausen]

@Warner welcome! Great to have you joining. Thanks a lot!

[Jwidess]

So async events are problems with the drive, usually temperature. Log page 2 is the SMART page and it should say what it is.

But if it's a constant spew, then maybe we aren't clearing enough bits in the event masks. Turning off logging almost certainly is the wrong approach, since all those interrupts are boggong down the system...

What does logpage 2 say? nvmecontrol logpage -p2 nvme0

Warner

Good point, I suppose just hiding these is not a great solution. My PR was primarily just a solution to give myself the option to suppress them to allow for an install. At the end of the bug report, I have my output from that machine with the drive experiencing the errors:

SP500GBP44UD900 nvmecontrol Output:

Code Select Expand

~ # nvmecontrol logpage -p 2 nvme0
SMART/Health Information Log
============================
Critical Warning State:        0x00
 Available spare:              0
 Temperature:                  0
 Device reliability:            0
 Read only:                    0
 Volatile memory backup:        0
Temperature:                    311 K, 37.85 C, 100.13 F
Available spare:                100
Available spare threshold:      10
Percentage used:                0
Data units (512,000 byte) read: 7531
Data units written:            10305
Host read commands:            216800
Host write commands:            150867
Controller busy time (minutes): 2596
Power cycles:                  32
Power on hours:                43
Unsafe shutdowns:              9
Media errors:                  0
No. error info log entries:    0
Warning Temp Composite Time:    0
Error Temp Composite Time:      0
Temperature 1 Transition Count: 0
Temperature 2 Transition Count: 0
Total Time For Temperature 1:  0
Total Time For Temperature 2:  0
Bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=292410

[Jwidess]

Something else I have noticed is that when using "nvmecontrol logpage -p 2 nvme0" the "Temperature:" is always exactly the same. I have never seen anything other than "311 K, 37.85 C, 100.13 F", the same goes for smartctl.
I will see tomorrow if I have a duplicate model drive I can test this on.