Severe Boot Delays & Recurring Network Outages Due to NVMe Timeouts (Heat-Relate

[aru.persia]

OPNsense Support Request – Severe Boot Delays & Recurring Network Outages Due to NVMe Timeouts (Heat‑Related)


PROBLEM SUMMARY

My OPNsense firewall (version 26.1) takes approximately 20 minutes to complete the boot process.

While the system does eventually become fully operational, this delay occurs reliably every two weeks. When the issue strikes, the firewall becomes unresponsive during this period, and internal DHCP clients (devices at home) are unable to obtain IP addresses – effectively taking the entire local network offline until the boot sequence fully recovers.

Critical physical observation: The device chassis becomes extremely hot – hot enough to "cook eggs on it." If I shut down the system, wait a short while for it to physically cool down, and then power it back on, the 20‑minute boot delay reliably manifests. The root cause appears to be a thermal‑related lockup of the NVMe storage controller during the cold‑start initialization phase.


SYSTEM INFORMATION

Operating System

Code Select Expand

OPNsense 26.1
FreeBSD 14.3-RELEASE-p15
stable/26.1-n272132-0101c21cac77
amd64

Hardware 

• DEC750 - OPNsense® Desktop Security Appliance - EU
• CPU: AMD Ryzen Embedded V1500B (4 cores / 8 threads)
• Memory: 8 GB RAM
• BIOS Version: 05.32.50.0014-A10.24 (Build Date: 12/14/2022)
• Firmware Vendor: INSYDE Corp.
• Boot mode: UEFI
• Console: Serial console via USB (115200 8N1)

DETAILED SYMPTOM TIMELINE & RECURRENCE PATTERN

• Recurrence: The problem manifests approximately every two weeks, suggesting a cumulative thermal or power‑state degradation over time.
• Operational Impact: When the NVMe controller stalls, the entire system stalls. Consequently, the DHCP service (and likely all disk‑dependent processes) becomes unresponsive, leaving clients unable to obtain IP addresses.
• Thermal Evidence: The physical chassis is dangerously hot to the touch. This corroborates the SMART temperature readings below (showing up to 79°C on Sensor 1).
• Post‑Cooling Behavior: Forcing a power‑off and allowing the hardware to cool down (even slightly) will cause the exact same 20‑minute timeout loop on the subsequent boot. This strongly indicates the controller's firmware or physical silicon is struggling to initialize at elevated temperatures.

STORAGE DEVICE

• Model: TS256GMTE710T
• Serial: H532690002
• Firmware: 82B0U9MP
• NVMe Version: 1.4
• Capacity: 256 GB
• Device: /dev/nvme0
• ZFS device: nda0p3

ZFS HEALTH CHECK (performed while cool/stable)

Code Select Expand

zpool status -v
pool: zroot
state: ONLINE
config:
  NAME        STATE     READ WRITE CKSUM
  zroot       ONLINE       0     0     0
    nda0p3    ONLINE       0     0     0
errors: No known data errors

A scrub completed successfully with 0 errors. The filesystem integrity is sound.


SMART / NVMe HEALTH INFORMATION (Critical Thermal Data)

Code Select Expand

smartctl -x /dev/nvme0
SMART overall-health self-assessment test result: PASSED
Critical Warning:                   0x00
Available Spare:                    100%
Percentage Used:                    26%
Data Units Written:                 32.0 TB
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Temperature Readings (Excerpt)
Initial reading (idle/operational):

• Temperature: 66°C
• Temperature Sensor 1: 79°C ⚠️ (This is dangerously high for NVMe and likely the source of the timeout.)

Later reading (stabilized):

• Temperature: 56°C
• Sensor 1: 67°C

Note: The sustained high temperature (Sensor 1 hitting 79°C) strongly suggests the controller is experiencing thermal throttling or outright thermal lockups during initialization.

BOOT PROBLEM DETAILS & KERNEL LOG EVIDENCE

During a cold boot following a heat‑soak period, the system pauses extensively. The relevant logs show a relentless NVMe reset loop:

Code Select Expand

nvme0: Resetting controller due to a timeout.
nvme0: resetting controller
nvme0: Waiting for reset to complete
nvme0: Waiting for reset to complete
nvme0: Waiting for reset to complete
...
# Eventually, after ~20 minutes:
nvme0: resubmitting queued i/o
nvme0: READ sqid:1 cid:0 nsid:1
nvme0: WRITE sqid:3 cid:0 nsid:1

Logs confirming recurrence:

Code Select Expand

/var/log/system/latest.log:
2026-07-09T17:29:35 nvme0: Resetting controller due to a timeout.
2026-07-09T17:38:03 nvme0: Resetting controller due to a timeout.
2026-07-09T20:10:22 nvme0: Resetting controller due to a timeout.

Additional Warnings (Cosmetic)

Code Select Expand

BOOT LOADER IS TOO OLD. PLEASE UPGRADE.
Please set hw.efifb.address and hw.efifb.stride.
These are noted but considered cosmetic/unrelated to the 20‑minute delay.


ACTIONS ALREADY PERFORMED

• Checked ZFS pool status – Healthy
• Completed ZFS scrub – 0 errors
• Checked SMART/NVMe health – PASSED
• Verified NVMe error log – Empty
• Confirmed no media/data integrity errors

QUESTIONS FOR OPNsense SUPPORT

Given the clear thermal component and the bi‑weekly failure cycle, we need targeted guidance:

• Known compatibility: Is this specific TS256GMTE710T controller known to exhibit timeouts at high temperatures under FreeBSD 14.3 / OPNsense 26.1?
• Thermal mitigation: Is there a way to monitor the NVMe temperature from the OS and trigger a graceful reboot before the thermal lockup occurs (mitigating the bi‑weekly DHCP outage)?
• Loader settings: Would you recommend adjusting the following, and could an increased timeout inadvertently prolong the boot if the drive is overheating?
  
  • Code Select Expand
    hw.pci.enable_aspm=0
  • Code Select Expand
    hw.nvme.timeout=120 (or higher?)
• BIOS configuration: Are there specific BIOS settings we should enforce regarding PCIe ASPM, NVMe power management, or AMD platform power states to reduce thermal buildup?
• Hardware assessment: Does the 79°C sensor reading indicate a failing NVMe unit (thermal pad degradation) or inadequate chassis cooling for this embedded platform? Would you recommend a firmware update for the SSD or immediate replacement?

CURRENT ASSESSMENT

• The ZFS pool is structurally sound.
• The system is stable once booted.
• Root cause: The NVMe controller is likely experiencing thermal instability during power‑on initialization. When the system is hot, the controller fails to initialize within the driver's default timeout window, triggering a ~20‑minute recovery/reset loop.
• Impact: Recurring network outages (DHCP failure) every two weeks due to the lockup.

Recommendation needed: Please advise whether this is a software‑configuration fix (power management/PCIe linkspeed tuning) or if we are looking at a critical hardware failure requiring immediate SSD replacement and improved thermal management.

[nero355]

Critical physical observation: The device chassis becomes extremely hot – hot enough to "cook eggs on it." If I shut down the system, wait a short while for it to physically cool down, and then power it back on, the 20‑minute boot delay reliably manifests. The root cause appears to be a thermal‑related lockup of the NVMe storage controller during the cold‑start initialization phase.

Temperature Readings (Excerpt)
Initial reading (idle/operational):

• Temperature: 66°C
• Temperature Sensor 1: 79°C ⚠️ (This is dangerously high for NVMe and likely the source of the timeout.)

Later reading (stabilized):

• Temperature: 56°C
• Sensor 1: 67°C

Have a look at this topic : https://forum.opnsense.org/index.php?topic=51263.0
You might have to do the same mod in order to keep the NVMe a bit cooler :)

smartctl -x /dev/nvme0
SMART overall-health self-assessment test result: PASSED

That's nice and all, but please post the full output for all S.M.A.R.T. data so we can double check for you ;)

[pfry]

[...]Temperature Sensor 1: 79°C ⚠️ (This is dangerously high for NVMe and likely the source of the timeout.)[...]

It's high, but not unusual - most SSD controllers have limits in the 75-85C range. They draw 2-20W, with M.2 devices generally limited to 6-8W. Most NVME devices will be right up at that limit. They have (generally) very effective thermal throttling, which you appear to be seeing. Yours could just be a hot runner (which would arguably be a hardware fault). Out of curiosity, what's the ambient temperature?

I'm not defending use of M.2 SSDs with no/minimal thermal management, by the way.

[aru.persia]

Thanks for taking a look nero355!

Code Select Expand

root@OPNsense:~ # date
Fri Jul 10 09:45:45 CEST 2026
root@OPNsense:~ # smartctl -x /dev/nvme0
smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.3-RELEASE-p15 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       TS256GMTE710T
Serial Number:                      H532690002
Firmware Version:                   82B0U9MP
PCI Vendor/Subsystem ID:            0x1d79
IEEE OUI Identifier:                0x48357c
Controller ID:                      0
NVMe Version:                       1.4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          256,060,514,304 [256 GB]
Namespace 1 Utilization:            246,970,736,640 [246 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            7c3548 51fc968452
Local Time is:                      Fri Jul 10 09:45:47 2026 CEST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f):         S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         64 Pages
Warning  Comp. Temp. Threshold:     85 Celsius
Critical Comp. Temp. Threshold:     90 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        54 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    26%
Data Units Read:                    111,316 [56.9 GB]
Data Units Written:                 62,633,447 [32.0 TB]
Host Read Commands:                 2,665,357
Host Write Commands:                1,007,349,067
Controller Busy Time:               3,068
Power Cycles:                       16
Power On Hours:                     28,706
Unsafe Shutdowns:                   11
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    6
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               65 Celsius
Temperature Sensor 2:               41 Celsius
Temperature Sensor 3:               54 Celsius
Thermal Temp. 1 Transition Count:   1844
Thermal Temp. 2 Transition Count:   537
Thermal Temp. 1 Total Time:         3954648
Thermal Temp. 2 Total Time:         14109868

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06, NSID 0xffffffff)
Self-test status: No self-test in progress
No Self-tests Logged


Thanks again for double-checking this for me—I appreciate the help!

[Patrick M. Hausen]

I have a DEC750, too, same brand and model of SSD. Since the unit is passively cooled and there is not much convection in the cupboard in my study where I keep my network infrastructure, I placed one of these on top of the firewall:

https://www.amazon.de/dp/B08QYY87XW (the "80mm-1" style)

Really quiet, the rubber feet keep it in place nicely, powered from the DEC750's own USB port. Blowing upward so (I hope) air is getting sucked in and over the upper part of the case from the sides.

[aru.persia]

[...]Temperature Sensor 1: 79°C ⚠️ (This is dangerously high for NVMe and likely the source of the timeout.)[...]

It's high, but not unusual - most SSD controllers have limits in the 75-85C range. They draw 2-20W, with M.2 devices generally limited to 6-8W. Most NVME devices will be right up at that limit. They have (generally) very effective thermal throttling, which you appear to be seeing. Yours could just be a hot runner (which would arguably be a hardware fault). Out of curiosity, what's the ambient temperature?

I'm not defending use of M.2 SSDs with no/minimal thermal management, by the way.

Ambient temperature is around 25–30°C. The SSD is a Transcend MTE710T 256GB NVMe (TS256GMTE710T). It is installed inside a rack.

The drive has reached over 90°C before, which is why I was looking into the thermal behavior. Current SMART data shows 54°C composite temperature and 65°C on sensor 1.

[nero355]

Thanks for taking a look nero355!

Code Select Expand

root@OPNsense:~ # smartctl -x /dev/nvme0Thanks again for double-checking this for me—I appreciate the help!

Why the -x option ?!

I usually just use -a to show as much data as possible and in case of some RAID controllers there is another option that's needed to look into the HDD/SSD via the controller, but that's not the case here luckily.

Maybe try also the 'nvmecontrol' command to see if there is any difference, but so far it seems to be mostly a heat thing...[/code]

[pfry]

[...]Warning  Comp. Temp. Threshold:    85 Celsius[...]

Hm. If the SSD controller is not running at or very near this temperature it should not throttle. Given the tiny thermal mass of the chip, the temperature change will be rapid - I don't know if you can get a diagnostic reading while it is... having difficulties. Deliberately testing the thing with a write benchmark ablates its endurance; the 256GB MTE710T is pretty good in that respect - 400GB/d (over the 3y warranty period). I wouldn't expect it to throttle under a read load, but then booting shouldn't be an issue.

[BrandyWine]

When the system is booting, is this from room ambient, or is the thing already toasty when you reboot it?


This however only works when the OS is running. So depends where in boot stage it is.
nvmecontrol logpage -p 2 nvme0



    Poll it repeatedly (example: once per second, modify as needed as print command only prints the value, etc):

sh

while true; do
  nvmecontrol logpage -p 2 nvme0 | awk -F': ' '/Temperature:/{print $2; exit}'
  [ do some checks and actions here ]
  sleep 1
done