Severe Boot Delays & Recurring Network Outages Due to NVMe Timeouts (Heat-Relate

[aru.persia]

OPNsense Support Request – Severe Boot Delays & Recurring Network Outages Due to NVMe Timeouts (Heat‑Related)


PROBLEM SUMMARY

My OPNsense firewall (version 26.1) takes approximately 20 minutes to complete the boot process.

While the system does eventually become fully operational, this delay occurs reliably every two weeks. When the issue strikes, the firewall becomes unresponsive during this period, and internal DHCP clients (devices at home) are unable to obtain IP addresses – effectively taking the entire local network offline until the boot sequence fully recovers.

Critical physical observation: The device chassis becomes extremely hot – hot enough to "cook eggs on it." If I shut down the system, wait a short while for it to physically cool down, and then power it back on, the 20‑minute boot delay reliably manifests. The root cause appears to be a thermal‑related lockup of the NVMe storage controller during the cold‑start initialization phase.


SYSTEM INFORMATION

Operating System

Code Select Expand

OPNsense 26.1
FreeBSD 14.3-RELEASE-p15
stable/26.1-n272132-0101c21cac77
amd64

Hardware 

• DEC750 - OPNsense® Desktop Security Appliance - EU
• CPU: AMD Ryzen Embedded V1500B (4 cores / 8 threads)
• Memory: 8 GB RAM
• BIOS Version: 05.32.50.0014-A10.24 (Build Date: 12/14/2022)
• Firmware Vendor: INSYDE Corp.
• Boot mode: UEFI
• Console: Serial console via USB (115200 8N1)

DETAILED SYMPTOM TIMELINE & RECURRENCE PATTERN

• Recurrence: The problem manifests approximately every two weeks, suggesting a cumulative thermal or power‑state degradation over time.
• Operational Impact: When the NVMe controller stalls, the entire system stalls. Consequently, the DHCP service (and likely all disk‑dependent processes) becomes unresponsive, leaving clients unable to obtain IP addresses.
• Thermal Evidence: The physical chassis is dangerously hot to the touch. This corroborates the SMART temperature readings below (showing up to 79°C on Sensor 1).
• Post‑Cooling Behavior: Forcing a power‑off and allowing the hardware to cool down (even slightly) will cause the exact same 20‑minute timeout loop on the subsequent boot. This strongly indicates the controller's firmware or physical silicon is struggling to initialize at elevated temperatures.

STORAGE DEVICE

• Model: TS256GMTE710T
• Serial: H532690002
• Firmware: 82B0U9MP
• NVMe Version: 1.4
• Capacity: 256 GB
• Device: /dev/nvme0
• ZFS device: nda0p3

ZFS HEALTH CHECK (performed while cool/stable)

Code Select Expand

zpool status -v
pool: zroot
state: ONLINE
config:
  NAME        STATE     READ WRITE CKSUM
  zroot       ONLINE       0     0     0
    nda0p3    ONLINE       0     0     0
errors: No known data errors

A scrub completed successfully with 0 errors. The filesystem integrity is sound.


SMART / NVMe HEALTH INFORMATION (Critical Thermal Data)

Code Select Expand

smartctl -x /dev/nvme0
SMART overall-health self-assessment test result: PASSED
Critical Warning:                   0x00
Available Spare:                    100%
Percentage Used:                    26%
Data Units Written:                 32.0 TB
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Temperature Readings (Excerpt)
Initial reading (idle/operational):

• Temperature: 66°C
• Temperature Sensor 1: 79°C ⚠️ (This is dangerously high for NVMe and likely the source of the timeout.)

Later reading (stabilized):

• Temperature: 56°C
• Sensor 1: 67°C

Note: The sustained high temperature (Sensor 1 hitting 79°C) strongly suggests the controller is experiencing thermal throttling or outright thermal lockups during initialization.

BOOT PROBLEM DETAILS & KERNEL LOG EVIDENCE

During a cold boot following a heat‑soak period, the system pauses extensively. The relevant logs show a relentless NVMe reset loop:

Code Select Expand

nvme0: Resetting controller due to a timeout.
nvme0: resetting controller
nvme0: Waiting for reset to complete
nvme0: Waiting for reset to complete
nvme0: Waiting for reset to complete
...
# Eventually, after ~20 minutes:
nvme0: resubmitting queued i/o
nvme0: READ sqid:1 cid:0 nsid:1
nvme0: WRITE sqid:3 cid:0 nsid:1

Logs confirming recurrence:

Code Select Expand

/var/log/system/latest.log:
2026-07-09T17:29:35 nvme0: Resetting controller due to a timeout.
2026-07-09T17:38:03 nvme0: Resetting controller due to a timeout.
2026-07-09T20:10:22 nvme0: Resetting controller due to a timeout.

Additional Warnings (Cosmetic)

Code Select Expand

BOOT LOADER IS TOO OLD. PLEASE UPGRADE.
Please set hw.efifb.address and hw.efifb.stride.
These are noted but considered cosmetic/unrelated to the 20‑minute delay.


ACTIONS ALREADY PERFORMED

• Checked ZFS pool status – Healthy
• Completed ZFS scrub – 0 errors
• Checked SMART/NVMe health – PASSED
• Verified NVMe error log – Empty
• Confirmed no media/data integrity errors

QUESTIONS FOR OPNsense SUPPORT

Given the clear thermal component and the bi‑weekly failure cycle, we need targeted guidance:

• Known compatibility: Is this specific TS256GMTE710T controller known to exhibit timeouts at high temperatures under FreeBSD 14.3 / OPNsense 26.1?
• Thermal mitigation: Is there a way to monitor the NVMe temperature from the OS and trigger a graceful reboot before the thermal lockup occurs (mitigating the bi‑weekly DHCP outage)?
• Loader settings: Would you recommend adjusting the following, and could an increased timeout inadvertently prolong the boot if the drive is overheating?
  
  • Code Select Expand
    hw.pci.enable_aspm=0
  • Code Select Expand
    hw.nvme.timeout=120 (or higher?)
• BIOS configuration: Are there specific BIOS settings we should enforce regarding PCIe ASPM, NVMe power management, or AMD platform power states to reduce thermal buildup?
• Hardware assessment: Does the 79°C sensor reading indicate a failing NVMe unit (thermal pad degradation) or inadequate chassis cooling for this embedded platform? Would you recommend a firmware update for the SSD or immediate replacement?

CURRENT ASSESSMENT

• The ZFS pool is structurally sound.
• The system is stable once booted.
• Root cause: The NVMe controller is likely experiencing thermal instability during power‑on initialization. When the system is hot, the controller fails to initialize within the driver's default timeout window, triggering a ~20‑minute recovery/reset loop.
• Impact: Recurring network outages (DHCP failure) every two weeks due to the lockup.

Recommendation needed: Please advise whether this is a software‑configuration fix (power management/PCIe linkspeed tuning) or if we are looking at a critical hardware failure requiring immediate SSD replacement and improved thermal management.

[nero355]

Critical physical observation: The device chassis becomes extremely hot – hot enough to "cook eggs on it." If I shut down the system, wait a short while for it to physically cool down, and then power it back on, the 20‑minute boot delay reliably manifests. The root cause appears to be a thermal‑related lockup of the NVMe storage controller during the cold‑start initialization phase.

Temperature Readings (Excerpt)
Initial reading (idle/operational):

• Temperature: 66°C
• Temperature Sensor 1: 79°C ⚠️ (This is dangerously high for NVMe and likely the source of the timeout.)

Later reading (stabilized):

• Temperature: 56°C
• Sensor 1: 67°C

Have a look at this topic : https://forum.opnsense.org/index.php?topic=51263.0
You might have to do the same mod in order to keep the NVMe a bit cooler :)

smartctl -x /dev/nvme0
SMART overall-health self-assessment test result: PASSED

That's nice and all, but please post the full output for all S.M.A.R.T. data so we can double check for you ;)

[pfry]

[...]Temperature Sensor 1: 79°C ⚠️ (This is dangerously high for NVMe and likely the source of the timeout.)[...]

It's high, but not unusual - most SSD controllers have limits in the 75-85C range. They draw 2-20W, with M.2 devices generally limited to 6-8W. Most NVME devices will be right up at that limit. They have (generally) very effective thermal throttling, which you appear to be seeing. Yours could just be a hot runner (which would arguably be a hardware fault). Out of curiosity, what's the ambient temperature?

I'm not defending use of M.2 SSDs with no/minimal thermal management, by the way.