OPNsense Protectli boot failure - ZFS root mount error, recovery attempts, kerne

[CursedGravity]

I am running OPNsense on the FW4B Protectli appliance. The system was previously working normally.  My internet just dropped out randomly. 

Storage is a ZFS root installation:

zroot
 └── ROOT
      └── default

The system originally failed to boot with:

Mounting from zfs:zroot/ROOT/default failed with error 5
Retrying for 3 or more seconds...
Initial troubleshooting

I created a verified OPNsense 26.7 VGA amd64 USB image and booted into the live environment.

From the OPNsense live shell:

zpool import

showed the pool was detected and reported:

state: ONLINE

The pool was not initially importable because it reported:

pool was previously in use from another system
the pool can be imported using 'zpool import -f'
ZFS pool checks

I imported the pool read-only:

zpool import -f -o readonly=on zroot

The import succeeded, although mounting datasets failed because the pool was read-only:

cannot mount /zroot: failed to create mountpoint: read-only file system

import was successful, but unable to mount some datasets

After import:

zpool status -v

showed:

state: ONLINE

errors: No known data errors

The boot environment exists:

zfs list

shows:

zroot/ROOT
zroot/ROOT/default
Attempted ZFS rewind/recovery

Because the original import suggested:

recovery is possible, but will result in some data loss

approximately 2 minutes of data must be discarded

recovery can be attempted by executing:

zpool import -F zroot

I attempted:

zpool import -f -F zroot

However, this caused a system panic.

The panic message was:

KDB: enter: panic

panic: zfs(panic): zfs: rt=... adding segment.... overlapping with existing one
Current situation

After the successful read-only import, I rebooted from the internal disk.

The system now gets further into the boot process but stops at:

KDB: enter: panic
db>

[Patrick M. Hausen]

Verify the boot device is ok, then reinstall. If it fails SMART and possibly bad block tests, replace it before reinstalling.

[CursedGravity]

The installed drive is:

Kingston Digital SSDNow UV500 mSATA 120GB (SUV500MS/120G)

SMART checks were performed from an Ubuntu Live environment.

Results:

SMART overall health: PASSED
SMART error log: No errors logged
SMART self-test: Completed without error
SATA PHY event counters:
Command failed due to ICRC error: 0
Device-to-host register FISes sent due to a COMRESET: 4

The SATA counters do not appear to indicate a failing SATA link or obvious communication problem.

Based on these results, the SSD does not currently show signs of failure. The remaining possibilities seem to be:

ZFS metadata corruption from an interrupted write/power event
OPNsense/ZFS issue during an update or shutdown
Some other filesystem/pool corruption event
Less likely, an intermittent hardware issue not caught by SMART

I am considering either:

Reinstalling OPNsense cleanly on the existing Kingston SSD and restoring only the configuration, or
Replacing the SSD anyway with a newer/larger mSATA drive (for example, a 256GB TLC mSATA drive) to provide more ZFS headroom.

At this point I am leaning toward the SSD being healthy and the issue being related to the ZFS pool state rather than a confirmed hardware failure.

Any recommendations on whether the existing pool is worth attempting further recovery on, or whether a clean reinstall is the better approach, would be appreciated.

[newsense]

If you have third party apps like AGH don't forget to retrieve the config files.

Otherwise simply install 26.7 and check for updates, install any plugins then import the config.xml.

[CursedGravity]

Is that ad guard home? 
Before I did the big recent upgrade (the upgrade wasn't done the same day this crashed - it has been running for some time), I did a config export.  That should have everything I need to restore, right?

[nero355]

The installed drive is:

Kingston Digital SSDNow UV500 mSATA 120GB (SUV500MS/120G)

I am not a big fan of those things : Cheapass Consumer Crap Class Hardware :(

SMART checks were performed from an Ubuntu Live environment.

Results:

SMART overall health: PASSED
SMART error log: No errors logged
SMART self-test: Completed without error
SATA PHY event counters:
Command failed due to ICRC error: 0
Device-to-host register FISes sent due to a COMRESET: 4

The SATA counters do not appear to indicate a failing SATA link or obvious communication problem.

Could you post the full output shown by 'smartctl -a /dev/sdX' ?

Also check for possible firmware updates that sometimes fix bugs that show up after a certain amount of working hours...

[CursedGravity]

Why? it's lasted me over 5 years now.  what do you use or recommend?
It doesn't look like smartctl is installed for me.

[Patrick M. Hausen]

Install the os-smart plugin to get smartmontools and a dashboard widget in OPNsense.