OPNsense Protectli boot failure - ZFS root mount error, recovery attempts, kerne

[CursedGravity]

I am running OPNsense on the FW4B Protectli appliance. The system was previously working normally.  My internet just dropped out randomly. 

Storage is a ZFS root installation:

zroot
 └── ROOT
      └── default

The system originally failed to boot with:

Mounting from zfs:zroot/ROOT/default failed with error 5
Retrying for 3 or more seconds...
Initial troubleshooting

I created a verified OPNsense 26.7 VGA amd64 USB image and booted into the live environment.

From the OPNsense live shell:

zpool import

showed the pool was detected and reported:

state: ONLINE

The pool was not initially importable because it reported:

pool was previously in use from another system
the pool can be imported using 'zpool import -f'
ZFS pool checks

I imported the pool read-only:

zpool import -f -o readonly=on zroot

The import succeeded, although mounting datasets failed because the pool was read-only:

cannot mount /zroot: failed to create mountpoint: read-only file system

import was successful, but unable to mount some datasets

After import:

zpool status -v

showed:

state: ONLINE

errors: No known data errors

The boot environment exists:

zfs list

shows:

zroot/ROOT
zroot/ROOT/default
Attempted ZFS rewind/recovery

Because the original import suggested:

recovery is possible, but will result in some data loss

approximately 2 minutes of data must be discarded

recovery can be attempted by executing:

zpool import -F zroot

I attempted:

zpool import -f -F zroot

However, this caused a system panic.

The panic message was:

KDB: enter: panic

panic: zfs(panic): zfs: rt=... adding segment.... overlapping with existing one
Current situation

After the successful read-only import, I rebooted from the internal disk.

The system now gets further into the boot process but stops at:

KDB: enter: panic
db>

[Patrick M. Hausen]

Verify the boot device is ok, then reinstall. If it fails SMART and possibly bad block tests, replace it before reinstalling.