Messages

So I clicked on "click to check for update" on the lobby page and instead of finding nothing to update (as expected) I got the below error and it did not complete the check:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 25.4.1 (amd64) at Mon Jun  9 20:15:10 CDT 2025
Strict TLS 1.3 and CRL checking is enabled.
Fetching subscription information, please wait... done
Fetching changelog information, please wait... done
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 875 packages processed.
Updating SunnyValley repository catalogue...
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/25.1/${SUBSCRIPTION}/meta.txz: Authentication error
repository SunnyValley has no meta file, using default settings
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/25.1/${SUBSCRIPTION}/packagesite.pkg: Authentication error
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
Certificate verification failed for /CN=zenarmor.net (44)
0020611EE81C0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/25.1/${SUBSCRIPTION}/packagesite.txz: Authentication error
Unable to update repository SunnyValley
Error updating repositories!
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***
Is this a me (I configured something wrong), a OPNSense, or a Zenarmor error?

That does not look consistent to me. Did you perform a fresh install? To which of the two drives? Why not a mirrored install right from the installer?
You can wipe both drives with nvmecontrol and finally they should both show up.

I did not wipe and reinstall as I got it booting from a live CD, figured out what was blocking nda0 and wiped it. and now want to restart from the original post (nda1 with original factory install, nvd0 blank) to prove the process for the next person who gets a factory Desico and wants to add mirroring of the nvmes.

[Wait]

So here is the revised plan to try this again:

Create partitions on new nvme:

Code Select Expand

gpart create -s gpt nda0
gpart add -s 532480 -t efi nda0
gpart add -s 1024 -t freebsd-boot nda0
gpart add -a 1m -s 8g -t freebsd-swap nda0
gpart add -a 1m -t freebsd-zfs nda0
create the zfs mirror with the two freebsd-zfs partitions to mirror the OS:

Code Select Expand

zpool attach zroot nda1p3 nda0p4
copy EFI partition:

Code Select Expand

dd if=/dev/nda0p1 of=/dev/nda1p1copy bootloader parition:

Code Select Expand

dd if=/dev/nda0p2 of=/dev/nda1p2
turn swap partition into mirrored device

Code Select Expand

gmirror load
swapoff -a
gmirror label -b round-robin swap nda1p4
gmirror configure -a swap
gmirror insert swap nda0p3
Wait for the resilvering to complete before proceeding with changing the old nvme to the standard partition format.

then stop the mirror operation on the old nvme:

Code Select Expand

zpool detach zroot nda1p3
unmirror swap to stop activity on old nvme:

Code Select Expand

gmirror stop swap
gmirror remove nda1p4
then nuke the partition table on the old nvme:

Code Select Expand

nvmecontrol format nvme1
then recreate the partition table on the old nvme following the standard

Code Select Expand

gpart create -s gpt nda1
gpart add -s 532480 -t efi nda1
gpart add -s 1024 -t freebsd-boot nda1
gpart add -a 1m -s 8g -t freebsd-swap nda1
gpart add -a 1m -t freebsd-zfs nda1
then zfs mirror the OS partiton:

Code Select Expand

zpool attach zroot nda0p4 nda1p4
now WAIT for resilvering to complete.

then copy the efi and boot partions to the old nvme:

Code Select Expand

dd if=/dev/nda1p1 of=/dev/nda0p1
dd if=/dev/nda1p2 of=/dev/nda0p2
then mirror the swap partitions:

Code Select Expand

gmirror load
swapoff -a
gmirror label -b round-robin swap nda0p3
gmirror configure -a swap
gmirror insert swap nda1p3
And NOW finally reboot?

I couldn't stay asleep.

So it isn't swap that is keeping the nda0 drive busy:

Code Select Expand

root@DEC850:~ # mdconfig -l -v
md98    vnode    2048M  /usr/swap0
root@DEC850:~ # swapinfo -k
Device          1K-blocks     Used    Avail Capacity
/dev/md98         2097152        0  2097152     0%
root@DEC850:~ # tail /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ufs/OPNsense_Install       /               ufs     ro,noatime      1       1
tmpfs                   /tmp            tmpfs   rw,mode=01777   0       0

However, I remembered we used gmirror when making the swap so I tried:

Code Select Expand

root@DEC850:~ # gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  nda0p3 (ACTIVE)

Which I think is the blocking issue that is keeping me from clearing the new nda0.

So if I'm understanding the mann page correctly for gmirror (and this worked so I think so)

Code Select Expand

root@DEC850:~ # gmirror stop swap
GEOM_MIRROR: Device swap: provider destroyed.
GEOM_MIRROR: Device swap destroyed.
GEOM_MIRROR: Device mirror/swap launched (1/1).
root@DEC850:~ # gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  gptid/8c120998-e736-11ef-9807-f490ea00f6a6 (ACTIVE)
root@DEC850:~ # gmirror destroy swap
GEOM_MIRROR: Device swap: provider destroyed.
GEOM_MIRROR: Device swap destroyed.
root@DEC850:~ # gpart destroy -F nda0
nda0 destroyed

and now when I go to opnsense-installer I can select both nvme drives:


I will try sleeping again and see how this works later.

So incase someone else turns this up when searching one of the bits of advice I got from my Desico email was to use

Code Select Expand

nvmecontrol format nvme0 to wipe the new NVME drive instead of gpart destroy.

I'm going to give making the mirror from my original ZFS install another try instead of wiping both and re-installing from the opnsense-installer. Since I have it booting from the original factory NVME, with the new NVME blank again, I'm going to first use the opnsense-installer to create a clone onto the USB SATA drive. If I understand the function of opnsense-installer from https://docs.opnsense.org/manual/install.html#opnsense-installer correctly this should make it easier to restore if something else goes wrong.

To do that I'm wiping the da0 drive with

Code Select Expand

gpart destroy -F da0 and when I run the opnsense-installer I will select 'other modes' then 'guided root on zfs' then select stripe on da0, force 4k, pool name 'backup' and with a 8GB swap and GPT(BIOS+UEFI). This should give me a cloned backup of my current NVME onto the USB SATA disk and then if something goes pear shaped I just boot from that while we work.

I'd like to complete what we started to validate the method for anyone following after us.

I couldn't stay asleep.

So it isn't swap that is keeping the nda0 drive busy:

Code Select Expand

root@DEC850:~ # mdconfig -l -v
md98    vnode    2048M  /usr/swap0
root@DEC850:~ # swapinfo -k
Device          1K-blocks     Used    Avail Capacity
/dev/md98         2097152        0  2097152     0%
root@DEC850:~ # tail /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ufs/OPNsense_Install       /               ufs     ro,noatime      1       1
tmpfs                   /tmp            tmpfs   rw,mode=01777   0       0

However, I remembered we used gmirror when making the swap so I tried:

Code Select Expand

root@DEC850:~ # gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  nda0p3 (ACTIVE)

Which I think is the blocking issue that is keeping me from clearing the new nda0.

So if I'm understanding the mann page correctly for gmirror (and this worked so I think so)

Code Select Expand

root@DEC850:~ # gmirror stop swap
GEOM_MIRROR: Device swap: provider destroyed.
GEOM_MIRROR: Device swap destroyed.
GEOM_MIRROR: Device mirror/swap launched (1/1).
root@DEC850:~ # gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  gptid/8c120998-e736-11ef-9807-f490ea00f6a6 (ACTIVE)
root@DEC850:~ # gmirror destroy swap
GEOM_MIRROR: Device swap: provider destroyed.
GEOM_MIRROR: Device swap destroyed.
root@DEC850:~ # gpart destroy -F nda0
nda0 destroyed

and now when I go to opnsense-installer I can select both nvme drives:


I will try sleeping again and see how this works later.

I'd contact Deciso about this. I stand by my claim that the partition layout on the former nda0, now nda1 does not look right. Maybe they know how that came to be. With a GPT scheme partitioned disk in FreeBSD the first partition always starts at 40 512 byte sectors, for example.

Sorry about any extra work or confusion I might have caused.

So I didn't have to reach out to them as sales contacted me to accuse me of running multiple machines without a volume license... I replied back with a link to this thread and a request for help.

I verified the right disk and tried wiping the new NVME and got a another problem: (< USB DISK 3.0 PMAP>   is the conf usb and <ASMT 2235 0> is the usb sata drive I've got the installer on)

Code Select Expand

root@DEC850:~ # camcontrol devlist
<TS256GMTE712A-LNW 82B2W2AM>       at scbus0 target 0 lun 1 (pass0,nda0)
<TS256GMTE710T 82B0U9MP>           at scbus1 target 0 lun 1 (pass1,nda1)
<ASMT 2235 0>                      at scbus2 target 0 lun 0 (pass2,da0)
< USB DISK 3.0 PMAP>               at scbus3 target 0 lun 0 (da1,pass3)

root@DEC850:~ # dd if=/dev/zero of=/dev/nda0 bs=1m
dd: /dev/nda0: Operation not permitted

So I noticed in the bootup scroll several times where I mentions 'zpool' even though the Live USB importer is using UFS. Do you think it could be also loading the zfs pool and that is why it won't permit this action?

and I tried

Code Select Expand

root@DEC850:~ # gpart destroy -F nda0
gpart: Device busy

Why is it busy when I'm not booted off it, nor am I running the installer but just on the shell via serial?

I tried checking my hunch and it doesn't look like it is caused by zfs:

Code Select Expand

root@DEC850:~ # zpool status
no pools available
root@DEC850:~ # zfs list
no datasets available
I'm giving up for tonight and heading to bed before I try chasing down GEOM and the swap. I wish there was a way to use https://github.com/Emrion/uploaders on Opnsense as I've used it to fix a NAS with boot issues.

So I'm trying to reinstall from a 24.10.1 img flashed to a SATA USB drive that I managed to get booted from the importer:

Code Select Expand

root@DEC850:~ # gpart show
=>       40  500118112  nda0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  482805760     4  freebsd-zfs  (230G)
  500117504        648        - free -  (324K)

=>        3  500118181  nda1  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

=>        34  1953525101  da0  GPT  (932G)
          34       66584    1  efi  (33M)
       66618         122    2  freebsd-boot  (61K)
       66740        1024    3  freebsd-swap  (512K)
       67764     5082432    4  freebsd-ufs  (2.4G)
     5150196  1948374939       - free -  (929G)

=>        3  500118181  diskid/DISK-I225250032  GPT  (238G)
          3     532480                       1  efi  (260M)
     532483        305                       2  freebsd-boot  (153K)
     532788  482344960                       3  freebsd-zfs  (230G)
  482877748   17240436                       4  freebsd-swap  (8.2G)

=>        34  1953525101  diskid/DISK-AAAABBBB0009  GPT  (932G)
          34       66584                         1  efi  (33M)
       66618         122                         2  freebsd-boot  (61K)
       66740        1024                         3  freebsd-swap  (512K)
       67764     5082432                         4  freebsd-ufs  (2.4G)
     5150196  1948374939                            - free -  (929G)

=>      63  60628929  da1  MBR  (29G)
        63      1985       - free -  (993K)
      2048  60626912    1  fat32lba  (29G)
  60628960        32       - free -  (16K)

=>      63  60628929  diskid/DISK-071C2B5B141FDC30  MBR  (29G)
        63      1985                                - free -  (993K)
      2048  60626912                             1  fat32lba  (29G)
  60628960        32                                - free -  (16K)

However, when I select ZFS and go to mirror it doesn't show both NVME drives:



I've never installed OpnSense from scratch as we've always purchased a factory device from Desico and have never had a problem before this so I may be doing something wrong?

So I tried going into the opensense-installer again but this time going down the 'other' path:






and device info can see both nvme but when I try to select it in the mirror dialog it still doesn't show both nvme to select them as mirror:



This is getting rather frustrating. Hopefully I can get Desico to help?

I'd contact Deciso about this. I stand by my claim that the partition layout on the former nda0, now nda1 does not look right. Maybe they know how that came to be. With a GPT scheme partitioned disk in FreeBSD the first partition always starts at 40 512 byte sectors, for example.

Sorry about any extra work or confusion I might have caused.

So I didn't have to reach out to them as sales contacted me to accuse me of running multiple machines without a volume license... I replied back with a link to this thread and a request for help.

I verified the right disk and tried wiping the new NVME and got a another problem: (< USB DISK 3.0 PMAP>   is the conf usb and <ASMT 2235 0> is the usb sata drive I've got the installer on)

Code Select Expand

root@DEC850:~ # camcontrol devlist
<TS256GMTE712A-LNW 82B2W2AM>       at scbus0 target 0 lun 1 (pass0,nda0)
<TS256GMTE710T 82B0U9MP>           at scbus1 target 0 lun 1 (pass1,nda1)
<ASMT 2235 0>                      at scbus2 target 0 lun 0 (pass2,da0)
< USB DISK 3.0 PMAP>               at scbus3 target 0 lun 0 (da1,pass3)

root@DEC850:~ # dd if=/dev/zero of=/dev/nda0 bs=1m
dd: /dev/nda0: Operation not permitted

So I noticed in the bootup scroll several times where I mentions 'zpool' even though the Live USB importer is using UFS. Do you think it could be also loading the zfs pool and that is why it won't permit this action?

Possibly wipe the new NVME first. From your running installation as root:

Code Select Expand

dd if=/dev/zero of=/dev/nvd0 bs=1m
Make double sure nvd0 is the new one which we configured. ;-)

I would not open a github ticket but contact their company head quarters. Even out of warranty and without an active support contract they should be able and willing to provide some help for their "official" hardware.

Kind regards,
Patrick

nda0 is the new one per the partition layout. but the code in your quote should be corrected as you have nvd0 in it.

It was purchased in March of 2024 so I hope it is still under warranty!

I'd contact Deciso about this. I stand by my claim that the partition layout on the former nda0, now nda1 does not look right. Maybe they know how that came to be. With a GPT scheme partitioned disk in FreeBSD the first partition always starts at 40 512 byte sectors, for example.

Sorry about any extra work or confusion I might have caused.

Thanks, I wish I'd been able to customize the original order with mirrored drives from the factory and have avoided all this.

Going to try a striped install but I do wish the new nvme would show in the installer. Do you think I should open a github ticket about that or is there something I can try first?

So I'm trying to reinstall from a 24.10.1 img flashed to a SATA USB drive that I managed to get booted from the importer:

Code Select Expand

root@DEC850:~ # gpart show
=>       40  500118112  nda0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  482805760     4  freebsd-zfs  (230G)
  500117504        648        - free -  (324K)

=>        3  500118181  nda1  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

=>        34  1953525101  da0  GPT  (932G)
          34       66584    1  efi  (33M)
       66618         122    2  freebsd-boot  (61K)
       66740        1024    3  freebsd-swap  (512K)
       67764     5082432    4  freebsd-ufs  (2.4G)
     5150196  1948374939       - free -  (929G)

=>        3  500118181  diskid/DISK-I225250032  GPT  (238G)
          3     532480                       1  efi  (260M)
     532483        305                       2  freebsd-boot  (153K)
     532788  482344960                       3  freebsd-zfs  (230G)
  482877748   17240436                       4  freebsd-swap  (8.2G)

=>        34  1953525101  diskid/DISK-AAAABBBB0009  GPT  (932G)
          34       66584                         1  efi  (33M)
       66618         122                         2  freebsd-boot  (61K)
       66740        1024                         3  freebsd-swap  (512K)
       67764     5082432                         4  freebsd-ufs  (2.4G)
     5150196  1948374939                            - free -  (929G)

=>      63  60628929  da1  MBR  (29G)
        63      1985       - free -  (993K)
      2048  60626912    1  fat32lba  (29G)
  60628960        32       - free -  (16K)

=>      63  60628929  diskid/DISK-071C2B5B141FDC30  MBR  (29G)
        63      1985                                - free -  (993K)
      2048  60626912                             1  fat32lba  (29G)
  60628960        32                                - free -  (16K)

However, when I select ZFS and go to mirror it doesn't show both NVME drives:



I've never installed OpnSense from scratch as we've always purchased a factory device from Desico and have never had a problem before this so I may be doing something wrong?

Just incase I copied both the EFI and bootloader partitions from nvda1 to nvda0 via:

#copy EFI

Code Select Expand

dd if=/dev/nda1p1 of=/dev/nda0p1#copy bootloader

Code Select Expand

dd if=/dev/nda1p2 of=/dev/nda0p2
And still no luck. Wondering if somehow the nda1 data got messed up?

I started down the path of the opnsense-installer while I was booted from the Live USB importer and saw something that makes me think that the install is still there if we could just correct the efi issue:



nda0 and nvd0 and the new NVME I was trying to expand onto.
nda1 and nvd1 are the original factory Desico NVME drive with the strange partitioning you IDed.
da1 is the usb stick on a hub that contains the original setup's config.xml
da0 is the SATA SSD in a usb enclosure that has the Opnsense 24.10 Live USB installed with rufus from the provided img when I purchased the DEC850 in March 2024.
md98 I'm not sure about, is it the ufs from da0?
zroot I think would be the mirror we created on the two NVME drives?

Probably all that is missing for now is the EFI partition content on nda0.

Code Select Expand

dd if=/dev/nda1p1 of=/dev/nda0p1
Then try a reboot after "office hours".

Yes, the original install (on nda1) is the factory one that came on my DEC850 so it was ZFS striped as from the factory they only include 1 NVME.

I've copied partition 1 across and that wasn't enough:

From the serial console during boot:

Code Select Expand

InsydeH2O Version : 05.22.01.0021.0013
BIOS Build Date : 09/06/2023
Processor Type : AMD EPYC 3201 8-Core Processor
System Memory Speed : 2133 MHz


CPUID : 800F12
Press Esc go to Setup Utility
Please set hw.efifb.address and hw.efifb.stride.
Consoles: EFI console
zio_read error: 5
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p1:
FreeBSD/amd64 EFI loader, Revision 1.1

   Command line arguments: loader.efi
   Image base: 0x740da000
   EFI version: 2.50
   EFI Firmware: INSYDE Corp. (rev 21024.4112)
   Console: efi (0x1000)
   Load Path: \EFI\BOOT\BOOTX64.EFI
   Load Device: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,85
00F6A6,0x28,0x82000)
   BootCurrent: 0003
   BootOrder: 0003[*] 0001 0000 2001 2002 2003
   BootInfo Path: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,
EA00F6A6,0x28,0x82000)
Ignoring Boot0003: Only one DP found
Trying ESP: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,85B5EA
A6,0x28,0x82000)
Setting currdev to disk0p1:
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(2,GPT,882B612B-E
x82028,0x400)
Setting currdev to disk0p2:
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(3,GPT,8C120998-E
x82800,0x1000000)
Setting currdev to
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(4,GPT,8E6BB550-E
x1082800,0x1CC70800)
Setting currdev to
Failed to find bootable partition
press any key to interrupt reboot in 1 seconds

I also tried going into the UEFI and switching from nda1 to nda0 and got the same:

Code Select Expand

Please set hw.efifb.address and hw.efifb.stride.
Consoles: EFI console
zio_read error: 5
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p1:
FreeBSD/amd64 EFI loader, Revision 1.1

   Command line arguments: loader.efi
   Image base: 0x740da000
   EFI version: 2.50
   EFI Firmware: INSYDE Corp. (rev 21024.4112)
   Console: efi (0x1000)
   Load Path: \EFI\BOOT\BOOTX64.EFI
   Load Device: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,85B5EA4E-E736-11EF-9807-F490EA
00F6A6,0x28,0x82000)
   BootCurrent: 0003
   BootOrder: 0003[*] 0001 0000 2001 2002 2003
   BootInfo Path: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,85B5EA4E-E736-11EF-9807-F490
EA00F6A6,0x28,0x82000)
Ignoring Boot0003: Only one DP found
Trying ESP: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,85B5EA4E-E736-11EF-9807-F490EA00F6
A6,0x28,0x82000)
Setting currdev to disk0p1:
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(2,GPT,882B612B-E736-11EF-9807-F490EA00F6A6,0
x82028,0x400)
Setting currdev to disk0p2:
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(3,GPT,8C120998-E736-11EF-9807-F490EA00F6A6,0
x82800,0x1000000)
Setting currdev to
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(4,GPT,8E6BB550-E736-11EF-9807-F490EA00F6A6,0
x1082800,0x1CC70800)
Setting currdev to
Failed to find bootable partition
press any key to interrupt reboot in 1 seconds


So, what's your current state and where do you want to go today? ;-)

Did you install with a ZFS mirror from the beginning?

EDIT: from your gpart output you did not. So please post the output of `zpool status` to check which SSD you are running from.

Right now it is running from a Live USB via the Importer. So no zfs.

Before reboot I had gotten the zpool mirrior up on nda1p3 and nda0p4 and not gotten the swap mirriored but swap was on nda0p3.

Oh dear. The EFI partition needs to be copied over before a reboot. That is not part of the zpool. Sorry, I should have made it clear that you cannot reboot until everything is finished.

Live and learn, fortunately nothing is destroyed yet.

Code Select Expand

root@DEC850:~ # gpart show
=>       40  500118112  nda0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  482805760     4  freebsd-zfs  (230G)
  500117504        648        - free -  (324K)

=>        3  500118181  nda1  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

=>      63  60628929  da1  MBR  (29G)
        63      1985       - free -  (993K)
      2048  60626912    1  fat32lba  [active]  (29G)
  60628960        32       - free -  (16K)

=>        34  1953525101  da0  GPT  (932G)
          34       66584    1  efi  (33M)
       66618         122    2  freebsd-boot  (61K)
       66740        1024    3  freebsd-swap  (512K)
       67764     5082432    4  freebsd-ufs  (2.4G)
     5150196  1948374939       - free -  (929G)

=>        34  1953525101  diskid/DISK-AAAABBBB0009  GPT  (932G)
          34       66584                         1  efi  (33M)
       66618         122                         2  freebsd-boot  (61K)
       66740        1024                         3  freebsd-swap  (512K)
       67764     5082432                         4  freebsd-ufs  (2.4G)
     5150196  1948374939                            - free -  (929G)

=>      63  60628929  diskid/DISK-071C2B5B141FDC30  MBR  (29G)
        63      1985                                - free -  (993K)
      2048  60626912                             1  fat32lba  [active]  (29G)
  60628960        32                                - free -  (16K)