Add mirrored drive to existing ZFS install

[Patrick M. Hausen]

I started down your path but got a message:

Code Select Expand

root@DEC850:~ # zpool detach zroot nda0p3
root@DEC850:~ # gpart destroy -F nda0
gpart: Device busy

Could this be from my having mirrored the swap already?

Yep.

Code Select Expand

gmirror remove swap nda0p4


[charles.adams]

I started down your path but got a message:

Code Select Expand

root@DEC850:~ # zpool detach zroot nda0p3
root@DEC850:~ # gpart destroy -F nda0
gpart: Device busy

Could this be from my having mirrored the swap already?

Yep.

Code Select Expand

gmirror remove swap nda0p4


That worked:

Code Select Expand

root@DEC850:~ # gmirror remove swap nda0p4
root@DEC850:~ # GEOM_MIRROR: Device swap: provider nda0p4 destroyed.

root@DEC850:~ # zpool detach zroot nda0p3
root@DEC850:~ # gpart destroy -F nda0
nda0 destroyed
root@DEC850:~ # gpart create -s gpt nda0
nda0 created
root@DEC850:~ # gpart add -s 532480 -t efi nda0
nda0p1 added
root@DEC850:~ # gpart add -s 1024 -t freebsd-boot nda0
nda0p2 added
root@DEC850:~ # gpart add -a 1m -s 8g -t freebsd-swap nda0
nda0p3 added
root@DEC850:~ # gpart add -a 1m -t freebsd-zfs nda0
nda0p4 added
root@DEC850:~ # zpool attach zroot nda1p3 nda0p4
root@DEC850:~ # zpool status
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Feb  9 16:38:23 2025
        33.4G / 33.4G scanned, 1.34G / 33.4G issued at 229M/s
        1.36G resilvered, 4.03% done, 00:02:23 to go
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            nda1p3  ONLINE       0     0     0
            nda0p4  ONLINE       0     0     0  (resilvering)

errors: No known data errors

I did get a strange error when I tried to add the new nda0p3 as swap but I'm rebooting right now and I'll try again after that.

I think the error when mirroring the swap is because the new swap size is smaller than the original partition?

Code Select Expand

root@DEC850:~ # gpart show
=>        3  500118181  nda1  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

=>       40  500118112  nda0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  482805760     4  freebsd-zfs  (230G)
  500117504        648        - free -  (324K)

[Patrick M. Hausen]

Yes. We need to repartition the original drive and completely recreate the swap. It's definitely bedtime for me now, but we can continue tomorrow. Just let the resilver finish.

[charles.adams]

Yes. We need to repartition the original drive and completely recreate the swap. It's definitely bedtime for me now, but we can continue tomorrow. Just let the resilver finish.

So after the reboot it does start and I get this message on the serial console:


free image hosting

When I try to boot from it anyway via it gives me:


free image hosting

I'd give this as a paste but I only have a phone to get to the internet and I don't have hotspot.

Going to see if I can download to the phone, transfer to the desktop, and reinstall from serial before I fall asleep.

[charles.adams]

So I got the network back by using the opnsense importer off a backup .XML I made before starting this.

Haven't figured this out on the nvme (and left it as it is), and zenarmor seems missing from this live disk import, but family will be happy when they wake and I can resume this after sleeping.

[Patrick M. Hausen]

Oh dear. The EFI partition needs to be copied over before a reboot. That is not part of the zpool. Sorry, I should have made it clear that you cannot reboot until everything is finished.

[charles.adams]

Oh dear. The EFI partition needs to be copied over before a reboot. That is not part of the zpool. Sorry, I should have made it clear that you cannot reboot until everything is finished.

Live and learn, fortunately nothing is destroyed yet.

Code Select Expand

root@DEC850:~ # gpart show
=>       40  500118112  nda0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  482805760     4  freebsd-zfs  (230G)
  500117504        648        - free -  (324K)

=>        3  500118181  nda1  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

=>      63  60628929  da1  MBR  (29G)
        63      1985       - free -  (993K)
      2048  60626912    1  fat32lba  [active]  (29G)
  60628960        32       - free -  (16K)

=>        34  1953525101  da0  GPT  (932G)
          34       66584    1  efi  (33M)
       66618         122    2  freebsd-boot  (61K)
       66740        1024    3  freebsd-swap  (512K)
       67764     5082432    4  freebsd-ufs  (2.4G)
     5150196  1948374939       - free -  (929G)

=>        34  1953525101  diskid/DISK-AAAABBBB0009  GPT  (932G)
          34       66584                         1  efi  (33M)
       66618         122                         2  freebsd-boot  (61K)
       66740        1024                         3  freebsd-swap  (512K)
       67764     5082432                         4  freebsd-ufs  (2.4G)
     5150196  1948374939                            - free -  (929G)

=>      63  60628929  diskid/DISK-071C2B5B141FDC30  MBR  (29G)
        63      1985                                - free -  (993K)
      2048  60626912                             1  fat32lba  [active]  (29G)
  60628960        32                                - free -  (16K)

[Patrick M. Hausen]

So, what's your current state and where do you want to go today? ;-)

Did you install with a ZFS mirror from the beginning?

EDIT: from your gpart output you did not. So please post the output of `zpool status` to check which SSD you are running from.

[charles.adams]

So, what's your current state and where do you want to go today? ;-)

Did you install with a ZFS mirror from the beginning?

EDIT: from your gpart output you did not. So please post the output of `zpool status` to check which SSD you are running from.

Right now it is running from a Live USB via the Importer. So no zfs.

Before reboot I had gotten the zpool mirrior up on nda1p3 and nda0p4 and not gotten the swap mirriored but swap was on nda0p3.

[Patrick M. Hausen]

Probably all that is missing for now is the EFI partition content on nda0.

Code Select Expand

dd if=/dev/nda1p1 of=/dev/nda0p1
Then try a reboot after "office hours".

[charles.adams]

Probably all that is missing for now is the EFI partition content on nda0.

Code Select Expand

dd if=/dev/nda1p1 of=/dev/nda0p1
Then try a reboot after "office hours".

Yes, the original install (on nda1) is the factory one that came on my DEC850 so it was ZFS striped as from the factory they only include 1 NVME.

I've copied partition 1 across and that wasn't enough:

From the serial console during boot:

Code Select Expand

InsydeH2O Version : 05.22.01.0021.0013
BIOS Build Date : 09/06/2023
Processor Type : AMD EPYC 3201 8-Core Processor
System Memory Speed : 2133 MHz


CPUID : 800F12
Press Esc go to Setup Utility
Please set hw.efifb.address and hw.efifb.stride.
Consoles: EFI console
zio_read error: 5
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p1:
FreeBSD/amd64 EFI loader, Revision 1.1

   Command line arguments: loader.efi
   Image base: 0x740da000
   EFI version: 2.50
   EFI Firmware: INSYDE Corp. (rev 21024.4112)
   Console: efi (0x1000)
   Load Path: \EFI\BOOT\BOOTX64.EFI
   Load Device: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,85
00F6A6,0x28,0x82000)
   BootCurrent: 0003
   BootOrder: 0003[*] 0001 0000 2001 2002 2003
   BootInfo Path: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,
EA00F6A6,0x28,0x82000)
Ignoring Boot0003: Only one DP found
Trying ESP: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,85B5EA
A6,0x28,0x82000)
Setting currdev to disk0p1:
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(2,GPT,882B612B-E
x82028,0x400)
Setting currdev to disk0p2:
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(3,GPT,8C120998-E
x82800,0x1000000)
Setting currdev to
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(4,GPT,8E6BB550-E
x1082800,0x1CC70800)
Setting currdev to
Failed to find bootable partition
press any key to interrupt reboot in 1 seconds

I also tried going into the UEFI and switching from nda1 to nda0 and got the same:

Code Select Expand

Please set hw.efifb.address and hw.efifb.stride.
Consoles: EFI console
zio_read error: 5
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p1:
FreeBSD/amd64 EFI loader, Revision 1.1

   Command line arguments: loader.efi
   Image base: 0x740da000
   EFI version: 2.50
   EFI Firmware: INSYDE Corp. (rev 21024.4112)
   Console: efi (0x1000)
   Load Path: \EFI\BOOT\BOOTX64.EFI
   Load Device: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,85B5EA4E-E736-11EF-9807-F490EA
00F6A6,0x28,0x82000)
   BootCurrent: 0003
   BootOrder: 0003[*] 0001 0000 2001 2002 2003
   BootInfo Path: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,85B5EA4E-E736-11EF-9807-F490
EA00F6A6,0x28,0x82000)
Ignoring Boot0003: Only one DP found
Trying ESP: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(1,GPT,85B5EA4E-E736-11EF-9807-F490EA00F6
A6,0x28,0x82000)
Setting currdev to disk0p1:
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(2,GPT,882B612B-E736-11EF-9807-F490EA00F6A6,0
x82028,0x400)
Setting currdev to disk0p2:
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(3,GPT,8C120998-E736-11EF-9807-F490EA00F6A6,0
x82800,0x1000000)
Setting currdev to
Trying: PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/NVMe(0x1,32-48-9C-55-52-48-35-7C)/HD(4,GPT,8E6BB550-E736-11EF-9807-F490EA00F6A6,0
x1082800,0x1CC70800)
Setting currdev to
Failed to find bootable partition
press any key to interrupt reboot in 1 seconds


[charles.adams]

Just incase I copied both the EFI and bootloader partitions from nvda1 to nvda0 via:

#copy EFI

Code Select Expand

dd if=/dev/nda1p1 of=/dev/nda0p1#copy bootloader

Code Select Expand

dd if=/dev/nda1p2 of=/dev/nda0p2
And still no luck. Wondering if somehow the nda1 data got messed up?

I started down the path of the opnsense-installer while I was booted from the Live USB importer and saw something that makes me think that the install is still there if we could just correct the efi issue:



nda0 and nvd0 and the new NVME I was trying to expand onto.
nda1 and nvd1 are the original factory Desico NVME drive with the strange partitioning you IDed.
da1 is the usb stick on a hub that contains the original setup's config.xml
da0 is the SATA SSD in a usb enclosure that has the Opnsense 24.10 Live USB installed with rufus from the provided img when I purchased the DEC850 in March 2024.
md98 I'm not sure about, is it the ufs from da0?
zroot I think would be the mirror we created on the two NVME drives?

[Patrick M. Hausen]

I would do a complete reinstall given that you have a saved config. That should fix any partition layout issues. You can select mirrored to both NVMe drives during installation.

[charles.adams]

So I'm trying to reinstall from a 24.10.1 img flashed to a SATA USB drive that I managed to get booted from the importer:

Code Select Expand

root@DEC850:~ # gpart show
=>       40  500118112  nda0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  482805760     4  freebsd-zfs  (230G)
  500117504        648        - free -  (324K)

=>        3  500118181  nda1  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

=>        34  1953525101  da0  GPT  (932G)
          34       66584    1  efi  (33M)
       66618         122    2  freebsd-boot  (61K)
       66740        1024    3  freebsd-swap  (512K)
       67764     5082432    4  freebsd-ufs  (2.4G)
     5150196  1948374939       - free -  (929G)

=>        3  500118181  diskid/DISK-I225250032  GPT  (238G)
          3     532480                       1  efi  (260M)
     532483        305                       2  freebsd-boot  (153K)
     532788  482344960                       3  freebsd-zfs  (230G)
  482877748   17240436                       4  freebsd-swap  (8.2G)

=>        34  1953525101  diskid/DISK-AAAABBBB0009  GPT  (932G)
          34       66584                         1  efi  (33M)
       66618         122                         2  freebsd-boot  (61K)
       66740        1024                         3  freebsd-swap  (512K)
       67764     5082432                         4  freebsd-ufs  (2.4G)
     5150196  1948374939                            - free -  (929G)

=>      63  60628929  da1  MBR  (29G)
        63      1985       - free -  (993K)
      2048  60626912    1  fat32lba  (29G)
  60628960        32       - free -  (16K)

=>      63  60628929  diskid/DISK-071C2B5B141FDC30  MBR  (29G)
        63      1985                                - free -  (993K)
      2048  60626912                             1  fat32lba  (29G)
  60628960        32                                - free -  (16K)

However, when I select ZFS and go to mirror it doesn't show both NVME drives:



I've never installed OpnSense from scratch as we've always purchased a factory device from Desico and have never had a problem before this so I may be doing something wrong?

[Patrick M. Hausen]

I'd contact Deciso about this. I stand by my claim that the partition layout on the former nda0, now nda1 does not look right. Maybe they know how that came to be. With a GPT scheme partitioned disk in FreeBSD the first partition always starts at 40 512 byte sectors, for example.

Sorry about any extra work or confusion I might have caused.