Add mirrored drive to existing ZFS install

[charles.adams]

I'd contact Deciso about this. I stand by my claim that the partition layout on the former nda0, now nda1 does not look right. Maybe they know how that came to be. With a GPT scheme partitioned disk in FreeBSD the first partition always starts at 40 512 byte sectors, for example.

Sorry about any extra work or confusion I might have caused.

Thanks, I wish I'd been able to customize the original order with mirrored drives from the factory and have avoided all this.

Going to try a striped install but I do wish the new nvme would show in the installer. Do you think I should open a github ticket about that or is there something I can try first?

[Patrick M. Hausen]

Possibly wipe the new NVME first. From your running installation as root:

Code Select Expand

dd if=/dev/zero of=/dev/nvd0 bs=1m
Make double sure nvd0 is the new one which we configured. ;-)

I would not open a github ticket but contact their company head quarters. Even out of warranty and without an active support contract they should be able and willing to provide some help for their "official" hardware.

Kind regards,
Patrick

[charles.adams]

Possibly wipe the new NVME first. From your running installation as root:

Code Select Expand

dd if=/dev/zero of=/dev/nvd0 bs=1m
Make double sure nvd0 is the new one which we configured. ;-)

I would not open a github ticket but contact their company head quarters. Even out of warranty and without an active support contract they should be able and willing to provide some help for their "official" hardware.

Kind regards,
Patrick

nda0 is the new one per the partition layout. but the code in your quote should be corrected as you have nvd0 in it.

It was purchased in March of 2024 so I hope it is still under warranty!

[charles.adams]

I'd contact Deciso about this. I stand by my claim that the partition layout on the former nda0, now nda1 does not look right. Maybe they know how that came to be. With a GPT scheme partitioned disk in FreeBSD the first partition always starts at 40 512 byte sectors, for example.

Sorry about any extra work or confusion I might have caused.

So I didn't have to reach out to them as sales contacted me to accuse me of running multiple machines without a volume license... I replied back with a link to this thread and a request for help.

I verified the right disk and tried wiping the new NVME and got a another problem: (< USB DISK 3.0 PMAP>   is the conf usb and <ASMT 2235 0> is the usb sata drive I've got the installer on)

Code Select Expand

root@DEC850:~ # camcontrol devlist
<TS256GMTE712A-LNW 82B2W2AM>       at scbus0 target 0 lun 1 (pass0,nda0)
<TS256GMTE710T 82B0U9MP>           at scbus1 target 0 lun 1 (pass1,nda1)
<ASMT 2235 0>                      at scbus2 target 0 lun 0 (pass2,da0)
< USB DISK 3.0 PMAP>               at scbus3 target 0 lun 0 (da1,pass3)

root@DEC850:~ # dd if=/dev/zero of=/dev/nda0 bs=1m
dd: /dev/nda0: Operation not permitted

So I noticed in the bootup scroll several times where I mentions 'zpool' even though the Live USB importer is using UFS. Do you think it could be also loading the zfs pool and that is why it won't permit this action?

[charles.adams]

So I'm trying to reinstall from a 24.10.1 img flashed to a SATA USB drive that I managed to get booted from the importer:

Code Select Expand

root@DEC850:~ # gpart show
=>       40  500118112  nda0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  482805760     4  freebsd-zfs  (230G)
  500117504        648        - free -  (324K)

=>        3  500118181  nda1  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

=>        34  1953525101  da0  GPT  (932G)
          34       66584    1  efi  (33M)
       66618         122    2  freebsd-boot  (61K)
       66740        1024    3  freebsd-swap  (512K)
       67764     5082432    4  freebsd-ufs  (2.4G)
     5150196  1948374939       - free -  (929G)

=>        3  500118181  diskid/DISK-I225250032  GPT  (238G)
          3     532480                       1  efi  (260M)
     532483        305                       2  freebsd-boot  (153K)
     532788  482344960                       3  freebsd-zfs  (230G)
  482877748   17240436                       4  freebsd-swap  (8.2G)

=>        34  1953525101  diskid/DISK-AAAABBBB0009  GPT  (932G)
          34       66584                         1  efi  (33M)
       66618         122                         2  freebsd-boot  (61K)
       66740        1024                         3  freebsd-swap  (512K)
       67764     5082432                         4  freebsd-ufs  (2.4G)
     5150196  1948374939                            - free -  (929G)

=>      63  60628929  da1  MBR  (29G)
        63      1985       - free -  (993K)
      2048  60626912    1  fat32lba  (29G)
  60628960        32       - free -  (16K)

=>      63  60628929  diskid/DISK-071C2B5B141FDC30  MBR  (29G)
        63      1985                                - free -  (993K)
      2048  60626912                             1  fat32lba  (29G)
  60628960        32                                - free -  (16K)

However, when I select ZFS and go to mirror it doesn't show both NVME drives:



I've never installed OpnSense from scratch as we've always purchased a factory device from Desico and have never had a problem before this so I may be doing something wrong?

So I tried going into the opensense-installer again but this time going down the 'other' path:






and device info can see both nvme but when I try to select it in the mirror dialog it still doesn't show both nvme to select them as mirror:



This is getting rather frustrating. Hopefully I can get Desico to help?

[charles.adams]

I'd contact Deciso about this. I stand by my claim that the partition layout on the former nda0, now nda1 does not look right. Maybe they know how that came to be. With a GPT scheme partitioned disk in FreeBSD the first partition always starts at 40 512 byte sectors, for example.

Sorry about any extra work or confusion I might have caused.

So I didn't have to reach out to them as sales contacted me to accuse me of running multiple machines without a volume license... I replied back with a link to this thread and a request for help.

I verified the right disk and tried wiping the new NVME and got a another problem: (< USB DISK 3.0 PMAP>   is the conf usb and <ASMT 2235 0> is the usb sata drive I've got the installer on)

Code Select Expand

root@DEC850:~ # camcontrol devlist
<TS256GMTE712A-LNW 82B2W2AM>       at scbus0 target 0 lun 1 (pass0,nda0)
<TS256GMTE710T 82B0U9MP>           at scbus1 target 0 lun 1 (pass1,nda1)
<ASMT 2235 0>                      at scbus2 target 0 lun 0 (pass2,da0)
< USB DISK 3.0 PMAP>               at scbus3 target 0 lun 0 (da1,pass3)

root@DEC850:~ # dd if=/dev/zero of=/dev/nda0 bs=1m
dd: /dev/nda0: Operation not permitted

So I noticed in the bootup scroll several times where I mentions 'zpool' even though the Live USB importer is using UFS. Do you think it could be also loading the zfs pool and that is why it won't permit this action?

and I tried

Code Select Expand

root@DEC850:~ # gpart destroy -F nda0
gpart: Device busy

Why is it busy when I'm not booted off it, nor am I running the installer but just on the shell via serial?

I tried checking my hunch and it doesn't look like it is caused by zfs:

Code Select Expand

root@DEC850:~ # zpool status
no pools available
root@DEC850:~ # zfs list
no datasets available
I'm giving up for tonight and heading to bed before I try chasing down GEOM and the swap. I wish there was a way to use https://github.com/Emrion/uploaders on Opnsense as I've used it to fix a NAS with boot issues.

[charles.adams]

I couldn't stay asleep.

So it isn't swap that is keeping the nda0 drive busy:

Code Select Expand

root@DEC850:~ # mdconfig -l -v
md98    vnode    2048M  /usr/swap0
root@DEC850:~ # swapinfo -k
Device          1K-blocks     Used    Avail Capacity
/dev/md98         2097152        0  2097152     0%
root@DEC850:~ # tail /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ufs/OPNsense_Install       /               ufs     ro,noatime      1       1
tmpfs                   /tmp            tmpfs   rw,mode=01777   0       0

However, I remembered we used gmirror when making the swap so I tried:

Code Select Expand

root@DEC850:~ # gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  nda0p3 (ACTIVE)

Which I think is the blocking issue that is keeping me from clearing the new nda0.

So if I'm understanding the mann page correctly for gmirror (and this worked so I think so)

Code Select Expand

root@DEC850:~ # gmirror stop swap
GEOM_MIRROR: Device swap: provider destroyed.
GEOM_MIRROR: Device swap destroyed.
GEOM_MIRROR: Device mirror/swap launched (1/1).
root@DEC850:~ # gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  gptid/8c120998-e736-11ef-9807-f490ea00f6a6 (ACTIVE)
root@DEC850:~ # gmirror destroy swap
GEOM_MIRROR: Device swap: provider destroyed.
GEOM_MIRROR: Device swap destroyed.
root@DEC850:~ # gpart destroy -F nda0
nda0 destroyed

and now when I go to opnsense-installer I can select both nvme drives:


I will try sleeping again and see how this works later.

[charles.adams]

I couldn't stay asleep.

So it isn't swap that is keeping the nda0 drive busy:

Code Select Expand

root@DEC850:~ # mdconfig -l -v
md98    vnode    2048M  /usr/swap0
root@DEC850:~ # swapinfo -k
Device          1K-blocks     Used    Avail Capacity
/dev/md98         2097152        0  2097152     0%
root@DEC850:~ # tail /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ufs/OPNsense_Install       /               ufs     ro,noatime      1       1
tmpfs                   /tmp            tmpfs   rw,mode=01777   0       0

However, I remembered we used gmirror when making the swap so I tried:

Code Select Expand

root@DEC850:~ # gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  nda0p3 (ACTIVE)

Which I think is the blocking issue that is keeping me from clearing the new nda0.

So if I'm understanding the mann page correctly for gmirror (and this worked so I think so)

Code Select Expand

root@DEC850:~ # gmirror stop swap
GEOM_MIRROR: Device swap: provider destroyed.
GEOM_MIRROR: Device swap destroyed.
GEOM_MIRROR: Device mirror/swap launched (1/1).
root@DEC850:~ # gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  gptid/8c120998-e736-11ef-9807-f490ea00f6a6 (ACTIVE)
root@DEC850:~ # gmirror destroy swap
GEOM_MIRROR: Device swap: provider destroyed.
GEOM_MIRROR: Device swap destroyed.
root@DEC850:~ # gpart destroy -F nda0
nda0 destroyed

and now when I go to opnsense-installer I can select both nvme drives:


I will try sleeping again and see how this works later.

So incase someone else turns this up when searching one of the bits of advice I got from my Desico email was to use

Code Select Expand

nvmecontrol format nvme0 to wipe the new NVME drive instead of gpart destroy.

I'm going to give making the mirror from my original ZFS install another try instead of wiping both and re-installing from the opnsense-installer. Since I have it booting from the original factory NVME, with the new NVME blank again, I'm going to first use the opnsense-installer to create a clone onto the USB SATA drive. If I understand the function of opnsense-installer from https://docs.opnsense.org/manual/install.html#opnsense-installer correctly this should make it easier to restore if something else goes wrong.

To do that I'm wiping the da0 drive with

Code Select Expand

gpart destroy -F da0 and when I run the opnsense-installer I will select 'other modes' then 'guided root on zfs' then select stripe on da0, force 4k, pool name 'backup' and with a 8GB swap and GPT(BIOS+UEFI). This should give me a cloned backup of my current NVME onto the USB SATA disk and then if something goes pear shaped I just boot from that while we work.

I'd like to complete what we started to validate the method for anyone following after us.

[charles.adams]

So here is the revised plan to try this again:

Create partitions on new nvme:

Code Select Expand

gpart create -s gpt nda0
gpart add -s 532480 -t efi nda0
gpart add -s 1024 -t freebsd-boot nda0
gpart add -a 1m -s 8g -t freebsd-swap nda0
gpart add -a 1m -t freebsd-zfs nda0
create the zfs mirror with the two freebsd-zfs partitions to mirror the OS:

Code Select Expand

zpool attach zroot nda1p3 nda0p4
copy EFI partition:

Code Select Expand

dd if=/dev/nda0p1 of=/dev/nda1p1copy bootloader parition:

Code Select Expand

dd if=/dev/nda0p2 of=/dev/nda1p2
turn swap partition into mirrored device

Code Select Expand

gmirror load
swapoff -a
gmirror label -b round-robin swap nda1p4
gmirror configure -a swap
gmirror insert swap nda0p3
Wait for the resilvering to complete before proceeding with changing the old nvme to the standard partition format.

then stop the mirror operation on the old nvme:

Code Select Expand

zpool detach zroot nda1p3
unmirror swap to stop activity on old nvme:

Code Select Expand

gmirror stop swap
gmirror remove nda1p4
then nuke the partition table on the old nvme:

Code Select Expand

nvmecontrol format nvme1
then recreate the partition table on the old nvme following the standard

Code Select Expand

gpart create -s gpt nda1
gpart add -s 532480 -t efi nda1
gpart add -s 1024 -t freebsd-boot nda1
gpart add -a 1m -s 8g -t freebsd-swap nda1
gpart add -a 1m -t freebsd-zfs nda1
then zfs mirror the OS partiton:

Code Select Expand

zpool attach zroot nda0p4 nda1p4
now WAIT for resilvering to complete.

then copy the efi and boot partions to the old nvme:

Code Select Expand

dd if=/dev/nda1p1 of=/dev/nda0p1
dd if=/dev/nda1p2 of=/dev/nda0p2
then mirror the swap partitions:

Code Select Expand

gmirror load
swapoff -a
gmirror label -b round-robin swap nda0p3
gmirror configure -a swap
gmirror insert swap nda1p3
And NOW finally reboot?

[Patrick M. Hausen]

That does not look consistent to me. Did you perform a fresh install? To which of the two drives? Why not a mirrored install right from the installer?
You can wipe both drives with nvmecontrol and finally they should both show up.

[charles.adams]

That does not look consistent to me. Did you perform a fresh install? To which of the two drives? Why not a mirrored install right from the installer?
You can wipe both drives with nvmecontrol and finally they should both show up.

I did not wipe and reinstall as I got it booting from a live CD, figured out what was blocking nda0 and wiped it. and now want to restart from the original post (nda1 with original factory install, nvd0 blank) to prove the process for the next person who gets a factory Desico and wants to add mirroring of the nvmes.

[Patrick M. Hausen]

If the factory SSD comes with a non standard partitioning you should IMHO reinstall with a standard one. I reinstall all factory new units first thing after unpacking. No idea how these odd partition boundaries cane to be.