Add mirrored drive to existing ZFS install

[abulafia]

, your OmniOS example seems to be Sparc? At least the device names hint at that.

Nope - OmniOS is pure x64. I think SPARC is only catered for by Triblix.

[tinderbox3168]

Great thread, exactly what I was looking for.

One more question: Can I run the commands directly from OPNSense shell while booted or do I have to boot from an external drive?

[Patrick M. Hausen]

All from the live system. For another complete walkthrough see

https://forum.opnsense.org/index.php?topic=32650.msg157910#msg157910

[tinderbox3168]

Thanks, worked nicely.

The only weird thing was, that gmirror load gave the following error message "gmirror: Command 'load' not available; try 'load' first.".
Don't know if it was because its is already loaded. (however gmirror unload gives a similar error).
In any case, I was able to complete the steps afterwards and gmirror status now shows me the swap mirror.

[charles.adams]

So I'm adding a second NVME drive to my DEC850v2 and I wanted to check the commands since all the examples above seem to be for SATA drives.

output of gpart show on original drive:

Code Select Expand

gpart show nda0
=>        3  500118181  nda0  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

Output of zpool status:

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nda0p3    ONLINE       0     0     0

What I think I should do:

Code Select Expand

#copy partition table
gpart backup nda0 | gpart restore -F nda1
#copy EFI
dd if=/dev/nda0p1 of=/dev/nda1p1
#copy bootloader
dd if=/dev/nda0p2 of=/dev/nda1p2
#mirror zfs
zpool attach zroot nda0p3 nda1p3
# turn swap partition into mirrored device
gmirror load
swapoff -a
gmirror label -b round-robin swap nda1p4
gmirror configure -a swap
gmirror insert swap nda0p4
Does this seem sound for how the NVME drive is configured on the DEC850 v2?

[Patrick M. Hausen]

Looks good.

[charles.adams]

Looks good.

So got around to doing this and ran into an issue.

First, although I didn't change the original NVME's location on the MB the assignments changed:

Code Select Expand

root@DEC850:~ # zpool status
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nda1p3    ONLINE       0     0     0

errors: No known data errors
root@lurch:~ # gpart show nda0
gpart: No such geom: nda0.
root@lurch:~ # gpart show nda1
=>        3  500118181  nda1  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

So I tried modifying the prior post instructions to be:

Code Select Expand

root@DEC850:~ # gpart backup nda1 | gpart restore -F nda0but the result I got back was:

Code Select Expand

gpart: entries '4': Invalid argument
When I checked nothing was transfered:

Code Select Expand

gpart: entries '4': Invalid argument
root@lurch:~ # gpart show nda1
=>        3  500118181  nda1  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

root@lurch:~ # gpart show nda0
gpart: No such geom: nda0.

Any suggestions?

[Patrick M. Hausen]

Code Select Expand

camcontrol devlist
please.

[charles.adams]

First checking that both drives are there and aren't throwing errors:

Code Select Expand

root@DEC850:~ # smartctl -a /dev/nvme0
smartctl 7.4 2023-08-01 r5530 [FreeBSD 14.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       TS256GMTE712A-LNW
Serial Number:                      J026240050
Firmware Version:                   82B2W2AM
PCI Vendor/Subsystem ID:            0x1d79
IEEE OUI Identifier:                0x48357c
Controller ID:                      0
NVMe Version:                       1.4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          256,060,514,304 [256 GB]
Namespace 1 Utilization:            0
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            7c3548 52559c4832
Local Time is:                      Sun Feb  9 14:50:16 2025 CST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0016):   Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f):         S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         64 Pages
Warning  Comp. Temp. Threshold:     110 Celsius
Critical Comp. Temp. Threshold:     115 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        23 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    5 [2.56 MB]
Data Units Written:                 10 [5.12 MB]
Host Read Commands:                 197
Host Write Commands:                105
Controller Busy Time:               0
Power Cycles:                       5
Power On Hours:                     0
Unsafe Shutdowns:                   3
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               29 Celsius
Temperature Sensor 2:               23 Celsius
Temperature Sensor 3:               23 Celsius

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06)
Self-test status: No self-test in progress
No Self-tests Logged

root@DEC850:~ # smartctl -a /dev/nvme1
smartctl 7.4 2023-08-01 r5530 [FreeBSD 14.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       TS256GMTE710T
Serial Number:                      I225250032
Firmware Version:                   82B0U9MP
PCI Vendor/Subsystem ID:            0x1d79
IEEE OUI Identifier:                0x48357c
Controller ID:                      0
NVMe Version:                       1.4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          256,060,514,304 [256 GB]
Namespace 1 Utilization:            81,589,567,488 [81.5 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            7c3548 5225de24f0
Local Time is:                      Sun Feb  9 14:50:18 2025 CST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f):         S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         64 Pages
Warning  Comp. Temp. Threshold:     85 Celsius
Critical Comp. Temp. Threshold:     90 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        22 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    24%
Data Units Read:                    1,349,946 [691 GB]
Data Units Written:                 60,833,497 [31.1 TB]
Host Read Commands:                 19,361,131
Host Write Commands:                593,589,915
Controller Busy Time:               2,688
Power Cycles:                       13
Power On Hours:                     5,031
Unsafe Shutdowns:                   9
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               27 Celsius
Temperature Sensor 2:               22 Celsius
Temperature Sensor 3:               21 Celsius

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06)
Self-test status: No self-test in progress
No Self-tests Logged

The output of

Code Select Expand

camcontrol devlist

Code Select Expand

root@DEC850:~ # camcontrol devlist
<TS256GMTE712A-LNW 82B2W2AM>       at scbus0 target 0 lun 1 (pass0,nda0)
<TS256GMTE710T 82B0U9MP>           at scbus1 target 0 lun 1 (pass1,nda1)


[Patrick M. Hausen]

OK. That's weird. So let's check if it's a problem with the backup of the current partition table or the restore operation that fails:

Code Select Expand

gpart backup nda1
I suspect it's the restore that fails, because the "3" for the start of the first partition looks strange. Commonly that is "40". My Deciso appliance looks like this:

Code Select Expand

groot@opnsense:~ # gpart show
=>       40  500118112  nda0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  482805760     4  freebsd-zfs  (230G)
  500117504        648        - free -  (324K)

You could try to create the partitions on nda0 manually and fix the odd sizes while going along. Then when you are finished creating the ZFS mirror, you could fix the odd layout on nda1.

Code Select Expand

gpart create -s gpt nda0
gpart add -s 532480 -t efi nda0
gpart add -s 1024 -t freebsd-boot nda0
gpart add -a 1m -s 482344960 -t freebsd-zfs nda0
gpart add -a 1m -t freebsd-swap nda0

As long as the ZFS partitions are the same size, you should be able to create the mirror. So if that works without a problem the next step would be:

Code Select Expand

zpool attach zroot nda1p3 nda0p3

HTH, please report back for the next steps.

[Patrick M. Hausen]

*** Danger, Will Robinson ***

What I outlined in my last post will not damage your system in any way. But possibly you would want to turn your installation into the "official" partition layout - however you created your current one in the first place.

For that - compare to the output for my own disk - we would need to swap the ZFS and the swap partition. Not a problem with a new SSD to work with. But I need to get a calculator and rewrite the operations from my last post.

So tell me what you would prefer.

I can get to the "fix everything" post tomorrow.

Kind regards,
Patrick

[charles.adams]

*** Danger, Will Robinson ***

What I outlined in my last post will not damage your system in any way. But possibly you would want to turn your installation into the "official" partition layout - however you created your current one in the first place.

For that - compare to the output for my own disk - we would need to swap the ZFS and the swap partition. Not a problem with a new SSD to work with. But I need to get a calculator and rewrite the operations from my last post.

So tell me what you would prefer.

I can get to the "fix everything" post tomorrow.

Kind regards,
Patrick

I'm aiming to keep it official as this is on a DEC850 and I figure supporting it will be easier if I keep it closer to stock.

I'm surprised that you're saying the DEC850 I got directly from Desico in March of 2024 did not have the official load out?

[charles.adams]

OK. That's weird. So let's check if it's a problem with the backup of the current partition table or the restore operation that fails:

Code Select Expand

gpart backup nda1
I suspect it's the restore that fails, because the "3" for the start of the first partition looks strange. Commonly that is "40". My Deciso appliance looks like this:

Code Select Expand

groot@opnsense:~ # gpart show
=>       40  500118112  nda0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  482805760     4  freebsd-zfs  (230G)
  500117504        648        - free -  (324K)

You could try to create the partitions on nda0 manually and fix the odd sizes while going along. Then when you are finished creating the ZFS mirror, you could fix the odd layout on nda1.

Code Select Expand

gpart create -s gpt nda0
gpart add -s 532480 -t efi nda0
gpart add -s 1024 -t freebsd-boot nda0
gpart add -a 1m -s 482344960 -t freebsd-zfs nda0
gpart add -a 1m -t freebsd-swap nda0

As long as the ZFS partitions are the same size, you should be able to create the mirror. So if that works without a problem the next step would be:

Code Select Expand

zpool attach zroot nda1p3 nda0p3

HTH, please report back for the next steps.

So here are the results:

Code Select Expand

root@DEC850:~ # gpart backup nda1
GPT 4
1          efi         3    532480 efifs
2 freebsd-boot    532483       305 bootfs
3  freebsd-zfs    532788 482344960
4 freebsd-swap 482877748  17240436 swapfs

Following your steps resulted in:

Code Select Expand

root@DEC850:~ # gpart create -s gpt nda0
nda0 created
root@DEC850:~ # gpart add -s 532480 -t efi nda0
nda0p1 added
root@DEC850:~ # gpart add -s 1024 -t freebsd-boot nda0
nda0p2 added
root@DEC850:~ # gpart add -a 1m -s 482344960 -t freebsd-zfs nda0
nda0p3 added
root@DEC850:~ # gpart add -a 1m -t freebsd-swap nda0
nda0p4 added
root@DEC850:~ # gpart show
=>        3  500118181  nda1  GPT  (238G)
          3     532480     1  efi  (260M)
     532483        305     2  freebsd-boot  (153K)
     532788  482344960     3  freebsd-zfs  (230G)
  482877748   17240436     4  freebsd-swap  (8.2G)

=>       40  500118112  nda0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528  482344960     3  freebsd-zfs  (230G)
  482879488   17238016     4  freebsd-swap  (8.2G)
  500117504        648        - free -  (324K)

Then added the partition to the zpool:

Code Select Expand

root@DEC850:~ # zpool attach zroot nda1p3 nda0p3
root@DEC850:~ # zpool status
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Feb  9 15:31:08 2025
        33.3G / 33.3G scanned, 747M / 33.3G issued at 187M/s
        751M resilvered, 2.19% done, 00:02:58 to go
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            nda1p3  ONLINE       0     0     0
            nda0p3  ONLINE       0     0     0  (resilvering)

errors: No known data errors
So that worked to make the new NVME match your layout.  Still feeling a little weird that my factory DEC850 didn't come with the official layout.

I did go ahead and mirrored the swap also:

Code Select Expand

root@DEC850:~ # gmirror load
root@DEC850:~ # swapoff -a
swapoff: removing /dev/gpt/swapfs as swap device
root@DEC850:~ # gmirror label -b round-robin swap nda0p4
GEOM_MIRROR: Device mirror/swap launched (1/1).
root@DEC850:~ # gmirror configure -a swap
root@DEC850:~ # gmirror insert swap nda1p4
GEOM_MIRROR: Device swap: rebuilding provider nda1p4.
root@DEC850:~ # GEOM_MIRROR: Device swap: rebuilding provider nda1p4 finished.

[Patrick M. Hausen]

Would you like me to outline the steps to turn the layout into the official one? Alrightee ... I'll bite :-)

Code Select Expand

# stop the mirror operation
zpool detach zroot nda0p3

# nuke the partition table on nda0
gpart destroy -F nda0

# recreate the partition table following the standard
gpart create -s gpt nda0
gpart add -s 532480 -t efi nda0
gpart add -s 1024 -t freebsd-boot nda0
gpart add -a 1m -s 8g -t freebsd-swap nda0
gpart add -a 1m -t freebsd-zfs nda0

You should then be able to add partition 4 - not 3 - of the new disk to the ZFS mirror:

Code Select Expand

zpool attach zroot nda1p3 nda0p4

Once that is completed, we can proceed. I might not be available for much longer today, but as long as we do not do anything destructive to nda1, you should be fine.

Or if you want to just proceed with ZFS and swap partitions reversed compared to the standard - fine. Just tell me how you wish to proceed.

[charles.adams]

Would you like me to outline the steps to turn the layout into the official one? Alrightee ... I'll bite :-)

Code Select Expand

# stop the mirror operation
zpool detach zroot nda0p3

# nuke the partition table on nda0
gpart destroy -F nda0

# recreate the partition table following the standard
gpart create -s gpt nda0
gpart add -s 532480 -t efi nda0
gpart add -s 1024 -t freebsd-boot nda0
gpart add -a 1m -s 8g -t freebsd-swap nda0
gpart add -a 1m -t freebsd-zfs nda0

You should then be able to add partition 4 - not 3 - of the new disk to the ZFS mirror:

Code Select Expand

zpool attach zroot nda1p3 nda0p4

Once that is completed, we can proceed. I might not be available for much longer today, but as long as we do not do anything destructive to nda1, you should be fine.

Or if you want to just proceed with ZFS and swap partitions reversed compared to the standard - fine. Just tell me how you wish to proceed.

I'd like to get things the official layout.

I started down your path but got a message:

Code Select Expand

root@DEC850:~ # zpool detach zroot nda0p3
root@DEC850:~ # gpart destroy -F nda0
gpart: Device busy

Could this be from my having mirrored the swap already?