how To Add Secondary SSD

[ticker]

I am looking to add a second SSD.
Upon first install, I installed using ZFS on a single SSD.
I recently acquired another SSD and I wanted to mirror it. I know it's an option on first install.

I am looking to avoid needing to backup, and fresh install.
How can I add it in, without fresh installing?

[Patrick M. Hausen]

Assuming your first SSD is ada0 and your second ada1. Further assuming your partition table looks like this (apart from the sizes, probably):

Code: [Select]

root@opnsense:~ # gpart show ada0
=>       40  537234688  ada0  GPT  (256G)
         40     409600     1  efi  (200M)
     409640       1024     2  freebsd-boot  (512K)
     410664        984        - free -  (492K)
     411648   16777216     3  freebsd-swap  (8.0G)
   17188864  520044544     4  freebsd-zfs  (248G)
  537233408       1320        - free -  (660K)

You can go about the task like this:

Code: [Select]

# copy partition table
gpart backup ada0 | gpart restore -F ada1
# copy EFI partition
dd if=/dev/ada0p1 of=/dev/ada1p1
# copy legacy boot loader partition
dd if=/dev/ada0p2 of=/dev/ada1p2
# attach second disk to ZFS pool
zpool attach zroot ada0p4 ada1p4
# turn swap partition into mirrored device
gmirror load
swapoff -a
gmirror label -b round-robin swap ada0p3
gmirror configure -a swap
gmirror insert swap ada1p3

Change the file /etc/fstab to look like this:

Code: [Select]

# Device Mountpoint FStype Options Dump Pass#
#
/dev/mirror/swap none swap sw 0 0

Re-enable swap:

Code: [Select]

swapon -a
HTH
Patrick

[hushcoden]

Many thanks for the instructions, Patrick.

I did add a second SSD, but in my case ada1 is the original one and ada0 is the second one:

Code: [Select]

root@hush:/home/pippog # gpart show
=>       40  250069600  ada1  GPT  (119G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  232757248     4  freebsd-zfs  (111G)
  250068992        648        - free -  (324K)

and

Code: [Select]

root@hush:/home/pippog # gpart show ada0
gpart: No such geom: ada0.

So, all I have to do is to invert ada0 and ada1 in Patrick's instructions, right?

If so, are those correct?

1. Assuming your first SSD is ada1 and your second ada0:

Code: [Select]

# copy partition table
gpart backup ada1 | gpart restore -F ada0

# copy EFI partition
dd if=/dev/ada1p1 of=/dev/ada0p1

# copy legacy boot loader partition
dd if=/dev/ada1p2 of=/dev/ada0p2

# attach second disk to ZFS pool
zpool attach zroot ada1p4 ada0p4

# turn swap partition into mirrored device
gmirror load
swapoff -a
gmirror label -b round-robin swap ada1p3
gmirror configure -a swap
gmirror insert swap ada0p3
2. Change the file /etc/fstab to look like this:

Code: [Select]

# Device     Mountpoint   FStype    Options     Dump      Pass#
#
/dev/mirror/swap     none   swap    sw     0         0

3. Re-enable swap:

Code: [Select]

swapon -a

Tia.

[Patrick M. Hausen]

Looks good. Make it so 

[ticker]

Hey so just getting around to trying this.
Sounds good, my new drive is also going to be ada0.
However when running

Code: [Select]

gpart backup ada1 | gpart restore -F ada0I run into a

Code: [Select]

gpart: size '983025664' invalid argument error. I assume this is because the new drive is smaller in size, so the partition sizes won't match.

Can you please help clarify how to safely proceed. Thanks.

Here is the disk layout

Code: [Select]

# geom disk list
Geom name: ada0 (new second disk adding in)
Providers:
1. Name: ada0
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Mode: r0w0e0
   descr: SanDisk SDSSDH3 250G
   lunid: 5001b448b243e95c
   ident: 230206A0004C
   rotationrate: 0
   fwsectors: 63
   fwheads: 16

Geom name: ada1 (original disk)
Providers:
1. Name: ada1
   Mediasize: 512110190592 (477G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: SanDisk SDSSDH3 512G
   lunid: 5001b448ba4679ef
   ident: 21120U801225
   rotationrate: 0
   fwsectors: 63
   fwheads: 16


[ticker]

..also for clarification I will never come close to even using 100gb (nevermind the 250-512gb which the drives are) so size should not be an issue.
I think I'm only using 2.2GB/452 on my current drive haha

[Patrick M. Hausen]

Short answer: You cannot add a smaller drive. Period. Do a fresh install. Then reimport your saved configuration.

Longer answer: You cannot shrink a ZFS vdev. Fundamentally impossible. So you can only ever attach equal size or larger disks to an existing vdev. You can do a mirrored install to a set of differently sized disks, but the resulting size will be that of the smaller disk. Since you already have a running installation the size of the larger disk, the only way is a clean reinstall.

[clarknova]

Thank you for this guide. The first steps work well for me, and then this:

Code: [Select]

root@OPNsense:~ # gmirror load
gmirror: Command 'load' not available; try 'load' first.
I searched this error and got a few hits. One was way over my head and the others didn't offer anything that appeared to help my situation.

This isn't a show-stopper for me, as it's just the swap partitions that are failing to mirror, but I'm still interested in any insights people may have on this error.

Edit: OPNsense 23.1-amd64
I haven't run updates since installing 23.1 fresh.

[Patrick M. Hausen]

Possibly it is already loaded? What is the result of "kldstat"?

[hushcoden]

So, with great courage I went ahead, restarted the firewall and yes, it worked 

How do I check the mirroring is up and runing? 

And is there way for newbies to monitor the mirror ?

[Patrick M. Hausen]

zpool status
gmirror status

No dashboard widget yet, as far as I know.

[hushcoden]

Sorry, can you please confirm if it looks fine?

Tia.

[Patrick M. Hausen]

Yes, it does. It will tell you with words quite different from anything like "complete" or "online" if anything breaks.