Add mirrored drive to existing ZFS install

[Moonshine]

I have an existing (single drive) ZFS based install which I'd like to add an (identical) mirror drive to -- ideally with out a re-install and config reload.

I feel the the following should do it, but considering my experience with BSD and ZFS are limited, perhaps someone could confirm?

Output from "gpart show":

Code: [Select]

=>        40  1000215136  ada0  GPT  (477G)
          40      532480     1  efi  (260M)
      532520        1024     2  freebsd-boot  (512K)
      533544         984        - free -  (492K)
      534528    16777216     3  freebsd-swap  (8.0G)
    17311744   982902784     4  freebsd-zfs  (469G)
  1000214528         648        - free -  (324K)

Thinking this should do it?

Code: [Select]

gpart backup ada0 | gpart restore -F ada1
dd if=/dev/ada0p1 of=/dev/ada1p1
dd if=/dev/ada0p2 of=/dev/ada1p2
zpool attach zroot ada0p4 ada1p4

Thanks!

-James

[Patrick M. Hausen]

That's exactly the procedure to do it. You could additionally create a GEOM mirror for swap.

[Moonshine]

Thanks!  I'm going to give it a shot tonight.

I hadn't considered the swap partition, but it seems like that would be safest in case ada0 has a complete meltdown.

I'm am a little curious if OPNsense would normally create a GEOM mirror for swap when choosing ZFS mirroring during install though -- as I generally like to keep as close to "stock" under the hood as possible.

In any case I took a stab at what I feel might be the procedure, but people feel free to pick it apart if I'm way off!

Code: [Select]

swapoff -a
dd if=/dev/zero of=/dev/ada0p3
dd if=/dev/zero of=/dev/ada1p3
gmirror label -v -b round-robin swap /dev/ada0p3 /dev/ada1p3

# Check if there is a mirror now?
gmirror status
ls /dev/mirror/swap

# If so edit /etc/fstab to ..
/dev/mirror/swap none swap sw 0 0

swapon -a

Thanks again,

-James

[Moonshine]

Just some notes and answers for anyone heading down this same path, or future me.

If you do a single drive ZFS install of OPNsense, the drive is drive is actually added to the pool by GPT lablel (gpt/zfs0) rather than partition name (e.g. ada0p3).  So you'll have to take that into account in the zpool attach command.  (Curiously if a mirror is configured during install the drives seem to be added by partition name)

In the end I ran into trouble with the adapter I was going to use to add the second drive.  To the point where I actually re-installed because of all the write errors I saw.    So to answer my question above about swap, when OPNsense does a ZFS mirrored install it appears to add a separate swap partition on each drive and includes both in fstab.

All good now, but was hoping to avoid the re-install and config fire drill.  Glad to see re-applying the backup config works well though!

[ender526]

Hello.  I just used your instructions and everything seems to be working, but the redundancy I was going for, was to still be able to boot if one drive dies.  Since the boot partition isn't a mirror, and fstab only references /boot/efi on one of the drives, not both, how does that work if that's the one that dies?  Do I need to add the second drive boot partition to fstab as well?

My fstab

Code: [Select]

# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ada1p1             /boot/efi       msdosfs rw              2       2
/dev/ada1p3             none    swap    sw              0       0
/dev/ada0p3             none    swap    sw              0       0

My drives

Code: [Select]

# gpart show
=>       40  250069600  ada0  GPT  (119G)
            40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  232757248     4  freebsd-zfs  (111G)
  250068992        648        - free -  (324K)

=>       40  250069600  ada1  GPT  (119G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  232757248     4  freebsd-zfs  (111G)
  250068992        648        - free -  (324K)


[Patrick M. Hausen]

The mountpoint of the EFI partition in FreeBSD is irrelevant for the system to boot. You need to copy the partition contents from the first to the second disk e.g. with dd. The command is in one of the earlier posts in this thread and I guess you already did that.

That's all. Your EFI BIOS must be able to pick up the boot loader on the second disk when the first fails.

The mount point exists to be able to change/update the contents of the EFI partition if necessary.

[ender526]

Great, thanks.  I should be good then.

[Patrick M. Hausen]

You should also use the commands in @Moonshine's post to turn your swap partitions into a mirror.

[ender526]

So I just ran the attach command (confusing because I started with ada1 as my original single drive zfs pool, and am adding ada0, which is the opposite of the OP).

The problem is that since the instructions specify to dd the partitions it also seems to have copied over whatever indicator that says the drive is part of the pool.  So when I try to attach it says ada0p4 is already part of the pool (which it's not), it just thinks it is because of the dd.  Am I safe to pass -f to override?

Code: [Select]

root@firewall:/etc # zpool status
  pool: zroot
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          ada1p4    ONLINE       0     0     0

errors: No known data errors
root@firewall:/etc # zpool attach zroot ada1p4 ada0p4
invalid vdev specification
use '-f' to override the following errors:
/dev/ada0p4 is part of active pool 'zroot'


[Patrick M. Hausen]

Huh? The swap partitions are p3, not p4 ...

So if you dd'ed anything to p4, that broke your pool. Also you do not really need the dd to the swap partitions.

Do a `gpart show` and a `zpool status` again, please, and we will start from there ...

[ender526]

Sorry, I switched topics and didn't clarify.  I wasn't trying to create the swap mirror, I was just trying to execute the attach command after completing the OP's first post of instructions.  I realized what I did though.  Instead of just dd'ing p1 and p2, I also dd'd the zfs partition (p4), so it copied over the zfs metadata.  That was my mistake.  I wiped the first 512K and last 512K from p4 on the new drive (which is where the zfs meta data lives) and tried attaching again.  It worked fine.

It only took 7 seconds to re-silver.  I'm hoping that was because the partitions were already almost identical, minus the first 512K and last 512k.

Thank you for taking the time to answer my questions.

[abulafia]

This looks horrible from a ZFS perspective :-)

Does FreeBSD really set up partitionson disk  instead of datasets in a ZFS pool?

I would have thought that basically the following should suffice:

Code: [Select]

Zpool attach zroot gpt/whateveryourdrivenameis

[Patrick M. Hausen]

This is sufficient. FreeBSD uses a single partition of type freebsd-zfs to store pool data and of course uses datasets.

But in general you need

- an EFI boot partition or
- a legacy boot loader partition
- a swap partition

all outside of ZFS.

This thread is about how to create these if you build a mirror setup after installation. The zpool attach is easy, but what good is a second disk if there is no boot loader on it?

[abulafia]

But in general you need

- an EFI boot partition or
- a legacy boot loader partition
- a swap partition

all outside of ZFS.

Thanks - I'm more familiar with OmniOS, which as far as I am aware does not use partitions and where you would just "install" the bootloader after "attaching" a second disk (https://omnios.org/info/migrate_rpool.html). That's Illumos and not *BSD, though.

[Patrick M. Hausen]

There's a current discussion about whether FreeBSD should get a bootadm command. The partition layout is due to the PC architecture, your OmniOS example seems to be Sparc? At least the device names hint at that.