Messages

I understand a company needs to make money. I am therefor happy to pay for my home subscription. If multicore support won't be in the free version, ok for me. Not nice, as multicore is plain standard nowadays as you other guys correctly stated, but ok.

It won't be ok however not to include multicore in home subscription. The upper plans are too pricey for my home purposes. I really think to quit home subscription, as I do not agree with that policy.

Think twice, if I were you, I would offer multicore for free, or include in home subscriptions whatever, to get more customers. Otherwise I guess you would lose them...

Locking multi-core support and limiting the number of policies in the home licensing is really gimping that tier IMO.  Making the tool cost prohibitive at a home license level. 

Just look how much Oracle charges per core... ROFLMAO...

I think you are confusing multi-core support with per-core licensing. Per-core licensing cost is pretty common across the enterprise stack. 

Oracle ZFS docs apply to Oracle hardware (former Sun) only, neither Linux nor FreeBSD.

FreeBSD uses GPT partitions. As do Ubuntu and Debian, if you follow the guide by zfsbootmenu.org. If you follow the OpenZFS guide for Ubuntu, it's /dev/disk/by-id.

FreeNAS and TrueNAS use GUUIDs. It's complicated.

Now the fact that the FreeBSD handbook is so outdated it is blatantly wrong - does not match in any way what the FreeBSD installer will (correctly) do - needs to be addressed. I'll poke some people.

Confusing is the big key word here, I'm a slight noob with FreeBSD and a full on noob with ZFS I've just been lost this whole time.

Regardless I appreciate the help and guidance here.

All good. I'd be interested into those guides you mentioned. If there is misleading documentation out there, we ought to do something about that.

https://sotechdesign.com.au/how-to-add-a-drive-to-a-zfs-mirror/

https://docs.oracle.com/cd/E53394_01/html/E54801/gayrd.html

https://askubuntu.com/questions/1301828/extend-existing-single-disk-zfs-with-a-mirror-without-formating-the-existing-hdd

https://www.devroom.io/2024/03/07/zfs-upgrade-single-disk-to-mirror/

I can go on and on, the issue is I didn't look up a freebsd specific guide. Figured ZFS was ZFS and a mirror is a mirror.  Ultimately I didn't know what I didn't know, so I guess my search was flawed.  But everything seemed to mostly agree, so figured that was the right path, and in my head a mirror is well... a mirror. I think in raid controller, on an old school RAID card I'd add in my disk, add it to the mirror, let it resilver/sync, go about my life and never think of it again.

Edit: I understand why this wouldn't have worked now, with a RAID controller, the OS and UEFI bootloader just points at the card instead of disks directly unlike ZFS.  I know know that boot info would have to exist on both disks in this instance.  IF it was just for storage I assume all that wouldn't be needed and the above guides would have been accurate.   

ada1p1, yes. Sorry.

ok, copies are done.  Waiting on resilver to finish, estimated to be about an hour. 

Appreciate the detailed guidance. If you have a way to accept it, I'd be happy to buy you a beer or coffee/tea depending on your elixir of choice for your trouble. 

Is the source wrong here dd if=/dev/da1p1 of=/dev/ada0p1 bs=1m?  I get "dd: /dev/da1p1: No such file or directory".   Assuming source should be /dev/ada1p1 but don't want to assume again.

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:08:05 with 0 errors on Fri May 23 17:33:06 2025
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          ada1p4    ONLINE       0     0     0

errors: No known data errors


Code Select Expand

gpart show
=>       40  468877232  ada1  GPT  (224G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  451563520     4  freebsd-zfs  (215G)
  468875264       2008        - free -  (1.0M)

Weird the device name changed again, now it's ada1p4 vs ada0p4 when it was broken.  I'm so confused by that naming.

OK. This is the state that I wanted and with which I intended to guide you through the recovery process step by step. Now that your zpool is a bit messed up let's fix that first.

I assume

Code Select Expand

zpool status
results in "ada1p4" and "ada0" as the mirror disks?

We need to remove that ada0:

Code Select Expand

zpool detach zroot ada0


Now we take a breath and grab a coffee ... about those device names ...

FreeBSD enumerates the devices by some "hardware order" inherent in the drive, the PCIe bus, whatnot. Starting with 0.

So initially you had ada0 and ada1. Fine. Then ada0 failed. You removed it and rebooted. With only a single drive now present what was formerly ada1 is now ada0. It starts at 0. Always.

Then you inserted a factory new drive in the "first" (whatever that means) hardware position. After another boot that one is now "first" and becomes ada0 and what was initially ada1, then ada0, is now ada1 again.

FreeBSD just counts.


Now the boot process. For a PC system to be able to boot there needs to be a partition table and either - depending on the system - legacy ("BIOS") or EFI boot code in a matching partiton. When you install stock FreeBSD you can pick which to install. OPNsense installs both, just so not to bother the user with questions they cannot answer and always be able to boot, even if you replace your hardware and move your drive from e.g. a legacy system to am EFI system.

You can see that in your "gpart show" output. An EFI partition followed by a freebsd-boot (legacy) partition. Followed by swap and ZFS. ZFS must go into a partition of type freebsd-zfs, never to the whole disk.

You need the "boot thingies" on both disks, because you want to be able to boot of either of them in case one fails.


So now if that removal of ada0 succeeded first we create a partition table. The easiest way in case of identical drives is to copy it from the good one to the new one:

Code Select Expand

gpart backup ada1 | gpart restore ada0

Should the "new" drive not be entirely new and the above command fail because gpart does check if there is a partition table present, already, you can add the "-F" flag to that "gpart restore" command. It's just a reasonable safety measure. But since your new drive never had a partition table it should go well without "-F".

You can then check with

Code Select Expand

gpart show
that now both drives are partitioned the same.


Now that we have a ZFS partition to keep or zpool data we can attach that to the mirror:

Code Select Expand

zpool attach zroot ada1p4 ada0p4

Didn't it appear odd to have "ada1p4" but just "ada0" without a partitin when you did it the first time?

Anyway the zpool should now be resilvering and be done in no time as you can check with

Code Select Expand

zpool status
again.

Good? Next step, copy that boot code.


We copy both the EFI and the legacy partitions from ada1 to their respective counterparts on ada0:

Code Select Expand

# copy EFI boot
dd if=/dev/da1p1 of=/dev/ada0p1 bs=1m

# copy legacy boot
dd if=/dev/da1p2 of=/dev/ada0p2 bs=1m


That's it. Grab a beer. You have a redundant bootable system again. If you want redundant swap, too, which I recommend, we can do that in another round after your system is healthy again.


Kind regards,
Patrick

ok, resilvering again, hopefully correctly this time.

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat May 24 10:50:06 2025
        137G / 137G scanned, 2.20G / 137G issued at 119M/s
        2.24G resilvered, 1.61% done, 00:19:21 to go
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p4  ONLINE       0     0     0
            ada0p4  ONLINE       0     0     0  (resilvering)

errors: No known data errors

Once that finish (assume I should wait till resilver has finished) I'll copy the boot stuff.

Nooooo!

That's exactly how not to do it!

Your system will not be able to boot, when ada1 fails.

Why can't you guys wait for me to react to the post with the info I asked for. Good grief!

Ugh, sorry, EVERY guide I've read showed that was the process to rebuild. 

I"m disappointed in ZFS, thought it was going to be a huge value add.  But I'm really thinking about taking all my stuff back to using a classic raid controller.  ZFS documentation is poor and confusing and covered in trip wires and mines.   

So this seemed to work:

Code Select Expand

zpool attach zroot ada1p4 ada0

Shows resilvering now, but the naming looks weird:

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat May 24 09:55:56 2025
        137G / 137G scanned, 3.30G / 137G issued at 113M/s
        3.34G resilvered, 2.41% done, 00:20:13 to go
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p4  ONLINE       0     0     0
            ada0    ONLINE       0     0     0  (resilvering)

errors: No known data errors


Code Select Expand

zpool attach zroot ada0
missing <new_device> specification
usage:
        attach [-fsw] [-o property=value] <pool> <device> <new-device>

I'm not sure what this is looking for, does it need this?

Code Select Expand

zpool attach zroot ada1 ada0

No. Because the disks are just enumerated, your old disk can well have been ada0 before, but is now detected after the new disk, thus this new disk takes the name ada0 now and the old disk now has ada1.

Therefore, you have to attach ada0, not ada1 now.

That's confusing as all heck. Alright let me see what it does, I guess it won't let me attach one that's already attached anyway. 

So I just need to run this?

Code Select Expand

zpool attach zroot ada1

The device names are just numbered as devices are being detected, this is why they are unreliable. It looks like ada0 is present, but the disk is not yet partitioned, henc is why it does not come up with "gpart show".

Given that your old device is now ada1, there must be an ada0 device. You can make sure using "camcontrol devlist".

Code Select Expand

<SanDisk SSD PLUS 240GB UF4500RL>  at scbus0 target 0 lun 0 (pass0,ada0)
<SanDisk SSD PLUS 240GB UF4500RL>  at scbus2 target 0 lun 0 (pass1,ada1)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus6 target 0 lun 0 (ses0,pass2)


Please post a

Code Select Expand

zpool status
gpart show
Kind regards,
Patrick

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:08:05 with 0 errors on Fri May 23 17:33:06 2025
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          ada1p4    ONLINE       0     0     0

errors: No known data errors


Code Select Expand

gpart show
=>       40  468877232  ada1  GPT  (224G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  451563520     4  freebsd-zfs  (215G)
  468875264       2008        - free -  (1.0M)

Weird the device name changed again, now it's ada1p4 vs ada0p4 when it was broken.  I'm so confused by that naming.

Download + Edit offline + Restore?

That's funny enough what I was just looking at, seems like it should work as long as you don't miss any.  Might try one of my less critical vlans and see what the outcome is.

Initial test seems to have worked without issue. 

Ok, I think the renaming via config file modification seems to have been successful.