Installation with ZFS - how long does it normally take?

[tdalej]

I have a Supermicro 1U Server I'm (going) to use as a firewall.
4 500GB enterprise SATA disks, so ZFS raidz3 might be a good approach.

It ran so long I stopped it and tried another firewall product that can use ZFS and on the same hardware installation time is minutes, not hours.

I'm running the installation of opnsense 23.7 again ...  I'm 3+ hours in and the screen says 38%.

Is this normal?
Anyone else using ZFS?

[lar.hed]

It should be minutes.

[tdalej]

It was not. 
It was hours.

Between the aborted install and when I let it run to completion, I installed another firewall product and ESXi 7.0U3n.

They installed normally and within minutes.

Most likely not a hardware issue -- not a lot of people using ZFS?

[Patrick M. Hausen]

I'm using ZFS practically everywhere. Installations take seconds to minutes.

What exactly is this server? Does it have a RAID controller to connect the SATA drives?

[CJ]

I agree that the install shouldn't take very long.  Also, I would probably go with mirrors over z3 in this setup as you don't really need a ton of space and more iops is better.

[tdalej]

It's a:
Supermicro 1U Firewall Server W/ X10SLH-N6-ST031
Processor: Xeon E3-1270 v3 3.5Ghz 4-Core Processor 
Memory: 32GB (4x 8GB) DDR3 ECC Unbuffered Memory
Storage Controller: Integrated Storage Controller

Using 4 onboard SATA ports of 6.
Onboard storage controller in AHCI mode.

In the BIOS the storage controller has three settings: (from memory, the BMC doesn't have any storage info)
RAID
AHCI
IDE

The RAID mode doesn't seem to have any effect.

[Patrick M. Hausen]

AHCI is good. Could you try installing to a mirror instead of the RAIDZ3?

[tdalej]

I was just modifying my previous reply and lost it when my session timed out :/

Other than the brutally slow install it seems to operate comparably to anything else I have installed -- once it's on the appliance.

If there are logs I can collect that may help I'll slog through those hours again -- but if it's just to watch the paint dry again I'd rather not :)

[lar.hed]

I wonder if the install process for some reason decides to write all blocks since it is raidz3? That seems a bit odd thought, but it could explain the time?

[cookiemonster]

Let's not forget. raidz is just a type of organising the vdevs one way or another. If you put aside the technicalities of how many disks for parity, how many you can lose before losing the pools, which one raidzX is faster to read or write, all that doesn't matter really for such a small number of physical devices and such a generic pattern of reads and writes, as oppossed to say databases. Read: any choice of raidz for this is good for "speed".
Then you have the zfs filesystem on top. Again, for OPN, is in terms of speed, almost as fast/slow in magnitude as the underlying physical media. Yes mirrored is faster than not. NVMe faster than SSD, etc. All in all, same ballpark.

That said, to the OP original question: no, it should not take longer than any other storage configuration. Again, ballpark. It should be minutes.
No, the installer I expect will just create the vdevs, the main pool and the filesystems. Takes seconds.
Then install by copying the files. This will be the longest part but again seconds, maybe a few mins.

As to why it went this slow? AHCI has been checked, so all I can think of is some problem with storage subsystem i.e. physical. You might want to keep an eye on the io stats.

[tdalej]

The unit with zfs isn't in use yet, so I don't know how useful looking at IO might be.
I did install os-smart plugin and although the UI widget doesn't appear to function (setup instructions at install time are unclear)  smartctl output shows the drives healthy.

I just initiated a long test on each drive so will see how that turns out.

I'm not that familiar with ZFS, much more familiar with mdadm, but with 4 500GB drives and raidz3 I expected a 500GB volume with three mirrors, but instead it looks like 1TB volume.

Until I get some other issues resolved I'm temporarily running 23.7.12 on a Dell 1950 with a PERC for two SAS drives.
If I can get the routing issue described in another thread resolved I'll move the (newer) hardware into use with ZFS.

[lar.hed]

I'm not that familiar with ZFS, much more familiar with mdadm, but with 4 500GB drives and raidz3 I expected a 500GB volume with three mirrors, but instead it looks like 1TB volume.

This would be wrong in assumption, you will end up with 3 parity bits distributed over all discs. That is not the same as mirror - not even close.

[Patrick M. Hausen]

Did you possibly create a RAIDZ2? Would be a recommended configuration for 4 drives.

[chemlud]

...would make sense

https://wintelguy.com/raidcalc.pl

[CJ]

I wonder if the install process for some reason decides to write all blocks since it is raidz3? That seems a bit odd thought, but it could explain the time?

That's not the way ZFS works.  It doesn't have to write to the entire drive on initialization like a raid card.

Let's not forget. raidz is just a type of organising the vdevs one way or another. If you put aside the technicalities of how many disks for parity, how many you can lose before losing the pools, which one raidzX is faster to read or write, all that doesn't matter really for such a small number of physical devices and such a generic pattern of reads and writes, as oppossed to say databases. Read: any choice of raidz for this is good for "speed".
Then you have the zfs filesystem on top. Again, for OPN, is in terms of speed, almost as fast/slow in magnitude as the underlying physical media. Yes mirrored is faster than not. NVMe faster than SSD, etc. All in all, same ballpark.

Agreed.  I just don't see the benefit in using z3 with 4 drives.  If you're that concerned about losing drives, do a 4 way mirror.

A pair of striped mirrors might make a difference depending on the plugins installed.  Not sure as I haven't done any testing in that regard.