OPNsense Forum
English Forums => 23.7 Legacy Series => Topic started by: tdalej on January 25, 2024, 12:00:31 am
-
I have a Supermicro 1U Server I'm (going) to use as a firewall.
4 500GB enterprise SATA disks, so ZFS raidz3 might be a good approach.
It ran so long I stopped it and tried another firewall product that can use ZFS and on the same hardware installation time is minutes, not hours.
I'm running the installation of opnsense 23.7 again ... I'm 3+ hours in and the screen says 38%.
Is this normal?
Anyone else using ZFS?
-
It should be minutes.
-
It was not.
It was hours.
Between the aborted install and when I let it run to completion, I installed another firewall product and ESXi 7.0U3n.
They installed normally and within minutes.
Most likely not a hardware issue -- not a lot of people using ZFS?
-
I'm using ZFS practically everywhere. Installations take seconds to minutes.
What exactly is this server? Does it have a RAID controller to connect the SATA drives?
-
I agree that the install shouldn't take very long. Also, I would probably go with mirrors over z3 in this setup as you don't really need a ton of space and more iops is better.
-
It's a:
Supermicro 1U Firewall Server W/ X10SLH-N6-ST031
Processor: Xeon E3-1270 v3 3.5Ghz 4-Core Processor
Memory: 32GB (4x 8GB) DDR3 ECC Unbuffered Memory
Storage Controller: Integrated Storage Controller
Using 4 onboard SATA ports of 6.
Onboard storage controller in AHCI mode.
In the BIOS the storage controller has three settings: (from memory, the BMC doesn't have any storage info)
RAID
AHCI
IDE
The RAID mode doesn't seem to have any effect.
-
AHCI is good. Could you try installing to a mirror instead of the RAIDZ3?
-
I was just modifying my previous reply and lost it when my session timed out :/
Other than the brutally slow install it seems to operate comparably to anything else I have installed -- once it's on the appliance.
If there are logs I can collect that may help I'll slog through those hours again -- but if it's just to watch the paint dry again I'd rather not :)
-
I wonder if the install process for some reason decides to write all blocks since it is raidz3? That seems a bit odd thought, but it could explain the time?
-
Let's not forget. raidz is just a type of organising the vdevs one way or another. If you put aside the technicalities of how many disks for parity, how many you can lose before losing the pools, which one raidzX is faster to read or write, all that doesn't matter really for such a small number of physical devices and such a generic pattern of reads and writes, as oppossed to say databases. Read: any choice of raidz for this is good for "speed".
Then you have the zfs filesystem on top. Again, for OPN, is in terms of speed, almost as fast/slow in magnitude as the underlying physical media. Yes mirrored is faster than not. NVMe faster than SSD, etc. All in all, same ballpark.
That said, to the OP original question: no, it should not take longer than any other storage configuration. Again, ballpark. It should be minutes.
No, the installer I expect will just create the vdevs, the main pool and the filesystems. Takes seconds.
Then install by copying the files. This will be the longest part but again seconds, maybe a few mins.
As to why it went this slow? AHCI has been checked, so all I can think of is some problem with storage subsystem i.e. physical. You might want to keep an eye on the io stats.
-
The unit with zfs isn't in use yet, so I don't know how useful looking at IO might be.
I did install os-smart plugin and although the UI widget doesn't appear to function (setup instructions at install time are unclear) smartctl output shows the drives healthy.
I just initiated a long test on each drive so will see how that turns out.
I'm not that familiar with ZFS, much more familiar with mdadm, but with 4 500GB drives and raidz3 I expected a 500GB volume with three mirrors, but instead it looks like 1TB volume.
Until I get some other issues resolved I'm temporarily running 23.7.12 on a Dell 1950 with a PERC for two SAS drives.
If I can get the routing issue described in another thread resolved I'll move the (newer) hardware into use with ZFS.
-
I'm not that familiar with ZFS, much more familiar with mdadm, but with 4 500GB drives and raidz3 I expected a 500GB volume with three mirrors, but instead it looks like 1TB volume.
This would be wrong in assumption, you will end up with 3 parity bits distributed over all discs. That is not the same as mirror - not even close.
-
Did you possibly create a RAIDZ2? Would be a recommended configuration for 4 drives.
-
...would make sense
https://wintelguy.com/raidcalc.pl
-
I wonder if the install process for some reason decides to write all blocks since it is raidz3? That seems a bit odd thought, but it could explain the time?
That's not the way ZFS works. It doesn't have to write to the entire drive on initialization like a raid card.
Let's not forget. raidz is just a type of organising the vdevs one way or another. If you put aside the technicalities of how many disks for parity, how many you can lose before losing the pools, which one raidzX is faster to read or write, all that doesn't matter really for such a small number of physical devices and such a generic pattern of reads and writes, as oppossed to say databases. Read: any choice of raidz for this is good for "speed".
Then you have the zfs filesystem on top. Again, for OPN, is in terms of speed, almost as fast/slow in magnitude as the underlying physical media. Yes mirrored is faster than not. NVMe faster than SSD, etc. All in all, same ballpark.
Agreed. I just don't see the benefit in using z3 with 4 drives. If you're that concerned about losing drives, do a 4 way mirror.
A pair of striped mirrors might make a difference depending on the plugins installed. Not sure as I haven't done any testing in that regard.
-
As I said, I don't know much about ZFS.
500GB of space is overkill for my setup, 855GB is even more overkill.
I only brought this up because another product lists ZFS first in the list of filesystem options and creates an identically configured ZFS filesystem in a fraction (very small fraction) of the time.
If I were creating an mdadm RAID it's the difference between using --assume-clean and not.
Is there a comparable flag for ZFS?
It's not _that_ critical of an issue as I will only install maybe a couple more times before try a recovery install before I put this in use.
What bothers me a _lot_ more is the routing issues I have hit :/
-
Could be more critical than you think. Can you afford this rebuild time once is in production? If yes, no worries.
It should not take long, just a few minutes. If it doesn't, something is wrong and is not zfs itself.
About storage requirements: With your spec, I would use only two discs in mirror config for OS.
In your shoes, I'd like to know why it does what it does before commissionig for production use.
-
I agree with CM. Something is screwy with your system. It could be that one of your discs is failing or something else.
I would start with SMART tests and maybe even a destructive burn in test for the drives, as well as running a memory test.