I need an overall storage size of around 60 TB and I am wondering about the best pool layout in my case.
My first idea was to simply use a RAIDZ3 with 7x 16TB which would result in around 64 TB of storage.
However, I am also thinking about a two pool setup consisting of
a 3-way-mirror with 3x 16TB, resulting in 16 TB storage
and a RAIDZ2 with 4x 20TB, resulting in 40 TB storage,
so in sum around 56 TB of total storage.
Which setup is better for the safety of the stored data?
With the RAIDZ3 there could fail 3 drives without losing data, with the other setup a pool could already be lost if the “wrong” 3 drives fail. However, the resilvering with the RAIDZ3 will take very long, while in the setup with the two pools at least the mirrored pool will be resilvered quickly.
The performance (speed) of the pools is not my primary goal, the data safety is more important.
Mind that it is recemmended to keep pools below 80% occupancy. So if you plan to have 60 TB of data in a few years (plan a little ahead…) you rather need 75-80 TB of storage.
I assume this is “bulk” storage, to be shared with NFS or SMB, not block storage (VMs, iSCSI). So raidz3 would be the call. With backup(s), offsite and offline, for the sake of safety.
What’s the reason for considering two pools with different layouts?
Unless “all the other data” is an awful lot of very small files which would do badly in raidz, you’ll be better with a single raidz2 or raidz3 vdev. Keep it simple.
Adding on to @etorix post, and if you do have a ton of those very small files, you can still not worry about raidz if you can create a zfs special device for them.
A dedicated, faster (SSD), vdev for metadata and small files (user-defined limit).
Mind that losing this vdev means losing the pool, so you’d need a 3-way or rather 4-way mirror to match the resiliency of the slower raidz3 data vdev.
If performance is not the primary goal, I would not pursue this lead.
