Hello everyone, newbie here when it comes to nas/san. I am building my first one and currently have (5) remanufactured (5yr warranty) 10TB drives. They’re all connected via sff-8643 to sata thru an SAS LSI card (yes it’s in IT mode). This new PC build has 64gb ram and 2x2tb NVMEs as well on a ryzen 8600GT AM5. My chassis has the capacity to hold a total of 7 HDDs.
I have been reading for the past couple of weeks and getting familiar with raid/zfs raids (raidz), reading the 180+ pages of truenas release notes and documentation, playing with calculators, etc.
My priority is redundancy/resiliency, and second performance. I already have a 20TB HDD backup of all my stuff in my DR site and plan on doing incrementals every month.
I’m looking for opinions on what is the best strategy going forward. I know there are hundreds of different opinions and that’s where I get lost so I figure I’d ask here.
My first thought was get all 5 HDDs in a raidz2, but what I gathered from tons of reading is that you want small vdevs and avoid large arrays since it increases the change of failure during resilvering. Raidz3 is probably the worst option (again based on what I’ve read) and z2 still has some risk and a performance penalty.
So I am thinking of buying a 6th drive and doing two vdevs, 3 drives each in raidz1. Another option that I am considering is doing (2) vdevs of 2 HDDs each mirrored, and then mirror both vdevs, and have two cold standby drives.
Can I get opinions? Is there a flaw in any of plans above? What do you all recommend?
Eh, kind of. Yes, the larger the vdev (both in terms of number of devices, and in terms of capacity), the larger the chance of failure during resilvering, ceteris paribus. However, ceteris is not paribus. Comparing a 3-disk RAIDZ1 against a 6-disk RAIDZ2, assuming one failed disk in each, the likelihood of a second disk failure during resilvering is greater in the 6-disk RAIDZ2, but the consequence of such a failure is far greater (indeed, catastrophic) in the 3-disk RAIDZ1.
If you compare one 6-disk RAIDZ2 vdev against 2 3-disk RAIDZ1 vdevs, again assuming the failure of one disk, a second disk failure while resilvering carries a 0% chance of pool loss in the RAIDZ2 pool, and a 40% chance of pool loss in the RAIDZ1 pool.
When you have more than one vdev in a pool, the vdevs are always striped; they cannot be mirrored or put in any other arrangement.
5 or 6 drives in Raid-Z2 or Raid-Z3 would be fine. I am guessing you can replace any failing drives quickly since this is your primary NAS and you have a DR site backup. We wouldn’t recommend Raid-Z1 but Z2 or Z3 should be fine with your disk size.
5-6 disks would be considered small VDEVs with Raid-Z2 or Z3.
All correct, but 5-wide is small for raidz2. (“Too large” would be beyond 10.)
If your use case is storing mostly large files for sharing with SMB or NFS, just go for raidz2.
" All correct, but 5-wide is small for raidz2. (“Too large” would be beyond 10.)
If your use case is storing mostly large files for sharing with SMB or NFS, just go for raidz2."
Thanks for confirming that. I never saw anywhere what is considered small, so with that I’ll get a 6th drive and add it to the z2. Yes, my use case is storing 4k movies and flac audio. I will have a small separate partition for storing about 4TB of old files/historical documents.
There’s no hard limit, but about 10-12 is generally regarded as the “too large” limit for raidz (depending on level, raidz3 may be safe at 12 but raidz1 would be risky at 10).
Raidz2 needs at least 4 drives, so that gives you an idea where 5 may stand on the scale…
Thanks for the reply. I ended up buying a 6th drive, and plan on using it as a cold spare for when one of the drives in the raidz dies, it’ll be there ready to resilver.
Is there a reason for a cold spare and not just adding it to the pool as an additional member of Raid-Z2 or Z3? A cold spare is helpful if you can’t get to the physical machine to replace a disk in a timely manner but if you have access, it may be better just as another member
Mostly convenience and laziness. It was also a suggestion from a friend that manages enterprise SANs at his org, and runs zfs at his house using almalinux.
Plus, I figure if I keep adding drives it goes back to the original problem of increase in risk of failure. And not doing z3 per all previous comments and suggestions in the thread
