When doing research before deciding how to finalize my first TrueNAS server build, one thing that surfaces in many threads on the forums, Reddit, and elsewhere is that RAIDZ1 is not for “big disks” or “wide vdevs”. But the posts that goes in to the specifics, i.e. mentions actual sizes, are often at least five years old – and they claim that as soon as your drives are bigger than 1TB (or even 750 GB!), RAIDZ1 should not be considered: Either mirrors or RAIDZ2.
If 1 TB was still considered to big for RAIDZ1 in 2025, I guess I would have seen more recent posts on the forum strongly advice against ever using RAIDZ1.
So, what does the wisdom of the crowd say: In 2025, how big (drive size and width) would you build a RAIDZ1 vdev?
EDIT: And a follow-up question already now: If the cutoff for when RAIDZ1 is out of the question have gone up, is that because more performant CPUs etc makes resilvering faster today, narrowing the window for more disk failures?
My opinion is roughly the same as @Fleshmauler, though not as high as 8TB. Basically, not too wide and not too large.
I’d say the width of the RAID-Z1 vDev would have an effect on the size of the disks. So, maybe something like this;
3 x 5TB
4 x 4TB
5 x 3TB
6 x 2TB
Part of the issue, as has been stated, is time. So, fewer columns for larger disks means fewer reads to replace a disk. And more columns for smaller disks means less data to replace a disk.
Going wider adds in the variable of more hardware needed when reconstructing a full width stripe.
It is all about what a person wants for risk mitigation. As long as a user knows they have 1 disks worth of redundancy, and a higher chance of URE, (Un-correctable Read Error), then let them do what they want.
Personally, I don’t like seeing someone do something they thought was good and solid and find out they did not know the risks. (And the risks of permanent data loss.)
Raid fault tolerance level only improvdes uptime and degrades performance. Nothing else. That said, in my opinion there is no limit. I build arrays based on other requirements and most of my arrays are 3-4 raidz1 vdevs of 4 disks each, 18-25TB each disk.
Note, when you are replacing the disk in raidz1, the vdev maintains fault tolerance level throughout the replacement, provided you first add new disk and only then remove the old. In that sense it behaves like conventional raid6, with respect to “secondary failure” fear mongering.
Raidz2 and higher are only appropriate when you absolutely cannot tolerate any downtime, mission critical applications, on a very high quality hardware (otherwise likelihood of other hardware failing (ram, MLB, power supply) is going to be much higher than than of your array making the point moot).
Not buying new disks and sticking to used disks from a secondary market is another great way to mimize failure (see bathtub curve).
So, answering your question – no limit. Choose arry geometry based on performance and cost, not fear.
…and provided you’re doing the disk replacement while the old disk is still online. But that isn’t the situation people are concerned about; they’re concerned about a hard failure of a disk, which means a complete loss of redundancy in that vdev–which in turn means that any data errors on the remaining disks will be uncorrectable.
Well, kind of. The near end of that curve has been absorbed by someone else, but the far end is rather closer than for a new disk.
Yep. But OP said above “RAIDZ1 should not be considered: Either mirrors or RAIDZ2.”.
I am more wanting to squash the old adage that a single URE error stops a rebuild, i.e., resilver. It CAN with consumer drives at times, but otherwise it should not. Therefore, the so called “risk” of a RaidZ1 resilver is not what it’s all cracked up to be historically. Mirrors do have the advantage of much faster resilver though.
Small RaidZ1 vdevs can give some decent performance vs a single Raidz2 vdev. I personally have no issue with smallish RaidZ1 vdevs, with proper backups and assuming no 99.999% uptime requirement. But one should always backup anyway, even RaidZ3. Would I use a Raidz1 in an enterprise though? Almost certainly not. I say almost as someone usually comes up with some example.
Well, saying that something should(!) not happen and therefore a risk is more or less non-existent, is an interesting line of argument. Risk is the possibility of a negative event happening. How probable that is going to be, is a property of that risk.
If a negative event is severe enough, I still take precautions. Example: In over 30 years of driving a car I have never needed my seat belt. Will I therefore stop using it? Of course not.
Risk is about probability and by that statistics. The latter is fundamentally different from “usual” mathematics and therefore evades/fools gut feeling for most people. In university I got a B- in my statistics exam, when 50% of the people failed it completely. Still I would never trust my gut when it comes to such topics.
Coming back to the practical side of things: I am running a RAIDZ2 pool with 8x Seagate Exos X16 16 TB SATA drives since September 2020. Statistically nothing should have happened. In practice, though, I have lost 6 of the original 8 drives and at least 3 of the replacement drives. Before that the last drive lost was an IBM DFRS 2.1 GB SCSI disk back in 1999.
As to mirrors: ZFS supports mirrors with more than 2 devices. So with a 3-way mirror one can have the redundancy of RAIDZ2 without its performance penalty. Yes, the net capacity is not great. But in a professional context that is negligible. Esp. if one looks at the price of high-end storage systems. But I digress …
You put words in my mouth. I never said there is NO risk. I am aware, I promise you. Parity is for uptime, not perfection. Re-read what I wrote, I believe it quite clear.
The most accurate answer is there is no answer. It is all a matter of personal tolerance to risk, and willingness to restore from backup.
You’ve probably come across the old “RAID5 is dead” article. Re-read and do the math.
Note that actual URE rate is unknown: Spec sheets always list “lower than…”.
Classical RAID would treat the entire disk, regardless of occupancy; and would stop upon encountering an URE. ZFS only resilvers what’s actually used, and reports the loss of a file upon encountering an URE, so it’s “less bad” than RAID—but still non zero.
The gist is that once you’ve lost all redundancy, you’re at risk, be that an URE (for which we can calculate a higher limit for the risk), or a further drive failure. The latter is a more complex calculation, and is skewed by the fact that failure could actually be triggered by the stress from a resilver—so here ZFS is worse than classical RAID.
Zfs MAY report the loss of a file on a URE. Some drives though, will hang, ruining that, depending on the error. Which is why I get the feature set of enterprise drives of course.
Definitely the answer is risk tolerance. For some reason, some are thinking I have a high risk tolerance, clearly they have not read much on what I have written on this site. Heck, I’m still on Eel.
That old article with a sorry excuse for math is one of the dumbest articles ever written. Unfortunately also popular.
