I’m currently planning out my next truenas server. I understand you can now increase the capacity in a zraid by replacing all drives with larger drives. Can you add additional drives to a zraid? Say it started as a 3 drive zraid1 with a single vdev, can I add a fourth to that vdev and realize that capacity (assuming it’s capacity is equal to or greater than the existing drives)? In my current truenas system (mini xl+) I went with 2 raidz1 4 disk vdevs in a single pool as I started with just 4 disks.
If the above is possible, and for my home setup I’m building, single disk parity is good enough, would there be a benefit to not just building my next system with a single vdev in a raidz1 (max of 15 drives)? I value total usable capacity over redundancy but won’t be starting with 15 drives, but also value performance over capacity but still keeping capacity in mind.
Just to restate, if I lost 100% of the data that will be on this system, it would suck but in no way be devastating and it’s all replaceable, so a single disk parity is ok if it can be expanded.
Not yet. I understand that functionality has landed in upstream OpenZFS, and should show up in the next major release of SCALE, which I presume will be 24.10 in October–though it’ll be CLI-only at that stage. But for now, no.
That is considered WAY too wide for safety, and even for performance and space efficiency.
Recommended limits are 10 to 12 wide for raidz2 or raidz3 (more like: 10-wide raidz2, 12-wide raidz3).
Thanks. Again, I’m not worried about the safety factor. I don’t want to be completely reckless and just stripe it but losing this data wouldn’t be a huge deal.
Though the performance and space efficiency topics are where I’d like to dig in a little. The little bit of digging I’ve done suggests that a 12x raidz has better performance than a 6x raidz (source). I do understand that zfs raidz loses more space than a typical raid does to parity and overhead, does that get worse the more drives you add?
Obviously, the wider the raidz vdev, the larger the smallest possible block, and any file smaller than that will require padding.
Incidentally, raidz expansion, when it eventually lands, willl come without reflowing existing data to the new width. So when you expand a 4-wide vdev to 5-wide, all exiting data remains striped as in 4-wide; only new data is striped 5-wide. As you repeat the operation, there is increasingly wasted space as a result and an incrementally expanded vdev will have less usable capacity than a native 15-wide. https://louwrentius.com/zfs-raidz-expansion-is-awesome-but-has-a-small-caveat.html
Interesting, great information, thank you. So expansion actually sounds like a bad idea if I care about capacity Hmm, maybe I should just get some cheap small drives to get me to my total number. Then I’m not expanding the width, just the capacity…but then I’d be limited to the smallest drive until all on the vdev are upgraded. So many decisions
Vdev expansion, with all caveats, may possibly be useful for limited adjustments.
As Louwrentius’ calculations show, starting with a narrow vdev and repeatedly expanding by one drive every time the vdev is 70-80% full is NOT a good strategy: One may easily lose one disk worth of usable space.
He does point out something I was thinking about, I could just create a script to rewrite the data. I think something as simple as crawling the directories, for each file (or maybe I could do it at the directory/sub-directory level) move the file to a temporary name (file.txt → file.txt.tmp), a move should be quick, then do a copy
back to the original name and remove the temp file (add some safe guards like checking to make sure the file hash matches before removing, etc). Though when we’re talking about hundreds of TBs of data, that’s a lot of data to rewrite.
Not at directory level, due to Copy-On-Write; one has to do full snapshots—and then delete the original. And not-so-simple if you want the script to be fail-safe…
A rebalancing script already exists which does just that. But, obviously, it needs free space to work (and no dedup!). So the idea to expand one drive at a time to keep costs low is not going to work well. ZFS wants you to frontload costs.
Hmm, maybe I should just get some cheap small drives to get me to my total number. Then I’m not expanding the width, just the capacity…but then I’d be limited to the smallest drive until all on the vdev are upgraded. So many decisions 
