Adding drives to a raidz3...any cons?

TrueNAS General
, ,
1

So I have read about the pool not updating the parity calculation but couldn’t find anything that really clarified the amount of space lost due to that.

I currently have 5 24tb drives in a raidz3. My plan is to go to 12 drives. I know it’s a wide vdev but it suits my layout. What are the cons of adding drive by drive as my storage needs expand? Or should I just bang out all 12 drives from the start?

Thanks for any and all advice!

2

12x 24TB RAIDZ3 seems OK to me but it is at the recommended upper limit.

There are three issues with RAIDZ expansion:

  1. Speed - ElectricEel expansions are slow - there is / will be a speed fix in Fangtooth.

  2. Parity - existing data retains the existing parity when you expand. So, for example, expanding from 5x to 6x RAIDZ3 you will have 2x data blocks and 3x parity blocks in each full record in your existing data, and 3x data blocks and 3x parity blocks in any new records. If you rewrite the data then for each 3 full records of existing data (6x data blocks and 9x parity blocks) these will be rewritten as 2 records (6x data blocks and 6x parity blocks) going from 15x blocks to 12 blocks and saving 20% of the used space. When you have 11x RAIDZ3 and are going to 12x RAIDZ3, the savings for 9 full blocks are from (9x(8+3) = 99 to 8x (9+3) = 96 i.e. c. 3%.

  3. Free space reporting - there is another bug which results in ZFS under-reporting the free space (so it looks like the expansion hasn’t worked properly when in reality it has). I have no idea whether this has been fixed in Fangtooth. However this is only a reporting issue, the extra space is definitely there.

There are Pros and Cons for both adding one by one or buying them all now - so I think it is a personal decision. But I would personally buy one by one because:

  1. I may not ever need all that space;
  2. Drives may (slowly) get cheaper;
  3. I prefer to keep the money in my bank account than have it as spinning disks I don’t need;
  4. Buying as I need the space means I get better warranty;
  5. Having drives of varying ages and batches reduces the risk of 4 drives failing at the same time.
1 Like
3

No space is lost to wrong calculations: Storage space is there, just not acurately reported.
Raidz expansion, however, does lose on space because old data is reflown without being rewritten to the new stripe width. The solution is to rewrite data (rebalancing script), which does not preserve snapshots.

1 Like
4

Does this rebalancing script avoid block cloning?

5

What a great answer! So what happens if I lose power during this process? Do I lose the whole pool?

6

Upon reboot, ZFS takes back where it was halted. (Normally…)

1 Like
7

Excellent…thank you!