RAIDZ expansion vs creating the pool with all drives from the start

TrueNAS General
1

Hi everyone,

I’m planning a new TrueNAS SCALE setup, but I won’t be able to buy all my drives at once.

My final goal is to have a single RAIDZ2 vdev with 9 × 8 TB drives. However, initially I’ll only have 5 or 6 drives available.

I’m wondering whether it makes sense to:

  • create the RAIDZ2 pool now with 5 or 6 drives,

  • start using it,

  • and then use the RAIDZ expansion feature in the future to grow the same vdev to 9 drives as I buy the remaining disks.

My main concern is long-term performance and efficiency. I’ve read that RAIDZ expansion may not distribute data as optimally as a pool that was created with all drives from the beginning, but I’m not sure how significant that difference is in practice.

Would you recommend this approach, or would it be better to wait until I have all 9 drives and create the pool from scratch?

I’m interested in both real-world performance and any long-term drawbacks (I/O performance, resilvering, fragmentation, capacity distribution, etc.).

Thanks!

2

You can search for the RAID-Zx-expansion tag and read those forum posts. It should cover how it works, the problems with space reporting and there should be a white paper or two explaining how the data is distributed before and after using the feature.

I would be more concerned about if this data is going to have a good backup source and how you would get more space in the future, if necessary. You’re stuck with 9 drive replacements and resilvers of larger drives or migrating the data to a new pool or vdev.
Raid-Z(1,2,3) setups are not a flexible as Mirrors. If you don’t like it, you are stuck with destroying the pool and recreating it to change the setup

3 Likes
3

Ideally, you’d want to create a raidz# at its final geometry.

May I ask why 9-wide raidz2 at 8 TB? 9 is a strange number, as few motherboards provide more than 8 SATA ports (and, increasingly, much less than that). And 8 TB feels small to me; larger drives may have a better cost per TB, and less drives means less noise and less power.

2 Likes
4

I have a jonsbo n6 that has 9 hdd slots. And im findin new/not so much used 8tb discks on vinted for dirt cheap.

5

Thanks for the explanation—partial as case and opportunity makes two sides of the triangle: What provides the 9 SATA ports? (Just wanting to check if you have the rare motherboard or a solid plan for a HBA.)

6

Hba with 8 ports mobo with 1 sata :wink:

2 Likes
7

Check the model numbers of the hard drives to make sure they are all CMR types. No SMR type drives with ZFS

SMR vs CMR ServeTheHome

TrueNAS Docs - WD Red SMR Drive Compatibility with ZFS

8

RaidZ expansion works, but it’s IMO a pretty ugly hack (at least compared to some of the rest of ZFS internals) since the initial RaidZ design was never really intended for it, so it shuffles data around on layers that initially ought not to do that.

It currently:

  • Redistributes all data once during expansion (can be quite slow)
  • Significantly skews the space reporting (especially without a second rewrite)
  • Requires you to rewrite existing data a second time, if you want the better storage efficiency from the new parity ratio for it (this doubles used space unless you delete all snapshots before)
  • Still has incorrect absolute used/free space reporting even on a fully rewritten pool (this gets proportionally worse the more you expand)

It’s usable (despite the quirks) and it’s nice for users that really don’t have another option, but if you can in any way plan around not needing it do that instead.

1 Like