Understanding Lower Than Expected Capacity in Expanded RAIDz2 Pool

TrueNAS General
1

Hi everyone,

I’m trying to understand why my storage pool’s usable capacity is significantly lower than expected on my TrueNAS SCALE ElectricEel 24.10 server.

My setup:

  • 8x 16TB drives in RAIDz2
  • Started with 4 drives in RAIDz2, expanded one by one to current 8 drives
  • To rebalance after expanding, I used zfs send | zfs receive where possible
  • Used zfs-inplace-rebalancing.sh script for remaining files

Current reported capacity: 56.26 TiB

Theoretical calculation:

  • Single drive capacity: 14.55 TiB
  • 8 drives × 14.55 TiB = 116.4 TiB total
  • RAIDz2 with 8 drives should give 6/8 of total space
  • Expected capacity: 87.3 TiB

That’s a difference of about 31 TiB between expected and actual capacity. I’ve done my best to rebalance the data using both zfs send | receive and the inplace rebalancing script.

Questions:

  1. Is this normal for an expanded RAIDz2 pool?
  2. What could explain such a significant difference in usable space?
  3. Are there any commands I can run to get more insight into how the space is being allocated?

Any help understanding this would be greatly appreciated!

2

There’s already a pretty long threat discussing this 24.10 RC2 Raidz expansion caused miscalculated available storage

2 Likes
3

@LarsR thanks!

To save others the time from reading 136 replies, here’s a clearer summary:

Key Technical Details:

  • Issue tracked at: ixsystems Jira NAS-131728 (I am not allowed to post links…)
  • Matthew Ahrens explained in gh:openzfs/zfs/pull/12225#issuecomment-1057679062 (I am not allowed to post links…) that:
    • Old blocks must maintain their original deflation ratio when freed to prevent over-accounting
    • A solution using “time-dependent deflation ratios” is possible but wasn’t implemented due to time constraints
    • This could be fixed in the future with “RAIDZ Expansion improved space accounting”

Can you use the “hidden” space?

  • Yes! The space is fully usable despite being underreported
  • The system will let you write more data than the reported available space
  • Only the reporting is affected, not actual storage capacity
  • The pool will still function normally and safely store data
  • You can write data up to the true physical capacity of the expanded pool

Credits to @Stux for seemingly doing most of the research in the previous topic. Please let me know whether any of my conclusions are wrong.

4 Likes