Incorrect usable capacity after RAIDZ2 expansion

TrueNAS General
, ,
1

Im quiet new to this but I originally had 4 drive with RAIDZ2, then expanded and added 2 more drives, adding the first one was fine, second one had some problems but I don’t recall exactly what happened, I did try to fix it but no luck.

With my current set up of 6x 6TB drives on RAIDZ2, i should have ~20TB storage but its only showing ~15. Any help would be appreciated, some outputs below.

# zpool status -v storage0

  pool: storage0
 state: ONLINE
  scan: scrub repaired 0B in 06:04:47 with 0 errors on Sun Jun 15 06:04:48 2025
expand: expanded raidz2-0 copied 16.9T in 1 days 04:23:23, on Sun May  4 00:08:43 2025
config:

        NAME                                      STATE     READ WRITE CKSUM
        storage0                                  ONLINE       0     0     0
          raidz2-0                                ONLINE       0     0     0
            a8aef3a0-5dad-4223-9168-b54ff28267db  ONLINE       0     0     0
            18f0a6ad-4337-4358-b3eb-15e5e73d3d09  ONLINE       0     0     0
            114f8f54-3e2d-4bac-85a5-f4d4fb1b570b  ONLINE       0     0     0
            98a4bddc-ba94-4fdb-bcb0-abcee722cc29  ONLINE       0     0     0
            f8404f8b-cac5-4254-8922-e4a20bdb99c1  ONLINE       0     0     0
            cd310abe-d110-46e4-b983-a653069975ed  ONLINE       0     0     0

errors: No known data errors

# zpool list -v storage0

NAME                                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
storage0                                  32.7T  18.2T  14.6T        -         -     0%    55%  1.00x    ONLINE  /mnt
  raidz2-0                                32.7T  18.2T  14.6T        -         -     0%  55.4%      -    ONLINE
    a8aef3a0-5dad-4223-9168-b54ff28267db  5.46T      -      -        -         -      -      -      -    ONLINE
    18f0a6ad-4337-4358-b3eb-15e5e73d3d09  5.46T      -      -        -         -      -      -      -    ONLINE
    114f8f54-3e2d-4bac-85a5-f4d4fb1b570b  5.46T      -      -        -         -      -      -      -    ONLINE
    98a4bddc-ba94-4fdb-bcb0-abcee722cc29  5.46T      -      -        -         -      -      -      -    ONLINE
    f8404f8b-cac5-4254-8922-e4a20bdb99c1  5.46T      -      -        -         -      -      -      -    ONLINE
    cd310abe-d110-46e4-b983-a653069975ed  5.46T      -      -        -         -      -      -      -    ONLINE
2

Search for RAID-Zx-expansion posts and see if any of those explain the issue. Did you do any rebalalancing of your data or just do the two drive expansion?

Your numbers show more space than a Raid-Z2, 5 drive set up and, maybe correct values, for a 6 wide Raid-Z2 using the expansion feature.

3

  1. 6x 6TB RAIDZ2 should give you c. 4x 6TB useable storage and 24TB useable storage is c. 21.8TiB. (Disks are sold in TB = 10^12, but sizes in TrueNAS are TiB which is 2^40. There is about 9% difference.)

  2. zpool list shows blocks not useable space - so it shows 6x 6TB = c. 32.7TiB and that is what you see.

  3. There is a bug in RAIDZ expansion which mis-reports available space - but the space is still there. (The useable available space is always an estimate because it depends on compression and record size and parity ratio, but zfs continues to use the original parity ratio of 2 data + 2 parity in its calculations, even though new data is written 4 data + 2 parity.)

  4. RAIDZ expansion moves existing records to the new drive in order to make all drives have the same free space. Existing data records were written on the 4x RAIDZ2 as 2 data + 2 parity, and after expansion they remain that way even if they are now spread across the 6 drives. New data is written as 4 data + 2 parity. If you use a rebalancing script to rewrite your existing data (and to get the space back you will need to delete all your snapshots) then 2 records of 2 data + 2 parity = 8 blocks will get rewritten as 1 record of 4 data + 2 parity = 6 blocks and you will get c. 25% of your existing used blocks back. Since you use 18.2TiB, then if these are all base data (rather than snapshot data) you should get c. 4.5TiB back after rebalancing.

2 Likes