Usable capacity on expansion question

TrueNAS General
, ,
1

Hi, I added a drive to a pool, and I feel like I should have more capacity than I currently have.

I had four 10TB drives, so 9.1TiB drives, set up in RAIDZ2. I added a 12.73 TiB drive to expand the pool, and TrueNAS is reporting that I have 21.9 TiB usable capacity. I understand that I am not going to get the full space of the larger drive, but I expected to have around 27 TiB of space?

Am I just misunderstanding how much space I should have?

2

Adding the drive takes a loooooong time (multiple days for larger drives/pools), which the gui sadly doesn’t report.
Go to shell and check with

sudo zpool status -v

if the expansion is already finished or if its still going.

3

According to the output, it appears to be finished.

  pool: HDDs
 state: ONLINE
  scan: scrub repaired 0B in 1 days 22:43:28 with 0 errors on Wed May  7 03:41:13 2025
expand: expanded raidz2-0 copied 24.8T in 2 days 16:30:41, on Sun Apr 13 09:29:16 2025
config:

        NAME                                      STATE     READ WRITE CKSUM
        HDDs                                      ONLINE       0     0     0
          raidz2-0                                ONLINE       0     0     0
            c009e050-a902-4ceb-abe5-b862431725c2  ONLINE       0     0     0
            261b738b-3e31-439c-8aa6-08e0a00b3643  ONLINE       0     0     0
            b9625277-d80b-4b8b-8c1c-7baa630cc105  ONLINE       0     0     0
            210ebdc8-069b-48ba-8b7c-8fb51718d0a6  ONLINE       0     0     0
            3a974030-41c9-45a1-a491-78c6b17d3604  ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:08 with 0 errors on Mon May  5 03:45:09 2025
config:

        NAME         STATE     READ WRITE CKSUM
        boot-pool    ONLINE       0     0     0
          nvme0n1p3  ONLINE       0     0     0

errors: No known data errors
4

I think your case is related to this drama with expansion.
Long story short – you probably have your space. But reporting of free/total space is now screwed.

1 Like
5

Do a sudo zpool list if you want to confirm that the space has been added.

The space is mis-reported in TrueNAS for 3 reasons:

  1. TrueNAS UI uses zfs stats rather than zpool stats when reporting pool usage.

  2. During expansion, the original files are all moved based on the original data:redundancy ratio i.e. for a 4x Z2, 2 data blocks and 2 redundancy blocks - whilst new files will be written 3 data blocks and 2 redundancy blocks. So an old file with 24KB of data will use 3x 4 blocks = 12 blocks, whilst the same file written anew will use 2x 5 blocks = 10 blocks which is c. 16% less space.

  3. There is a bug / feature in the ZFS expansion whereby (AFAIK) it reports free space based on the old redundancy ratio rather than the new redundancy ratio i.e. it reports free space as 16% too low.

P.S. Drives are sold in TB = 10^12, but Linux/ZFS/TrueNAS report them in TiB = 2^40 - and 1TB = c. 0.9TiB.

6

That is really interesting. So according to zpool list, I do have 19.8T free instead of the UI reporting that I only have 9.4TiB free.

Can I just keep using things and it will all be fine? What happens if I use over the 9.4TiB that is being reported as free?

7

The 9.4TiB is a misreported value. As you start to fill the space, apparently the amount it is wrong will reduce.

P.S. I think that the 19.8TiB is blocks on all drives, and the 9.4TiB is an estimate of useable space i.e. taking into account redundancy.

1 Like
8

You can keep using the pool. As it fills in, reported available space will decrease. After filling in 10 TiB in these reported 9.4 TiB you’ll still have some (wrong) free space…
Space is there, it will just never be reported correctly.

1 Like