I had a RaidZ2 pool of 4 x 8TB drives and decided to add another 8TB drive using the extend feature. Took a couple of days but didn’t report any errors.
Seems like the new numbers don’t add up…
I now have
Usable Capacity of 17.49TiB
Used: 12.46TiB
Available: 5.03TiB
added context
admin@truenas[~]$ sudo zpool list -v data
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
data 36.4T 25.7T 10.6T - - 0% 70% 1.00x ONLINE /mnt
raidz2-0 36.4T 25.7T 10.6T - - 0% 70.7% - ONLINE
admin@truenas[~]$ sudo zfs list -o space data
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
data 5.03T 12.5T 0B 209K 0B 12.5T
A TiB is 2^40, whilst a TB is 10^12 which is c. 10% less.
So your 8TB drives are c. 7.2TiB, so a 5x 8TB RAIDZ2 will be 5x 7.2TiB = 36TiB in size which is what the zpool list is reporting. Presumably your 4x RAIDZ2 was full-ish - 4x 7.2TiB is 28.8TiB, so alloc. of 25.7TiB seems about right for your before usage too. These numbers are calculated from the actual blocks used and free in the pool and are likely to be pretty accurate.
The zfs list is calculated somewhat differently based on actual data, and the calculations are not anywhere as easy because of lots of factors like snapshots and block cloning and overheads and the variable way that records are allocated (they can be the width of the RAIDZ pool, but they can be smaller too - in a 4x RAIDZ2 they can be 3 or 4 wide) - and it gets more complicated when you expand a pool with an extra disk because of the way that the expansion works. But there could also be a ZFS bug too after expansion - it is pretty new functionality.
That’s disappointing, I probably should have done a bit more research prior to expanding. Just assumed everything was sorted once officially released.
So it’s possible that the free space reported by zpool list is accurate and the 5TB discrepancy between that and zfs list is just differences in calculation methods? Seems like this variation between the two could lead to some weird unforeseen errors as the pool fills up
Yes -but whilst there is an iX ticket for it, it is actually a limitation / bug of ZFS rather than TrueNAS.
I can understand the general difficulties of calculating the actual space used vs. space available in user space terms because of different record sizes for different datasets, snapshots, block cloning etc., and RAIDZ expansion certainly makes this even more difficult, but given that RAIDZ expansion moves existing data but doesn’t widen it, then in essence whatever number of pool blocks were used before expansion doesn’t change - they may be moved about in order to put some of them on the new drive, however the number of blocks is the same. So by adding another drive you are adding a certain number of pool blocks all of which are free, so for RAIDZx of width w you should be able to estimate additional free space as the proportion (w-x)/w of the additional disk space added. (So for example, if you have a 4x RAIDZ2 and you add another disk to make a 5xRAIDZ2, the additional disk space ends up being distributed evenly across the 5 drives by the expansion process, and then when you write 3 blocks + 2 parity, the useable space is this 3/5 of the size of the new disk (assuming that the pool disks are all the same size).
However, users should be reassured that the additional space is there even if it is not reported by the datasets - and should look to the pool size instead for more reassurance.
appreciate the detailed response, what you said does make sense.