well if you insist 
The actual consumed space went from 2.8G ā 2.1G, or in other words a 33% reduction in space when doing a rewrite of 3wZ1 to 5wZ1 ā¦ I know itās not to the same extent, but my experiment was admittedly a lot smaller in scope.
Have a little more data, on tests run on a little more data.
3wZ1, data ingested.
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
expanding 47.5G 41.2G 6.34G - - 0% 86% 1.00x ONLINE /mnt
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD VOLSIZE
expanding 3.24G 27.4G 0B 128K 0B 27.4G -
expanding/expandme 3.24G 27.4G 0B 27.4G 0B 0B -
Expanding to 4wZ1 added ~16G to FREE in zpool list but only ~10G to AVAIL in zfs list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
expanding 63.5G 41.2G 22.3G - - 0% 64% 1.00x ONLINE /mnt
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD VOLSIZE
expanding 13.6G 27.4G 0B 128K 0B 27.4G -
expanding/expandme 13.6G 27.4G 0B 27.4G 0B 0B -
Same for expanding again to 5wZ1:
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
expanding 79.5G 41.2G 38.3G - - 0% 51% 1.00x ONLINE /mnt
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD VOLSIZE
expanding 23.9G 27.4G 0B 128K 0B 27.4G -
expanding/expandme 23.9G 27.4G 0B 27.4G 0B 0B -
After rebalancing we gain a decent chunk of both FREE and AVAIL space.
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
expanding 79.5G 34.3G 45.2G - - 0% 43% 1.00x ONLINE /mnt
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD VOLSIZE
expanding 28.5G 22.8G 0B 128K 0B 22.8G -
expanding/expandme 28.5G 22.8G 0B 22.8G 0B 0B -
1 Like
I already used the zfs-inplace-rebalance script to rewrite all of my files, and it did recover about 2TiB of space. I know itās intended for rebalancing across vdevs, but it just copies everything and deletes the original, which is effectively the same thing. I should be getting accurate accounting from zfs list already.
It if itās triggering block cloningā¦
I would use dataset replication personally.
I agree. Itās not like you filled your pool to 80% capacity before you expanded your RAIDZ2. It was only about 25% full, right?
So even if you did not do any in-place rebalancing, you should still expect to see the total usable storage capacity much higher than 47 TiB.
Well, keeping in mind this is a brand new feature of zfs, one that I wouldnāt dare use at this point (glad others are testing!). Not too surprising given that.
1 Like
Out of curiosity, do you happen to have snapshots on your datasets? If you use that script, you actually double the used space, because your snapshot holds all the old data as well.
If would recommend to copy your whole datasets with zfs send/recv, delete the old dataset and rename the new one to the same name the old one had.
Thank you for the suggestion, but with this serverās role for hosting media, I donāt keep snapshots. I did have some when I used replication instead of zfs send | receive, but those were cleared before I ran the script.
I do not wish for this to come off as rude, but tone is difficult to convey via text. If you look at the 2 pictures in my OP, you will note that the āusedā space value decreases after the script was ran, yet in both, they still show an incorrect value for total capacity.
I expanded a 6 wide RaidZ2 vdev to 10 wide using 16tb drives. Iāve found the same problem, zpool list -v shows available space increasing after running the inplace balancing script (I watched the free space increase whilst it was taking place, and the capacity used decrease but the dashboard widget never changed (96 tb capacity reported in widget, zpool reported 117 tb capacity).
Watching with interest, if there is any information I can offer happy to help.
This confirms there is a bug somewhereā¦ please report it. Did the problems appear after the 1st drive?
Will do, but I did this expansion when in Beta (literally took a week to add the four disks and then about 60 hours to run the ZFS rebalance script on around 24 Tb of data) and it finished the weekend before RC1 was released. One complication: I also added a metadata vdev prior to running the rebalance script so something else to affect the trouble shooting.
Apologies for double post: bug report filed as requested
1 Like
If my memory serves me correctly, the ZFS Scrub is mandatory after each expansion. This is due because the re-striping does not confirm checksums, in order to speed up the process.
So, start a scrub of the pool and donāt reboot until it is finished.
To be clear, I am not sure this is the source of the problem. But, as I said, I remember a ZFS Scrub being automatically started after a RAID-Zx column expansion.
2 Likes
I finally got word back from my Jira ticket. Apparently the usable space looks weird not because of any parity ratio mismatch, or because a drive wasnāt inserted properly, but because zfs/truenas is deliberately using pre-expansion values to calculate available storage space. Space used is underrepresented proportionally, so used capacity % is accurate - as seen here.
Lol Im so dumb I donāt even understand the solution 
1 Like
Think of it like this:
You go into a store to purchase a USB flash drive. Thereās no capacity printed on the box. You ask the sales rep āHow much data can this hold?ā They just shrug their shoulders and say āNot sure lol. Itās a lot though. Youāll find out when you get home. Or maybe you wonāt. Letās just say 512 GiB. Okay, maybe more like 700 GiB, but it depends how you plan to use it.ā
I think Iām just as confused as you.
3 Likes
in shortā¦ this isnāt an error with zfs, itās just the UI being funky with how it reports stuff.
I mean, thatās kind of impacted by the inline compression as well. You might store 100G of ālogical dataā on there, but if it compacts 2:1, you might have ā100G used: 462G freeā and wonder where your extra 50G of space came from.
@HoneyBadger about to market and sell the first consumer USB sticks pre-formatted with ZFS pools? 
Badger Stickā¢:
Pre-formatted with a ZFS pool, with LZ4 compression.
Badger Stickā¢ Plus:
Pre-formatted with a ZFS pool, with ZSTD-19 compression, and deduplication enabled.[1]
āAt Badger Labs, our methodical research shows that our āPlusā series can store an extra 50% data per stick! More data, same great Badger technology!ā ā©ļø
4 Likes
I think I understand what youāre saying, but is this the intended behavior? If I download whatās reported as a 10GB file, will truenas report it as smaller than that? And consequently the same behavior for drives, I purchase a 5TB drive but once expanded the pool only reports some fraction of that space available? This behavior seems misleading and counterintuitive to me, although I know there is some complex compression and zfs magic going on behind the curtains.