It doesn’t do anything when checking space over SMB, File Explorer, or du, and you’ll also still get invalid “disk usage” of files that report smaller if they were saved prior to the expansion. This can give the wrong impression about compressible files, since they are expected to be smaller to some degree.
A comment which I do not quite understand. (This may have to do with the fact that I am not a genius.)
It seems to me that:
The level of risk is inversely proportionate to your ability to restore the data from backup.
1 Like
You are of course right.
Wow. This topic has been a mind blowing read. For the decade plus that the ZFS zealots have been talking about how it’s the best filesystem in the world, we’re literally in a spot where if you use a supported system feature, it gives you inaccurate information about total available space, and we’re considering this to be “fine.” I mean other than whether or not your storage is accessible, I’d say available space / capacity is literally the most important thing a storage system can tell you…
3 Likes
You’re not wrong on the user side, but with CoW and transparent compression “space”, “used”, and “available” have always been complicated matters with ZFS (which is still arguably the most advanced file system, but not necessarily the most suitable for all cases).
I woke up today feeling like everything I ever thought was wrong.
I can’t believe that in an enterprise grade tool like TrueNAS, when you go through VDEV expansion, there’s not a little yellow warning saying, “Hey by the way, yes you can expand your storage, BUT, if you do that, it’s a one-way street, and you’ll never be able to get an accurate view of how full your VDEV is!”
How can this not be like one of the highest freaking priorities? Or… remove the feature. It’s not working right. Disable it until there’s a workaround.
3 Likes
It’s a reporting issue, not a functioning issue.
You’re not affected if you do not use raidz expansion… and enterprise users don’t use this: Backup-Destroy-Restore works for them.
2 Likes
Well of course they don’t use it if it has detrimental side effects. If it didn’t, why would an enterprise go through backup / destroy / restore, when expansion would work just fine, assuming a backup already in place?
Enterprise users don’t do expansion because it’s not useful to them. That’s not what enterprise does. Even backup-destroy-restore would only happen when they made a mistake planning their storage.
An enterprise that has a raidz2/3 use case would build N vdevs that are X wide each, and just add more vdevs as they go.
Needing to change the width means someone made a mistake during planning.
5 Likes
If ZFS was a bank…
Bank of ZFS: Your balance is $400.
You: I’m depositing this $200.
Bank of ZFS: Thank you. Your balance is $450.
4 Likes
But also:
ZFS Bank: Your spending limit [available space] is $400.
User: I’m withdrawing [saving new data] $100.
ZFS Bank: With your zstd5 plan, your spending limit is now $330.
3 Likes
For users who have used Zfs expansion and experiencing this problem, is it best to just use a platform like backblaze to backup and recreate the pool? Is it possible for a fix to work for previous expansions or is this data currently lost and a future fix would not be able to fix it retroactively. I kind of understand the cause and am wondering if it’s worth trying to fix it, or to just wait and see what comes from zfs issue 17784
There is no data lost, it’s just the UI partially (the % of used space is accurate, the actual TB numbers aren’t) misreporting the amount of space. You could delete and recreate the pool using the cloud provider of your choice… but at that point why are you using zfs expansion and not just recreating the pool and restoring from backup anyways?
I already have done the expansion and spent the last few days trying to figure out why the space wasn’t adding up before landing on this issue. There’s no real issue other than size reporting being out, but I do transfer data between pools quite often and it was annoying that the space taken by the same dataset across pools was unreliable. I have slowly expanded from a zfs2 x4 to a x6 and am planning on settling at x7. I’ll probably just recreate the pool from a cloud backup when I get that seventh drive if this issue is still outstanding.
So how many people have to go through this headache before someone puts a proper warning in the UI prior to utilizing Raidz expansion? I saw the warning about rebalancing data/files afterwards. That’s no biggie and I was planning on doing it after both disk expansions. There should be a warning about this, as well. Was there one? I guess I could have missed it.
Yes, Raidz expansion works. But some people may value accurate storage monitoring more and will have to decide if they want to destroy, rebuild and recover their data just to get that back. I now face that dilemma. Luckily I’m in the process of building another server. However, I was planning on using Raidz expansion in that server, as well. So now I’m just going to bite the bullet and buy all the disks at once, I guess. I know I could do striped mirrors, but I don’t want that big a hit on my total capacity, or the risk that comes along with it.
Sorry for the rant. Thank you for everyone who’s contributed information to this thread.
6 Likes