Short story time: I’m in the process of backup/destroy/recreate my main storage pool.
I’ve reached the point where I’m creating a 12 wide RaidZ2 vdev of 16TB drives (14.55 TiB). I’m also including a 3 way mirror svdev (2TB NVMe drives).
When the pool is created, it is reporting a usable storage amount of 132 TiB, which I believe matches the output of the Truenas ZFS calculator when using 128kB record size. However I’m using a 2MiB record size, at which point the calculator reports there should be 144TiB usable.
I don’t remember where I read this, and I forget if it applies to TrueNAS specifically or ZFS in general, but the usable storage calculation for RAIDZ is always based on 128KB recordsize, no matter what you might actually be using. You’ll just have to assume your usable capacity is greater than what is being reported, since you know you’re using a recordsize of 2MB, even though the calculation and reporting assumes 128KB.
So this would mean that the display widgets, and Truenas reporting, are always wrong in relation to RaidZ storage, because of record sizes, not considering svdev space, and raidz expansion miscalculations.
Fantastic.
Still doesn’t mean mirrors are better though, but the gap is closing.
I hate how ZFS gaslights you. How is one supposed to know how much capacity they truly have?
176TiB? 144TiB? 132TiB?
My guess is that you have 144TiB usable capacity, but the TrueNAS GUI (and zfs command) wrongly report 132TiB, due to the calcuation being fixed on a 128K recordsize, even though your datasets are using a 2M recordsize. (Is 176TiB the “raw” capacity? A useless metric for the user, since we only care about how much data we can store with recoverable redundancy.)
About the calculation being fixed at 128K, I’ll try to find who explained this in the past. I think I read it on one of Matt Ahren’s posts or in the FreeBSD Magazine.
Cheers, I agree with you in the main. Time to go down the rabbit hole and see if there are alternatives that make sense with record size etc but I suspect that it’s simply the case of “Don’t worry about it, just add more vdevs at 80% capacity”.
