Pool replace/expand and Record Size Change with Decreased Capacity

I recently replaced an 8x 1TB vdev with 8x 8TB. The dataset assigned to this vdev was instantiated with 128k record sizes, but I changed it before replacing the drives to 1M. When using the TrueNAS Raid Calculator online, my 8x 8TB with a record size of 1M should give me 2 extra TiB than I currently show. Is there anything I can do about this without starting completely from scratch? Even then, if I do start from scratch, is there a way to define record size when creating a VDEV from the GUI? I cannot seem to find one and it seems to be tied to the dataset.

Based on the this post, is this just a GUI estimate for capacity based on the default 128KiB?

My current expanded pool

Expected capacity with 8x 8TB (7.27TiB) and the default 128KiB record size

Expected capacity with 8x 8TB (7.27TiB) and the 1MiB record size

OS Version:TrueNAS-SCALE-
Product:Standard PC (i440FX + PIIX, 1996)
Model:QEMU Virtual CPU version 2.5+
Memory:6 GiB

I wonder if you need to turn on zstd compression, followed by running a rebalancing script? Is the data small files, databases, virtual machines, etc. or images, movies, and the like? Large record sizes are only beneficial for large files.

So could you verify in the Datasets GUI that the recordsize is what you expect it to be and that compression has been turned on? Thereafter, I’d run the rebalancing script, see here.

Most of the storage is linux isos and similar media, but it is littered with small metadata files as well. Additionally, it is used for Proxmox Backup Server, so larger files there as well.

It looks like the new record size is effective for new files based on the histogram shown in the linked article, though it would probably be best to run that rebalancing script too.

Thanks for the pointer!

Block Size Histogram

  block   psize                lsize                asize
   size   Count   Size   Cum.  Count   Size   Cum.  Count   Size   Cum.
    512:   123K  61.6M  61.6M  3.82K  1.91M  1.91M      0      0      0
     1K:  23.4K  24.3M  85.9M  23.8K  24.5M  26.4M      0      0      0
     2K:  45.6K   155M   240M  4.34K  11.7M  38.2M      0      0      0
     4K:   185K   744M   985M  5.76K  28.8M  67.0M      0      0      0
     8K:   171K  1.94G  2.90G  3.22K  34.8M   102M  30.1K   361M   361M
    16K:  42.1K   926M  3.80G   173K  2.72G  2.82G   371K  8.69G  9.04G
    32K:   102K  4.73G  8.53G   296K  9.31G  12.1G   228K  10.7G  19.8G
    64K:   349K  31.4G  39.9G  5.38K   507M  12.6G   209K  20.6G  40.4G
   128K:  37.3M  4.66T  4.70T  37.8M  4.72T  4.73T  37.4M  6.58T  6.61T
   256K:  15.0K  5.54G  4.70T  5.91K  2.17G  4.73T  13.2K  4.85G  6.62T
   512K:  22.7K  16.3G  4.72T  11.1K  8.36G  4.74T  20.5K  15.3G  6.63T
     1M:   190K   190G  4.90T   224K   224G  4.96T   200K   265G  6.89T
     2M:      0      0  4.90T      0      0  4.96T      0      0  6.89T
     4M:      0      0  4.90T      0      0  4.96T      0      0  6.89T
     8M:      0      0  4.90T      0      0  4.96T      0      0  6.89T
    16M:      0      0  4.90T      0      0  4.96T      0      0  6.89T
1 Like

You can also consider making different datasets for the different data sizes for more optimization instead of placing everything in a single sized data ‘bucket’.

1 Like

Most of the metadata files are intermingled with the larger files, but creating additional unique datasets for other uses in the pool is probably smart regardless.

Doesn’t record size only dictate the maximum record size though? My research concluded that as long as you’re not running a DB with a specific record size requirement/optimization, bigger is better, especially if you have a lot of large files. I don’t know if I could find it again but I also thought that smaller files would utilize smaller records such that ZFS doesn’t waste space with small files with large record sizes.

1 Like

Larger blocks can improve the efficiency of read and write operations for large files, Large block size can also lead to increased fragmentation and wasted space if there are many small files.
Linux Journal has a decent article that came out recently on Linux file systems and block sizes and data structures.

1 Like


I try to demystify it here.

1 Like

Reply to @PhilD13

I could be wrong, but I believe that block size and record size are different. Block size can be defined on creation (with ashift?) and record size is set by the dataset and can be updated to affect future files. Some research indicates that the two terms are used interchangeably but I don’t know if that is done correctly most of the time.

https://www.usenix.org/system/files/login/articles/login_winter16_09_jude.pdf describes that “Writing a 16 KB file should take up only 16 KB of space (plus metadata and redundancy space), not waste an entire 128 KB record.”

Regardless, this is out of the scope of the question- I am wondering why the GUI is showing a lower total capacity than expected. Unless this is just an estimate using the default 128KiB record size, I want to make sure I set everything up correctly.

Reply to @winnielinnie
I am correct in my understanding of record sizes or the fact that the GUI only shows an estimate of total capacity based on the default 128KiB record size?

Also thanks for the writeup, really insightful and solidifies my understanding of record sizes and how it affects datasets. You should consider putting that info into another post under the ‘Resources’ section of the forum for other people to reference!

I’m not sure how the GUI and/or ZFS estimates come up with this number for a pool comprised of RAIDZ vdevs.

You have to remember: A pool can be comprised of a mixture of different vdevs (mirror, RAIDZ1, RAIDZ2, RAIDZ3), and furthermore, datasets in the pool can have different recordsize policies.

So how could the GUI / middleware / ZFS accurately land on the same numbers as the RAIDZ calculator? There are too many variables that can dynamically change throughout the life of a pool.

You may be correct that it just assumes 128-KiB recordsize for everything.

That’s an excellent point, thanks. I’ll just assume that for now.