Optane or PLP never hurts, but sVDEV has no special requirement. I would just avoid QLC drives, as the workload will be lots of small writes and rewrites.
As discussed here, “asize” should be the right metric. So 137 GB up to 32k blocks, and 298 GB up to 64k blocks.
(Not entirely sure why “asize” swings under or over “psize” as it does, all for 54 GB of logical size, but that’s ZFS at work…)
I’m using two WD Black 770 2TB (dramless, bought new) and a Samsung 980 Pro 2TB (with DRAM, reused from my current desktop), all NVMe. Two in separate PCIE bifurcating cards and the third in a motherboard NVME slot.
There isn’t an awful lot of writing done to them, but there are a fair few reads. It’s not like L2ARC which has constant writes.
So, those mostly matters to the SLOG i think then?
Yes, so as per this metric, how large sVDEV should i buy? Also why other block size are not relevant here?
Yes, just the Metadata writes, which is supposedly lower.
How big of a sVDEV to buy should be determined ahead of time by looking through your file system re how many small files you expect to have to deal with vs. the size of the metadata needed to keep track of your pool. I put together a resource on that.
Also, to get all the small files onto the sVDEV out of an existing pool, you will have to rebalance the pool. My resource details that process also.
I get that, but i’m new to it and trying so much hard to understand. Could you please have a look at my output and let me know what capacity should i buy?
I have added the @Fastline data to the bottom of the sVDEV resource page as a further example of how to look at the data, please see the sVDEV resource for more details.
Thank you for using my data as an example in your guide. That has helped me a lot to understand things. However, i have a few questions:
Is there any reason why the block size above than 64K are not taken into account? Why not up to 128K?
I’m using default settings. Should I adjust the record size?
How do I know what block size should I cut and what should I use? Also, for up to what block size it is beneficial to add an sVDEV?
I would adjust record size by dataset / purpose. As per the sVDEV resource, consider 1M record sizes for large media files like images, videos, and like content.
Smaller record set sizes make more sense for databases and VMs. You can dramatically reduce the need for metadata that way and help speed things up by allowing more continuous writes and less time getting spent calculating metadata.
In my instance, I reduced the metadata needs of the pool by 4x and sped up writes from about 250MB/s sustained to 400 via sVDEV and recordsize adjustments. So I definitely recommend adjusting record size by media / data type. It saves space and speeds things up, at ZERO cost.
Now, is it possible that larger small file cutoffs could be beneficial in your use case? It’s entirely possible, but with your default setting of 128k recordsizes, it’s impossible to know the amount of actual 128k data in the pool since it’s commingled with data that’s 256k+ in size.
To get ZFS to tell you, you’d have to change the record sizes (to higher #s) of various datasets, rebalance the pool, and rerun the ZFS command to see what the actual 128k file size content is.
So if keeping larger files in the sVDEV is important to you, I’d definitely buy more than the bare minimum sVDEV capacity. For that case, I’d consider 2TB since the cost is relatively low and it buys ample room to grow. As for where to set the small file cutoff, it’s variable by dataset / use case.
For example, you may have a dataset where you’d want to keep it 100% on the sVDEV due to speed / IOPS / whatever (ie virtual machines). For that use case, setting the small file cutoff to the same size as the record size for that dataset ensures the vm data stays 100% on the SSDs of the SVDEV.
Similarly, if you have a photo archive and most of the files are huge but a few small files are in there for thumbnails, etc. you should set something like a 1M record size for the data set and a 64k or maybe 128k small file cutoff to speed up the little stuff.
Can someone experienced please suggest me the best SSD size for the fusion VDEV in my case, and the best allocation size for the pool?
I’ve been strugling for days to do it myself, but can’t seem to make it.
Main goal is to speed up Windows SMB thumbnail loading for the folders that have alot of images. “Main HDD” pool in question will not grow past 4 TB (2x4 TB mirrored drives). It will however lose many large video files in the following year.
Please find the attached result of my pool, from the command “zdb -LbbbA -U /data/zfs/zpool.cache”. Command was used while I was on theTrueNAS Core, and I’m now on a TrueNAS Scale v24.10 (upgrade went well).
You could hardly make it LESS readable, couldn’t you?
All necessary information to read this long blurb is above:
You have 2.76 GB of metadata, which may very well fit all in ARC if the NAS has enough RAM, and most of the data is in 128k records.
You could easily have a sVDEV of 128 GB at 32k cutoff (29.1 GB now at 50% occupancy); 256 GB would be a bit tight for 64k (146 GB).
Really, for less than 4 TB of data you could drop hard drives, go all SSD and forget about a special vdev.
Couple of thoughts,
Thanks you very much for the reply, and the valuable information provided!
Sorry for the blurb, I didn’t think that the forum post would look like that, my intention was to have it as a downloadable attachment, all in a single image. Afterwards, I didn’t find it hard to read, but I will surely increase the quality next time.
I would love to have all SSD setup, but my options are limited atm. I have two good quality SSDs to spare, they are 256 GB each.
Thank you too for the reply Constantin, means alot to me.
If those are quality SSD’s they can likely be used for the sVDEV on a pool that small. I’d add them, set the record sizes as appropriate for each dataset, adjust the small file cutoff to something like 64k, then rebalance.
At this point, metadata should be flying locally, ditto small files. You could rerun the analysis and see if having even more small files in the sVDEV would make sense. If so, adjust the small file cutoff again, rebalance again.
Anything you try to run over a VPN will likely be throttled by your ISP upload speed capacity, not the NAS. For example, my ISP intentionally only allows 1/10-1/20 the upload speed vs. download speed. So keep in mind that local connections / responsiveness will be much faster, esp. for directory traversals, but remote connections will be hamstrung by your modem / ISP / internet connection.
