i have space for 10 500gb SSD’s how to handle them? i have 10 plus some spares.
i was thinking of just putting them in a Raid2z but im getting mixed info some people say its fine and some say it will mess the drives up on the way they handle the blocks and will Wear out the drives faster?
so leave them as raid2z
set them up in a Raid10
or set them up in a bunch of raid 1’s per app.
The end uses is to Hold the “ix-apps dataset” and my apps and VM’s
I think a 4-wide stripe of 2-drive mirror VDEVs, and 2 spares, will give you the best balance of performance and data safety in this pool.
The VM part (zvols: block storage) is where raidz is less appropriate.
How much space do you need?
You know, everyone has an opinion. What you want are facts, not opinions.
I think you have your answer in the two replies above, but these do not take into account your desired capacity, which you did not state.
Desired capacity of the storage pool is pretty important. Also, how important this data is to you (affecting data loss and/or recovery time), which is where redundancy plays a role. These are two very important factors. I would suggest you provide that data to see if this changes the answers provided.
ok that did not help my problem. it’s not speed even in raid2z that i have it setup in i’m maxing out my 10gb network.
and all the apps and VM boot crazy fast.
i just wanted to know if the raid2z will burn out the drives faster?
well as much as i can get but will a little fault tolerance..
i will be backing up the SSD pool every few months..
Your OP was more wide of a question than that, you also asked about leaving them in raidz2, Raid10 (we don’t use RAID10 here in ZFS but we got the point), or set up as raid 1’s. We were trying to answer all your questions.
I have never heard of a SSD becoming burned out faster due to the configuration. However, technically speaking (and some people really hang on to “technically”): Certain configurations will write more data to a drive than other configurations. However, is it actually “significant” enough to be concerned about? Most people would say No. What is a few extra TBs over years of use? That is up to you I guess. My answer would be to not be concerned about it in this respect. I would monitor the Wear Level of each drive and if it looks to be excessive, determine why. It would not be due to your drive configuration. Consumer SSD’s generally wear out significantly faster than enterprise SSD’s. This is where TBW comes into play.
Since you have your system already setup, you could analyze the drive data and calculate your expected drive life based on TB Written and the Wear Level. Collect your data for all your drives on June 1st, 2026. Then collect a second set of data on 1 July 2026. Now you can perform a realistic evaluation. If your system is highly active, maybe 1 week of data would be enough to extrapolate the longevity of the drives.
If you would rather make it a little easier, somewhat, you could install the Multi-Report script, run it every day as designed, and every day you would get an emailed report with all of this data. The far right columns display TB Written / TB Read, and of course there is Wear Level data. The default setting will display a rolling 30 day average as well. This is great for identifying heavy usage.
Now this does not exactly answer your question, but it provides you a method to help you see where you are at.
You also should look up the Warranty of those SSDs, specifically how many TBW are covered under the warranty. If you cannot locate that, then the Wear Level will help out.
If you desire to try out Multi-Report, please read up about it, what it does, and read the Quick Start Guide before installing. Installing a script like this might be out of your comfort zone, that is okay. I can tell you how to collect the data manually if desired.
I hope I didn’t give you too much information, just trying to ensure you have something to read and try to understand what is at play.
Totally agree with what joeschmuck said, just want to add a few things. Hopefully other can chime in, I am not certain on all of these things, so take it with a grain of salt.
SSD internal write amplification
SDD firmwares are like black magic. You don’t really know what is happening inside.
So just because you only write 4k, that does not mean that this is what is happening inside. That is also why TBW alone is not the whole story. For cheap consumer drives, TBW could have been calculated on the assumption that you only write larger than 64k files to it. Writing 1TB onto them can show up as 4TB written in the SMART data. That is why believe to remember some enterprise drives to specify the TBW for small 4k writes.
Firmware risks
Are all drives the same model? If yes, realize that a single firmware bug (this is not theoretical, we had those in the past) can wipe out your pool. If you have two different models, I would put model A and model B in a mirror and then stripe over these 5 mirrros. We would call that RAID10 in a non ZFS context.
write ampflification ZIL
You will get some write ampflification for sync writes. Sync writes happen twice. One time onto disk, another time onto the ZIL (which is also on the pool, unless you have a SLOG)
write ampflification from volblocksize
default blocksize or volblocksize is 16k. So your VMs will get offered 16k blocks from ZFS. But what if your Windows VM opens up a 4k txt file? First of all, you need to read the whole 16k block containing that file. You make some small changes and save it. Now you might think that you only have to write 4k, but you have to rewrite a 16k block. That 16k block itself will need to write 36k on a mirror or 24k (8k parity + 16k data) on a RAIDZ2. But yeah, your 4k write ended up worst case being a 36k write.
storage efficiency
You might be tempted to use all your 1TB (just an assumption) SSDs in a 10wide RAIDZ2. You would assume that just like with RAID6, this will give you (10-2)*1TB=8TB usable storage. But blockstorage behaves differently.
All your storage is just many, many 16k blocks. How do 16k block behave on ZFS?
With 10 drives, we need a stripe only 6 drives wide.
This is because we don’t need 10 drives to store 16k of data.
Each stripe has four 4k data blocks and two 4k parity blocks.
That gets us to 24k in total to store 16k of actual data.
We expected a storage efficiency of 80%, but only got 66.66%!
Because we can’t use a stripe that is 10 drives wide, only 6 drives.
So we have 4/6*100 = 66.66% and not 8/10*100 = 80%!
If you are interested in this topic, here is an more detailed explanation.
You can counter that efficiency problem by using 64k as volblocksize instead. But remember the 4k write we described earlier? Well, that is now a 128k write in a mirror.
TLDR: 66.66% instead of 50% is not worth the performance downsides of RAIDZ2 over striped mirrors (RAID10). But of course this depends on what VMs you plan on running, what drive models you have and so on.
@Sara
Thanks for posting all the great details. It is good to see those published periodically from someone who has all that experience. My experience is significantly less.
@Matt2
Don’t let this data overwhelm you. It is great information (well I find it invaluable so I understand what is actually going on or what we think is going on) and if you are designing a server with certain needs, this information can become quite handy. There are people who go into this detail to squeeze every bit of possible performance they can. Most people don’t need to do this.
The drive firmware issue mentioned, I too have seen it just once, however it was a Windows drive that took a dive on me. I would not be terribly concerned if you are doing good backups but it is a possibility.
ok thanks for all the info ill have to read this a few time
and BTW i am running Samsung evo 860 drives with the latest firmware.
And remember, these “problems” mostly apply to zvols and their static volblocksize.
Datasets have a none static record size with a upper limit record size value (by default 128k).
Blockstorage sucks!
In so many ways! You don’t want to use it unless you have to. Do not create a 8TB Jellyfin VM and put 8TB movies into it! Backing up such a VM is a PITA. Instead use datasets. Use a small 32GB Jellyfin VM that mounts a dataset and put the movies there. AFAIK you could also use the Jellyfin app and would be using datasets. I use TrueNAS only for storage and Proxmox for VM/Apps, so I don’t really know.
The problem you’ll face isn’t technical - it’s supply: When one or more of these drives fails, from where will you find a replacement? It’s difficult enough to find a 2TB drive these days.
With that in mind, if you create a series of mirrored pairs, you’ll be able to upgrade a pair at a time to newer drives.
oh i have a few spares but i ended up slimming it down to only 5 drives.
