Mixing different drive speeds

TrueNAS General
,
1

Hi,

I’m considering adding a “fast volume” to my pool consisting of SSDs next to my HDDs. The new volume should be independent of the exisiting store, not a cache to the existing VDEV (not enough RAM for a L2ARC, and not willing to dip into special/SLOG and other dark magic.)

How would you go about mixing different performance levels? For sake of example, let’s make it 2 SATA SSDs and 2 NVME SSDs. Would you…

a) just put them all together in a single VDEV with RAIDz-something?
b) create two VDEVs containing two mirrorored drives each, with two independent datasets?
c) create two VDEVs, each mirrored; then create a single dataset spanning both VDEVs?

I’m well aware that either will make little practical difference in my case, because my bottlenecks are elsewhere (starting with network speed.)

But which would you prefer?

2

You can think of this a few ways:

  1. Cost: 2.5" SSDs (SATA connection) are significantly less expensive than NVMe drives.
  2. Physical Space. SSDs require a place to mount them and while some cases have limited identified locations, a smart person would drill some holes in a few strategic locations to mount additional 2.5" SSDs. Or use velcro tape. M.2 is limited to the actual connections.
  3. Maximum speed of your pool. If you are running VMs and you need the fastest operation, well NVMe is the answer. It doesn’t need to be about network connectivity speed. But if you are using an N100 motherboard, why waste the money on M.2.
  4. Pool growth. M.2 is limited to a very specific connection which in a physically large system is likely not the issue, however SATA connected drives can easily be added to most systems, if they already do not come with enough on the motherboard.
  5. Pool recovery. If your motherboard dies, using SATA connected drives it is an easy move to relocate these 2.5" drives into a new computer and boot up TrueNAS, and your data be there. With M.2, you need those connections and if you don’t have those, you are screwed and have to wait on a solution that allows you to do that. With that in mind, my personal NVMe server has a 4 NVMe slot PCIe card so I can move the entire pool easily, UNLESS the adapter card fails. So I’m up shit creek for a while where with SATA I would be able to recover in hours.

As for mixing speeds, I don’t see any issue for or against it given the media we are talking about. Spinners would have obvious latency concerns, even though it would functionally work.

I think a better question for you is: Are you able to keep the M.2 NVMe drives cool? Are you buying Gen 5 or better? Are you buying one that supports SMART fully? Yes, that last one is a thing you generally never see an issue with for a 2.5" SSD.

Hopefully my words have given you something to think about. M.2 NVMe drives are great, but they do have a few minor drawbacks: Cost, Heat, Interface availability, Recovery moving pool to a different system.

If you are trying to design a system, think about all of these factors, they very well could impact you. Good luck.

1 Like
3

That doesn’t make sense.
Either you’re literally “adding a fast vdev to your pool”, and then it’s not “independent of the existing store” (ZFS will stripe data across vdevs), or you’re “adding another pool to your NAS”, and then it’s indeed independent but the terminology is all wrong.

What is the purpose of the additional storage, and what will it be used for?
Pool design is all about use case.

5

You can add a Z-volume to a pool.

Anyway, you want to create a separate pool from your ssds.

Then you can make as many datasets as you want on that pool and use them as fast “volumes”

6

I expected to specify on creation of the new (“fast”) dataset which vdevs it is supposed to use. It was my understanding that two or more vdevs can be organized within the same pool, but still be independent of each other, as long as there are separate datasets that use the respective vdevs exclusivly. Is this wrong, would the two vdevs automatically “mix” (= be striped together even without my explicit say-so?)

If so, I only now understand @etorix comments: In that case of course my terminology did not make sense after all. Sorry about that. :wink:

I see. In that new “fast pool”, would you create a single vdev for all (SATA and NVME) drives, or two vdevs, one for each type?

7

All vdevs in a pool are striped, unless they are of a special type (special/dedup/slog/l2arc). You cannot assign a dataset to some particular vdev: A pool is a whole.

So back to the question: What is the use case for a “fast pool”?
Do you have a use case for a “fast” (SATA SSD) and a “super-fast” (NVMe) pool?

8

I wouldn’t mix nvme and sata vdevs in the same pool (excluding special vdevs)

A pool is made of one or more vdevs striped together.

A VDev is made of one or more drives either mirrored, or raidZn where the n is how many drives can fail in the VDev before risking data loss

Vdevs provide redundancy.

The number of vdevs provide iops

A pool can then have datasets and volumes. Which are used for shares and vm disks etc

You can have multiple pools.

And you would want a slow+big pool and a fast small pool.