TrueNAS Scale: Creating a Mirror Pool with Disks of Different Sizes

Hello everyone,

My NAS is running TrueNAS Scale 23.10 and I have 12 hard drives of different sizes, totaling 32TB of capacity.

I would like to create a mirror pool with a first 16TB stripe vdev and a second mirror of the first one, also 16TB.

The issue I’m facing is that the TrueNAS interface doesn’t allow me to configure this setup.

How can I proceed to create such a disk array to have the performance of the stripe and the redundancy of the mirror?

Thank you for your responses.

PS: I have attached a quick diagram of the layout I want to implement.
image

That’s not really how zfs works.

You can do what you want with partitions and md.

Would be better to get another 8TB.

Use your triples to make raidz1s. Use your doubles to make mirrors. And stripe the lot.

3 Likes

If you really want to use this mishmash of variable size disks, you might want to look into unRAID.

It allows you to assemble a JBOD of any and all types and sizes of drives, where you can dedicate one or two drives to hold parity data.

unRAID uses a proprietary “data protection” method, where it uses the XFS filesystem, adheres it across multiple drives, and then saves recovery/parity information on the dedicated parity drive.

Unlike ZFS, it is per file-based parity. Think along the lines of “PAR2”, but for a NAS.

In your case, you could use the 8-TiB drive as the “parity drive”, which allows for the loss of two 4-TiB drives, or three 2-TiB drives, or a 4-TiB drive and two 2-TiB drives, or any combination of drives that are equal to or less than the size of your parity drive.

I wanted to test this approach. However, when I create my first vdev with RAIDZ1, for example for the 2TB disks, I can no longer use another RAID mode for the other disks. The TrueNAS interface blocks this possibility for me.

Using unRAID wouldn’t have bothered me if they had stuck to their formula: you buy once and you have access to the system with updates. And then the second problem, I’m used to TrueNAS; I used FreeNAS back then, which, by the way, I find allowed more possibilities than TrueNAS Scale now, but it’s probably just me being the problem.

What does that mean?

Are you referring to vdev types?

I believe there is an “override” option that allows you to mix mirrors with RAIDZn in the same pool.

An image is worth a thousand words. I’ve created a pool to show my problem, so I have 2 vdevs in RAIDZ1 with 2TB and 4TB hard drives. However, for the 3TB hard drives, I only have two and I don’t have the option to change that, so I can’t use them.


If there is an option allowing me to disable this limitation, I’m interested in it.

There was a discussion about this in the old forums.

Judging by your screenshots, they never implemented an “override” option.

You could add the two-way mirror vdev to the pool via the command-line, but I’m not sure how the middleware/GUI will behave afterwards?

Keep in mind that once your pool is comprised of at least one RAIDZn vdev, you cannot change your mind later (such as wishing to remove the mirror vdev at a later date.) This is likely why the GUI doesn’t allow you to mix RAIDz and mirrors in the same pool.

That’s why I’m doing research now, to avoid having to redo everything later. But basically, if I understand correctly:

  • Either I switch everything to mirror and in this option, I have to add a disk of each capacity.
  • Or I add a 3TB disk and two 8TB disks to continue with RAIDZ1.

I find this very limiting to impose such a way of doing things. Whereas stripe with mirror would provide the same security, but not all this complicated mechanics.

But what’s surprising is that even Stux suggested having different vdevs in the same pool, so it’s possible! However, I can’t find any information about this on the internet, other than going through the shell. But I’m not convinced that forcing a mechanism that wasn’t intended is the best option for having stable and trouble-free storage over time.

With ZFS. The TrueNAS middleware/GUI is what prohibits this.

If you purchase an additional 8-TiB drive, then you can construct a pool with mirrors, totaling anywhere from 15 to 17 TiB usable capacity; depending if you keep the 2-TiB drives.

So if I understand correctly, I’m using the TrueNAS shell with zpool to define the types I want to use. But won’t this make my pool unstable if the interface doesn’t offer it?

In terms of the GUI/middleware behaving unpredictably, since it is not expecting mixed vdev types in a pool? I’m not entirely sure.

I think @HoneyBadger has tested this out, and he said it worked as expected, but the GUI complains that the pool has mixed vdevs.

I will test this out and we’ll see.
Thank you very much for your responses.

You may get different results if you create your pool, say with just 1 raidz1 vdev, and then add different vdevs after creation.

With raidz vdevs, you can always add, but you can never take away

I do it this way, I create the first RAIDZ1 during the pool creation, and then I add the others.
The screenshots I shared above are after creating the pool and returning to the window to create the new VDEVs.

In the olden days there was no trouble doing this.

Now, well, the issue is you want to partition a set of disks equivalent to the way the middleware does, then you add the zfs partitions in the cli via its partid.

zfs add blah blah etc

I don’t know of a a recipe to do it for scale. It probably exists on the old forum.

(I think I need to play with a VM and some disk images again…)

You can try the “test” yourself, assuming you are not yet storing valuable data.

You can also ask @HoneyBadger about his experience with it, since I believe he tried out this very thing with SCALE in the past: bypassing the GUI when adding a mirror vdev to a pool that already contains a RAIDZ1 vdev.

Yes, the pool is empty so I can test as I like. That’s why I’m doing it now

Going to the CLI will of course bypass all of the recommendations and protections from the middleware - which will allow you to do all of the things that ZFS will allow - including mismatched vdev types/redundancy levels - but you will get a “Mismatched Vdev” alert in the GUI, as well as the potential for unexpected or undefined behavior on things like drive replacement.

2 Likes

If I understand correctly, if one drive fails, there is a nonzero possibility that the pool may not be able to be rebuilt, despite a replacement?
So if I want to increase the capacity of my NAS, I have no choice but to buy more 8TB drives.
Managing disks with different capacities is really not recommended in any case.
That’s really unfortunate.
Thank you for the feedback.