Vdev layout (between 18 to 30 disks)

TrueNAS General
1

Hi
So I have my server now (see here).

Now I am wondering about vdev layouts.
This will primarily be used as Plex storage.
I already have a 12-bay SSD pool on my Synology that I am using for my photos and other IOPS-heavy stuff.
That said, I don’t mind if the TrueNAS server has good IOPS (that is always nice).

Now, there are many posts about vdev layouts, but often the number of disks is fixed for these posts. My situation is a bit different, as I have quite some freedom on the number of disks (starting with an empty 45 bay JBOD).

I will likely eventually go for 196 GB of RAM (and thus possibly not need an L2ARC device, though it will be trivial to install an M.2 SSD if needed).
I understand I won’t benefit from a SLOG device.
Also, I do not want to mess with a metadata device (though I understand a three-way mirror of SATA SSDs would be a good choice for this).

I will likely go for 28 TB disks (or similar, depending on market prices).

Regarding storage needs: any of the numbers in the table below are ‘ok’ for me. Clearly I don’t want to waste money though.

I think I understand the parity implications. I drafted the table below.

The IOPS and throughput implications are a bit less clear to me.

My favourite layout is currently 3 vdevs of width 8 using RAIDZ2, equating to 24 drives.

What is your view on a good vdev layout?

disks usable TiB parity drives per vdev vdevs vdev width ‘usable’ drives per vdev Parity %
18 356 2 2 9.00 7.00 22%
18 306 2 3 6.00 4.00 33%
20 407 2 2 10.00 8.00 20%
21 382 2 3 7.00 5.00 29%
24 407 2 4 6.00 4.00 33%
24 458 2 3 8.00 6.00 25%
28 713 2 4 7.00 5.00 29%
2

I think this guide can help. At least it gives a theoretical estimation of pool performance.

3

thanks so IOPS seems linear in the number of vdevs. But I think ‘few vdevs’ can still get a high throughput if ‘a lot of disks’ are used

1 Like
4

I think that the sweet spot in price per TB is currently at slighly lower capacities thant 28 TB. Check that for your calaculations.

Do you have a backup of your data?
If no, you may want to avoid 10-wide raidz2 and add a few spares to increase resiliency. Or go for raidz3.

Do you expect that storage may expand to fill all bays in the enclosure? If so, 6-wide or 7-wide, with up to 3 spares, looks like nice divisers. (With raidz3, 4*11-wide + 1 spare.)

5

It is unclear to me if I will ever fill out the entire JBOD. But 6* 7 is 42 leaving 3 spares (obviously 5*9 is 45 which would be another option). So right now, going with 21 drives initially seems tempting.

I will need a few months before I can invest in HDDs.
I have a backup of the important bits (which is only a small portion of the total storage).

I can check again regarding pricepegig etc. serverpartdeals has an interesting offer for 28TB drives these days.

1 Like
6

My system has two VDEVs in its pool, each consisting of 8 spinning disks in RAID-Z3, with no cache or slog SSDs.

Together they can achieve a whole gigabyte per second on a 10-gigabit network.

I don’t know how much storage capacity you aim to achieve, but if you have extra bays perhaps you could create a second pool to use as a backup for the first?

7

Thanks. Saturating 10G would be nice. Seems like this should be achievable if I set up things right.

Raidz3, so many options. I guess one could also go 12 wide with three disks of parity (or more?). But due to IOPS I think I lean towards more vdevs.

8

I’m guessing 5 or 6 VDEVs like mine would keep up with your Mellanox NIC.

I prefer RAID-Z3 for extra peace of mind if one has the bays available.

Also, you can save money by getting decommissioned drives in bulk. Make sure you get a few more than you need for spares too.

May I ask how many movies you plan to store on your PLEX server? I’m hosting about 360 movies on my Jellyfin server (mix of ripped DVDs and Blu-Rays) and only using about 5TB worth of MKV files.

1 Like
9

I am not sure. I have 12 x 18 TB today in my Synology. Btrfs likes to have some 20% free, maybe a bit less. Not sure if this will also apply to ZFS.

In any case my current storage is essentially full. Thus I definitely need something along 200 TB preferably more.
Storage can be nice for seed pools too.