VDEV Layout question

TrueNAS General
,
1

I am trying to figure out the best VDEV layout for the following disks:

  • 9 x 9.1 TiB HDD
  • 3 x 3.49 TiB SSD SAS
  • 4 x 3.49 TiB SSD SAS NVME
  • 2 x 476.94 GiB M.2 NVME

System has:

  • AMD EPYC 7282 16-Core Processor
  • 125.7 GiB Memory

My plan now is to layout as follows:

VDEV1: 3 x 9.1 TiB HDD RAIDZ1
VDEV2: 3 x 9.1 TiB HDD RAIDZ1
VDEV3: 3 x 9.1 TiB HDD RAIDZ1
VDEV4: 3 x 3.49 TiB SSD RAIDZ1
VDEV5: 3 x 3.49 TiB SSD NVME RAIDZ1
LOG VDEV: 2 x 476.94 GiB

Should I assign one of VDEV4 or VDEV5 as Metadata VDEV or Cache VDEV?

2

Is each VDEV a pool?

BASICS

iX Systems pool layout whitepaper

3

There are no good and bad layouts, there are good and bad layouts for a given application.
So waht are you storing? Documents, VMs, iSCSI shares? A mix of all?
What’s the desired performance?

4

My use case

I store Documents, Photos, Source Code Files, HTML Files, CSS File, Flac’s, MKV’s, …

I run about 20 apps in containers, like: Immich, OpenHAB, Picard, roonserver, telegraf, influx, grafana, to name a few.

5

Nope, I would like one pool. My question is on the mix of SSD based and HDD based VDEV’s. Will the play nice?

6

Are these NVMe cards in an adapter with a SAS interface? Or something else?

Once I understand what these are I will come back to you with a suggested use for all the NVME drives, but

For the non NVMEs I would create an HDD pool with a single 9x 9.1TiB RAIDZ2 vDev, and an SSD pool with the 3x 3.5TiB SAS SSDs in RAIDZ1 (possibly with a replica backup to HDD).

I will almost certainly allocate the two M.2 NVME as a mirrored boot drive. The remaining 4 large NVMe drives still to be allocated but likely to use them as a 4x mirror for special metadata for the HDD pool.

Edit: Or I might use the NVMe drives for an SSD pool RAIDZ1, and the SAS SSDs as a metadata mirror for the HDD pool.

1 Like
7

SAS and NVMe do not go together.

So both big files and many small files.
MKV and photos would do fine on raidz#. The small files would fare better on mirrors.
But since you have large SSDs, you have the option to make a special vdev with a suitable cut-off size for small files.
HDDs is 9-wide raidz2 or raidz3, and 3 or 4 SSDs (same type) in a 3/4-way mirror as special vdev. Check that the size is right.

These could be a separate pool just for the containers if you don’t want to put .ix-apps together with the documents. Otherwise, they have no obvious use here. Except boot, but you can find even smaller drives and use these bigger drives elsewhere.

One of these sets is unlikely to find a use with a single pool. With the HDDs, that would be three tiers of storage.

I do not see a need for sync writes, so no SLOG, and it’s not apparent whether these drives have PLP.

L2ARC is unlikely to be of use with 128 GB RAM.

1 Like
8

I forgot to mention. I also have
2 x 894.25 GiB SSD as a boot pool

The NVME disks are actually SATA 6Gb/s drives. My mistake.

9

My gut reaction is that you have more sets of devices with different speeds than you can use. If you want to use all the drives then I would group the SATA and SAS SSDs as being broadly similar speeds, use them both in the same SSD pool but in different vdevs.

So my recommendation is now:

  • hdd-pool - 9x 9.1TiB RAIDZ2
  • ssd-pool - 3x 3.49TiB SAS RAIDZ1 + 4x 3.49TiB SATA RAIDZ1
  • boot-pool - 2x 900GiB SSDs mirrored (mostly wasted space)

Not sure where 2x NVME drives would be most useful - but I suspect that they should be used as a mirrored pair for some type of non-data vDev but which type and on which pool really depends on the workload and where they can be most beneficial.

10

PM983 are NVMe drives with PLP. SATA would be PM8xx.

:100: But no one has to use all available hardware, and all possible features of ZFS.