I’m setting up a new TrueNAS Scale server. I’m using a DL380 Gen9 with 128GB of RAM and 48x (+2 Spare) 900GB 15k SAS drives via a 16e HBA. Currently connected through 10Gbit.
I’m currently pondering how exactly to configure the storage pool.
Should I go with 12 VDEVs Z1, each with 4 disks, or
4 VDEVs Z2, each with 12 disks?
Both options roughly yield the same capacity.
What about Cache and LOOG VDEVs? I have an HP (Samsung 1725a) 1.6TB NVMe PCIe card available. Would this make sense for the following use case?
My applications are quite diverse. I use ESXi and Proxmox virtualization for cloud, media, and database servers. Protocols include iSCSI, NFS, and SMB shares.
Thank you for the two very detailed posts. However, what confuses me is the following statement: “Buy 5400/5900RPM drives much larger than you’d otherwise think you need if you want fast write speeds. I think 7200RPM drives are for chumps.” Does this mean that my 15k RPM disk ared crap and would slow down the system?
If I understand correctly, the best approach would be to upgrade the hard drives and, for example, set up a pool with 8x8 or 10x6 drives.
If I’m reading it correctly, it does make sense for my use case to use an SLOG device to improve synchronous writing. How large should this partition be? But does it also make sense to use a cache VDEV L2ARC with a 128GB ARC?
I wouldn’t think so–all other things being equal, they’d still be faster than slower drives. But the main points for performance are (1) lots of free space, and (2) lots of vdevs.
A SLOG probably does make sense, but it should be a dedicated device, not a partition on another device. I’ll have to leave device recommendations and sizing to others as I don’t have any use for one and thus haven’t looked into them in any real depth.
As to L2ARC, I’d suggest waiting and seeing if you need it; it can easily be added to the system later.
SLOG only needs to hold two transaction groups—with defaults settings, ten seconds.
10 s * 10 Gb/s = 100 Gb, a little over 10 GB.
A 100 GB Optane P4801X would serve you better here.
As for L2ARC, if you actually have a use for it (test first!), the recommended size is 5 to 10 times the size of ARC (RAM); more than that is actually detrimental for performance. The PM1725a is too big.
Thank you very much for the response.
I know it’s not officially supported, but would it be theoretically possible to divide my 1.6TB Enterprise NVMe into a e.g. 15GB partition for SLOG and a 1TB partition for L2ARC?
Normally, partitions shouldn’t be used, but would there be significant disadvantages if one were to do so? I’ve read here in the forum that some have successfully implemented this.
That’s possible, though unsupported, and Optane handles that double duty gracefully.
Most drives other than Optane, however, have a “bathtub curve” for R/W workloads: They are good at 100% read or 100% write workloads, but not so good at anything between 80/20 and 20/80 mixed workloads—which is exactly what SLOG (write only) + L2ARC (mostle reads, if it serves its purpose) would throw at them. So it is likely NOT a good idea for your Samsung drive.
