When to use a NVMe cache

You should only use synchronous writes for zVolumes, iSCSI, databases. Because synchronous writes have such a large performance impact, everything else should be asynchronous unless you have a very specific issue about losing the last 5 secs of data.

SLOG is NOT a cache, and is only beneficial for synchronous writes, and only if the SLOG device is much faster than the data devices - and even then it only reduces significantly the performance impact of synchronous writes rather than eliminating them altogether. You only need a very small SSD / NVMe for SLOG.

L2ARC is only for when you have maxed out your memory and your ARC hits are below 99% due to random reads of a large amount of data. Reading complete files sequentially will benefit from sequential pre-fetch and doesn’t need large ARC. You need a minimum of 64GB before L2ARC can provide any benefit because you use quite a lot of memory supporting the L2ARC device.

You can remove both SLOG and L2ARC if you find they don’t benefit you.

Special allocation vDevs allow you to store your metadata and small files on NVMe whilst the rest of the pool is on HDD. You should be storing your most active metadata in ARC anyway, and you can run a cron job to scan directories to pre-load it too. But it can help segregate small files on NVMe and large ones on HDD. Because the special allocation vDev is critical to the pool, it must be redundant - ideally you want your special vDev to be a mirror of the same width as the redundancy of your data vDev - so if your data vDev(s) are RAIDZ2, then you should ideally have a 3x mirror for your special allocation vDev, however because resilvering is less stressful on NVMe devices you can decide to have a slightly lower level of redundancy if you accept the risks. For RAIDZ pools you MUST be sure you need one of these before you use it because once added it cannot be removed. Again check your ARC stats to see if your metadata hit rate is 99%+, and consider adding more memory and pre-caching metadata if you think you need this.

None of these different vDev types is a substitute for using SSDs / NVMe for data vDevs if that is what you need, and for read performance a larger ARC and pre-caching may be what you need.

3 Likes