I am currently using SCALE ElectricEel-24.10.1 on a UGREEN DXP4800 (16 GB). I am testing with three 500 GB spinning disks. Will change to a pair of 8 TB Barracuda drives. It’s on a 2.5 GB network. Copy files to and from the spinning disk pool I created, I’m getting transfer speeds of greater that 2.0 Gb/sec which I consider great.
This device has a 1 TB NVMe drive installed.
When does a cache (any cache) become useful? From what I am seeing a 2.5 GB network is not fast enough to use the cache.
What are people’s experience? When would it advisable to use an NVMe cache and what type?
ZFS has both read and write caching to memory as standard.
If you are thinking of either or both of L2ARC or SLOG, neither is applicable to your build.
If you want to run VMs or apps, then you should use the NVMe for that. Also what are you planning to use as a boot drive?
But most importantly, 8TB Barracuda disks are shown by Seagate as being SMR and so unsuitable for ZFS mirrors. If you want to use Seagate, then use Barracuda Pro or Ironwolf drives.
You should only use synchronous writes for zVolumes, iSCSI, databases. Everything else should be asynchronous unless you have a very specific issue about losing the last 5 secs of data.
SLOG is NOT a cache, and is only beneficial for synchronous writes, and only if the SLOG device is much faster than the data devices. You only need a very small SSD / NVMe for SLOG.
L2ARC is only for when you have maxed out your memory and your ARC hits are below 99% due to random reads of a large amount of data. Reading complete files sequentially will benefit from sequential pre-fetch and doesn’t need large ARC. You need a minimum of 64GB before L2ARC can provide any benefit because you use quite a lot of memory supporting the L2ARC device.
You can remove both SLOG and L2ARC if you find they don’t benefit you.
Special allocation vDevs allow you to store your metadata and small files on NVMe whilst the rest of the pool is on HDD. You should be storing your most active metadata in ARC anyway, and you can run a cron job to scan directories to pre-load it too. But it can help segregate small files on NVMe and large ones on HDD. You need your special vDev to be a mirror of the same width as the redundancy of your data vDev - so if your data vDev(s) are RAIDZ2, then you need a 3x mirror for your special allocation vDev. You MUST be sure you need one of these before you use it because once added it cannot be removed. Again check your ARC stats to see if your metadata hit rate is 99%+, and consider adding more memory and pre-caching metadata if you think you need this.
None of these different vDev types is a substitute for using SSDs / NVMe for data vDevs if that is what you need, and for read performance a larger ARC and pre-caching may be what you need.
