LOL, I just started a long reply, so I will recycle some of that content here.
Planning / Preparation
A sVDEV requires a lot more planning but it can speed up operations for small files and metadata consistently, see here. For sVDEV use, your drive will be split by TrueNAS into two halves, one is for small files, the other is for metadata. Unlike L2ARC, if your sVDEV goes, so does your pool, so I use a three-way mirror of enterprise-grade SSDs. You will have to look into how much room your small files take up to plan for the proper SSD capacity to use as a sVDEV.
To plan for sVDEV, you need to review the size distribution of small files in your pool - current and expected. ATM, I cannot recall the CLI commands that compile that information, apologies. If your sVDEV small file catalog overflows, the small files will be written into the slower main pool and the benefit of a sVDEV re: small files will be diminished.
For my use case, a 1.6TB sVDEV seems to be sufficient (50TB pool). However, I made a point of nuking every small file I could by consolidating small files into sparsebundles and like archives on the server. My file size limit for small files is 32kB, however your limit will vary based on your use case and should be investigated carefully.
Similarly, you should determine how much metadata your pool needs. This helpful post has the CLI command for that. Remember, only 1/2 of your sVDEV pool will go towards metadata, so plan accordingly (i.e. allow sufficient room based on how big you expect your pool to be). The rule of thumb (and it will vary as the use case dictates) is 0.3% of your pool size will be metadata.
Performance
I found finder / directory traversal / rsync performance to better than via the use of a persistent, metadata-only L2ARC, but not by that much. Small file performance is no comparison vs. a single-VDEV HDD Z3 pool. I would only use enterprise-quality SSDs in at least a 3-way mirror. I have cold, qualified spares on hand (Intel S3610 @ 1.6TB, IIRC)
What happens if the sVDEV fills up?
Unfortunately, the GUI still gives the admin zero insight into how full a sVDEV is - either on the metadata or small files side. So you will have to brush up on your CLI skills to check occasionally, especially if your pool is undergoing major changes like the addition of a busy database, for example. If the partitions fill up, performance will likely become very uneven as some files end up in the pool vs. on the SSDs in the sVDEV. I have a SLOG in my system (p4801x) and it’s likely way faster than the older S3610s. So if a fast SLOG helps, I have one.
When to use it
Besides the standard Edmund Hillary / George Leigh Mallory response (“because it’s there!”), I’d argue sVDEVs are very useful in pools where there is a lot of changing, small data. Unlike a L2ARC, a sVDEV does not need to get “hot” as the metadata are already on a fast SSD. Similarly, HDDs and ZFS in particular suffer from writing small files due to all the latency involved - waiting for write confirmation, and all that.
I use a sVDEV simply for the speed it provides consistently when it comes to browsing directories or making backups. Rsync backups just blow along at ridiculous speeds as unchanged files are compared. However, at least the metadata bit can be emulated to a large extent with a properly-sized L2ARC (that is persistent and metadata=only) and the better ARC implementation that is coming may further speed things along.
Anything else?
I have found sVDEVs to be a great tool, if handled carefully. For example, DO NOT believe ZFS if if tells you that a perviously 100% healthy pool is suddenly dead w/o reason. It could be something as simple as your sVDEV SSDs having lost a power connection or the TrueNAS update process causing your SSDs to soft-brick themselves, necessitating a power cycle. Check the cabling and ignore bad advice from the GUI and the ZFS command line until you have exhausted all other options.
