I have a bunch of HDDs (5x30TB) and a bunch of SSDs (7x8TB). Instead of having two pools and having to manually decide what goes onto the fast and what goes onto the slow pool, I was planning to make one pool, with the SSDs being used for metadata and small files, particularly now that the ZFS rewrite feature would allow a potential future rebalancing after changing the file size threshold.
But it turns out that while I can make a RAIDZ2 from the SSDs and the HDDs separately, the special VDEV can’t be in RAIDZ2 format, only striped or mirrored. Is that a ZFS limitation, or a limitation of the TrueNAS WebUI?
It seems a bit odd, that the same drives that can host both metadata and files big and small in a RAIDZ2 configuration, are limited to being mirrors and stripes when using only metadata and small files.
Yes, I understand that the speedup won’t be as big, as RAIDZ2 is slower than a mirror, BUT it’s still going to be faster than HDDs by themselves, and certainly a lot more convenient than having two pools and manual assignment of duties and potential file shuffling/symlink nightmares.
Now, if that’s a UI shortcoming, I’d love to see that fixed, it that’s a ZFS shortcoming, then, well, disappointing, but nothing that can be done about it in the foreseeable future…
special vdevs were designed for more efficient storage of small files, and of the tails of large files, in combination with dRAID, which is especially bad at dealing with small blocks. So it’s mirrors only because raidz is also bad at storing small blocks, though not as horribly bad as dRAID.
But conflating tails as well as small files, makes svdev somewhat less desirable with raidz, and harder to plan for. Metadata-only svdev still makes sense with raidz, but then it only requires small capacity, and is only useful if there are many (re)writes; for read workload, tuning L2ARC for metadata provides similar benfits without introducing a pool-critical vdev.
8 TB SSD are huge compared with a 5-wide raidz2 of 30 TB drives. In theory, you could have one or two 3-wide mirrors of 8 TB as svdev alongside the 5*30 TB raidz2 (to keep a consistent redundancy), but does that make sense?
Makes sense in the context of the original design intent.
From my point of view, it’s all going to be RAIDZ2 anyway. So there won’t be any special VDEV to speed anything up, my use case was to get RAIDZ2 metadata of spinners to RAIDZ2 SSDs (should be faster) and to mostly transparently have small files on SSD rather than spinning drives.
So it’s for me not a question of how can I improve the speed of RAIDZ2, for me its a question of which part of my data is on slow spinning RAIDZ2 and which is on fast SSD RAIDZ2.
I guess from the afterthought (if we have metadata and tails there, we might as well have small files there) came a new use case (“Fusion Drive”) where large media files (e.g. movies) and documents, databases should reside on a single RAIDZ2 pool, but the data automatically end up on the media type most appropriate for the job, and the metadata always being on SSD rather than spinning drives.
Sadly, I get it, that use case is currently not supported in ZFS, hope they will eventually add that functionality…
From what I remember, removal is not really impossible for RAIDZ. But since device removal works in context of allocated space and does not verify checksums in process, it has to copy all the data copies as they are, preserving vdev topology. So removal from mirror to raidz is indeed impossible, while from raidz to identical raidz IIRC should be possible, just was never implemented. The original motivation of device removal was to remove single disks added as vdev instead of attaching to existing vdevs, in which case they are not raidz, and likely don’t match other vdevs topology.
This works with a stripe of mirrors, where one could extend the single drive to go back to safety. But it doesn’t work when mistakenly striping a single drive to a raidz instead of extending the raidz, does it?
Right. Copying one set of data identically into two for mirror is easy, but to RAIDZ I guess impossible, since we don’t know where each block starts to generate proper parity.
So device removal is feature which may be occsionally useful for shrinking a mirror stripe, but it does not solve what the most damaging, and possibly the most common, case of mistakenly attaching a single drive.
If TrueNAS automatically created a pool checkpoint prior to adding a new vdev or device to an existing vdev, it could save a lot of people from permanently condemning themselves.
Special devices can be any type, including raidz2 or draid. They just have to match the redundancy level of the pool. You can verify the behavior by making a pool of sparse files:
$ truncate -s 100M file{1..20}
$ sudo ./zpool create tank draid ./file{1..10} special draid ./file{11..20}
$
Anyone know if/when we will see RaidZ2 option for special vdev in the nightly builds, or when beta is coming that has it in there for next version 26.04 halfmoon? Thought there was mention of mid Feb 2026 for beta/moving from nightly.
Really keen on having this option for building out a storage server to be able to combine Metadata, SLOG and Small blocks onto a 4 x enterprise NVMe raidZ2 setup to assist a 12 x 22TB (2 x 6wide vdev) SAS hdd pool.
