Hi I’ve got a SCALE with a four wide RAIDZ1 pool that has a metadata VDEV that consists of a pair of NVMe devices (500GB). At some point in time the metadata VDEV will grow full at which point I plan to replace the NVMe’s one at a time to a set of 2TB instead in the hope that the VDEV can the be expanded to the new capacity lige the RAIDZ1 VDEV.
Indeed; The special VDEV is a mirror, so it does have the same fault tolerance as the other storage VDEV.
I acknowledge the downside of once you’ve added a special VDEV to a pool you can’t remove it. I’m also aware that the system will be “fragile” while the special VDEV resilvers, but it’s a home system with all really important stuff backup up remotely.
To avoid that, you can replace a drive while keeping the old device connected. If you’re short of M.2 and PCIe slots, use a USB adapter. Redundancy through a flaky link beats no redundancy.
That’s a good point. While shopping I could get one of those PCI cards with two NVMe slots. I’m using the two on the motherboard, but one of them is x4, three other only x2.
This adapter holds one x4 M.2 NVMe drive in a PCIe x4 slot, and one M.2 SATA drive, for which you have to provide a cable to a SATA port on the motherboard (power is taken from the PCIe slot).
Good if you have the right assortment of drives. Not good if you have a pair of NVMe drives.
If you want to use NVMe drives, and the motherboard will allow you to bifurcate a x16 slot down to (x4/x4/x4/x4) then this is the sort of card you would need:
For the avoidance of doubt, the card you purchased does not involve bifurcation at all, and should indeed be “plug-and-play”—just not for the NVMe drives you have.
I use this personally in an ASRock X570M Pro4 with x4/x4/x4/x4 bifurcation and it works well. I couldn’t have fit the x3 mirror of Special VDEV drives without it. It seems like sometimes difficult to determine if your board supports bifurcation as some companies can slack in the manual, but hopefully it does list it. Some only allow x8/x8 and not x4/x4/x4/x4 so watch out for that too.
I’ll try to summarize my key findings from this thread.
Yes it is possible to expand a special VDEV once all storage devices have been replaced.
It is possible to add a third device to a mirror VDEV in order to replace devices without compromising redundancy
It is possible to have multiple NVMe devices on a single PCIe card, if your motherboard supports PCI lane bifurcation.
It is possible to add a PCIe card with a single NVMe device with bifurcation
My conclusion:
Since I have a fairly old motherboard (Kaby Lake) PCI lane bifurcation is out of the question and not really needed. The motherboard has two PCIe 3.0 NVMe slots (1 x4 and 1 x2), that I currently use. Adding a third NVMe device during migration can be done either via a USB connection or via an PCIe adapter like Startech x4 M.2 NVMe adapter. Even though the special VDEV will the effectively operate at x2 speed, that shouldn’t impose a significant bottleneck, as the system has a single 10G NIC and is for all practical purposes a single user system.
However since the other x4 slot is occupied by the NIC, I’ll stick with using a USB3-NVMe enclosure during the migration.
In order not to risk simultaneous failure of the two devices, I’ll repurpose the two 1TB SSD’s currently in my desktop. One is a 980 pro, the other a 990 Pro.
Once again, thanks for all the constructive feedback and advice given so far!