Can a metadata VDEVs expand when disks are replaced

TrueNAS General
1

Hi I’ve got a SCALE with a four wide RAIDZ1 pool that has a metadata VDEV that consists of a pair of NVMe devices (500GB). At some point in time the metadata VDEV will grow full at which point I plan to replace the NVMe’s one at a time to a set of 2TB instead in the hope that the VDEV can the be expanded to the new capacity lige the RAIDZ1 VDEV.

Is this a viable path?

NAME                 SIZE  ALLOC   FREE  FRAG    CAP  DEDUP    HEALTH  ALTROOT
SpinningRust        44.1T  12.6T  31.5T    6%    28%  1.00x    ONLINE  /mnt
  raidz1-0          43.6T  12.3T  31.4T    6%  28.1%      -    ONLINE
special                 -      -      -     -      -      -      -         -
  mirror-2           464G   352G   112G   56%  75.8%      -    ONLINE
spare                   -      -      -     -      -      -      -         -
2

Yes.

Edit: I misread at first and thought you had a single device special VDEV, you do in fact have a mirror.

3

Indeed; The special VDEV is a mirror, so it does have the same fault tolerance as the other storage VDEV.
I acknowledge the downside of once you’ve added a special VDEV to a pool you can’t remove it. I’m also aware that the system will be “fragile” while the special VDEV resilvers, but it’s a home system with all really important stuff backup up remotely.

1 Like
4

To avoid that, you can replace a drive while keeping the old device connected. If you’re short of M.2 and PCIe slots, use a USB adapter. Redundancy through a flaky link beats no redundancy.

1 Like
5

That’s a good point. While shopping I could get one of those PCI cards with two NVMe slots. I’m using the two on the motherboard, but one of them is x4, three other only x2.

6

That won’t work without either suitable bifurcation from the motherboard or a PCIe switch (more expensive).

7

I’ve got a suitable PCIe free for a card like this. DIGITUS M.2 NGFF / NVMe SSD PCI Express 3.0 (x4) Add-On Card

I suppose that should work out of the box, or am I missing something?

8

This adapter holds one x4 M.2 NVMe drive in a PCIe x4 slot, and one M.2 SATA drive, for which you have to provide a cable to a SATA port on the motherboard (power is taken from the PCIe slot).
Good if you have the right assortment of drives. Not good if you have a pair of NVMe drives.

9

If you want to use NVMe drives, and the motherboard will allow you to bifurcate a x16 slot down to (x4/x4/x4/x4) then this is the sort of card you would need:

https://uk.store.asus.com/10418-90mc06p0-m0eay0.html

10

Prior to this thread I had never encountered the word “bifurcate”. And most certainly didn’t expect it to be connected with PCI lanes.

I thought that add-in cards was a question of plug and play besides physical compatibility since I bought a PC with EISA bus back in the days.

So thanks for enlightening me to the world of PCI lane bifurcation. And what a twist this thread took. I love this community🎉

1 Like
11

For the avoidance of doubt, the card you purchased does not involve bifurcation at all, and should indeed be “plug-and-play”—just not for the NVMe drives you have.

12

I haven’t purchased any cards yet. But I realize that I need to plan well ahead long before my special VDEV becomes full.

Once again thanks for your insights🤗

13

I use this personally in an ASRock X570M Pro4 with x4/x4/x4/x4 bifurcation and it works well. I couldn’t have fit the x3 mirror of Special VDEV drives without it. It seems like sometimes difficult to determine if your board supports bifurcation as some companies can slack in the manual, but hopefully it does list it. Some only allow x8/x8 and not x4/x4/x4/x4 so watch out for that too.

1 Like
14

I too use the Asus Hypercard and was quite impressed with its build quality.

It is big though!

There are also 8x4x4 riser cards which allow using two NVMEs AND a low profile x8 card, such as an HBA or small GPU.

15

I’ll try to summarize my key findings from this thread.

  • Yes it is possible to expand a special VDEV once all storage devices have been replaced.
  • It is possible to add a third device to a mirror VDEV in order to replace devices without compromising redundancy
  • It is possible to have multiple NVMe devices on a single PCIe card, if your motherboard supports PCI lane bifurcation.
  • It is possible to add a PCIe card with a single NVMe device with bifurcation

My conclusion:
Since I have a fairly old motherboard (Kaby Lake) PCI lane bifurcation is out of the question and not really needed. The motherboard has two PCIe 3.0 NVMe slots (1 x4 and 1 x2), that I currently use. Adding a third NVMe device during migration can be done either via a USB connection or via an PCIe adapter like Startech x4 M.2 NVMe adapter. Even though the special VDEV will the effectively operate at x2 speed, that shouldn’t impose a significant bottleneck, as the system has a single 10G NIC and is for all practical purposes a single user system.

However since the other x4 slot is occupied by the NIC, I’ll stick with using a USB3-NVMe enclosure during the migration.

In order not to risk simultaneous failure of the two devices, I’ll repurpose the two 1TB SSD’s currently in my desktop. One is a 980 pro, the other a 990 Pro.

Once again, thanks for all the constructive feedback and advice given so far!

16

Given enough slots, it’s also possible to plainly add a third drive and turn a 2-way mirror into a 3-way mirror.

You KabyLake motherboard could have x8x4x4 bifurcation on its x16 slot, which could be handy for a pair of NVMe drives and a half-height HBA/NIC/GPU.