One data pool (1 x 4-wide RAIDZ1) with Samsung PM883 1.92TB SSDs
16GB RAM
I still consider myself a TrueNAS newbie. I’m looking to expand with 4 more equal-sized SSDs (and more RAM).
Thinking of 3 options:
Add second VDEV 4-wide RAIDZ1 to the existing pool
New pool with 4-wide RAIDZ2 (using a sparse file during migration), then migrate data and existing SSDs ending with 1 x 8-wide RAIDZ2
Same as option 2 but using RAIDZ3
I don’t like option 1, but it is the simplest migration path. I’m leaning towards option 2. Is there a compelling reason to choose option 3, or is option 2 a good performance and redundancy compromise? The storage supports VMs running on xcp-ng with a bias towards reads over writes. All data are backed up regularly to cloud storage.
To migrate existing ZVOLs (iSCSI MPIO to separate VLANs/subnets) and datasets I assume local replication is the answer, but any pointers to a procedure including how to rename the new pool to the original at the end (or is that not necessary)? Any “gotchas” to be aware of during the migration?
RAIDZx isn’t optimal for your use case. You would be better served with n-wide stripes of 3-drive mirror VDEVs, as this would result in better redundancy (you can lose 2x drives from any VDEV and still be able to resilver back to good) and better throughput due to the striping and lack of RAIDZx overhead.
In your case, if your max number of drives is 8, I would recommend a 2-wide stripe of 3-drive mirror VDEVs with 2 spares.
There’s no easy way to migrate from your current pool layout to the above. You’ll need to backup everything, destroy and recreate your pool, and then restore from backup.
Create new pool as 2-wide stripe of 2-way mirrors.
Migrate data.
Remove old pool and rename new pool.
Expand mirrors to 3-wide.
This, however, sacrifies space for double redundancy and I suppose that OP wanted to increase storage space. Maybe a 4-wide stripe of 2-way mirrors is acceptable?
@etorix, yes, I do wish to expand storage space. A 4-wide stripe of 2-way mirrors is an intriguing option with 50% storage efficiency, but providing improved resiliency and performance. It is also one that would be easier to achieve in the manner you describe!
Thank you both for the ideas – and the pointer to how to rename a pool!
When starting a new pool with 2 x 2-wide mirrors, replicating datasets, renaming the pool and finally expanding to 4 x 2-wide mirrors, will ZFS auto-balance data over time or just newly-written data?
Since I also have one remaining NVMe slot, can an NVMe drive act as a hot spare for VDEVs comprised solely of SATA SSDs, or do hot spares have to be of the same type, making it necessary to keep a SATA SSD onhand and manually substitute in the event of a drive failure?
For the first question, ZFS is copy on write. Blocks which aren’t written to stay in their original stripe. If you want to rebalance over the new topology, snapshot your pre-expansion datasets, and then promote the snapshots. The originals will become snapshots of the newly promoted clones. You can then delete the demoted snapshots.
For the second question, spares don’t need to be identical, although it’s strongly encouraged. They just need to be as large as the smallest disk in your data VDEV.
If you want to rebalance over the new topology, snapshot your pre-expansion datasets, and then promote the snapshots. The originals will become snapshots of the newly promoted clones. You can then delete the demoted snapshots.
Just so I’m clear (since I’ve never done this before and not sure what “promote the snapshots” means), the e2e process is then:
Create a new pool B with 2 x 2-wide mirrors
Use local replication for all datasets and zVols from pool A to pool B
Rename pool B to pool A (to include moving the system-dataset to, in my case, the boot-pool).
Snapshot all datasets and zVols on the new pool A
Expand the new pool (now A) to 4 x 2-wide mirrors (create two new VDEVs from the drives in the original pool A)
“Clone to new dataset” from the snapshots in (4) – this forces rewriting the data and thus balances across all 4 VDEVs
Delete the original snapshots created in (4)
Do I have it correctly? Does step 6 retain the same dataset names or is there something to watch out for?
I’m not sure if the UI function Clone to new dataset will do what you want. I was thinking more of the CLI command zfs promote. See zfs-promote.8 — OpenZFS documentation.
And in your case, step 2 replication isn’t needed. You’ll need to perform a zfs clone from A to B.
If you have backup (or do not fear the moment where there’s no redundancy), you can create the new pool as a stripe of four drives, move data and then attach a second drive to each member to arrive at a stripe of mirrors. No rebalancing.
If you have backup (or do not fear the moment where there’s no redundancy), you can create the new pool as a stripe of four drives, move data and then attach a second drive to each member to arrive at a stripe of mirrors. No rebalancing.
