While adding a new vdev and pool to my TrueNAS and re-organizing files to spread them out across the new storage space ( I just hit 80% usage on a pool ), I learned / realized that you can add vdevs to an existing pool. Mind blown.
I currently have 3 raidz1 vdevs, each in a separate pool
vdev 1 = 6 x 18 TB Hard Disk Drive = zpool 1
vdev 2 = 4 x 12 TB Hard Disk Drive = zpool 2
vdev 3 = 4 x 10 TB Hard Disk Drive = zpool 3. Newly added of yesterday
Luckily, I’ve only rsynced files from zpool 2 to zpool 3. I haven’t deleted anything from 2.
Is there an issue if I:
delete zpool 3
assign vdev3 to zpool 1
move files from zpool 2 to the newly expanded zpool 1
Delete zpool 2
assign vdev 2 to zpool 1.
That would make file management a lot easier. I’ve been struggling with how to balance television shows and movie files across 3 pools. If I can have 1 pool and let TrueNAS manage the space, my life just got a lot easier.
The only potential con I see is when I replace the smaller drives with larger ones after Hard Disk Drive prices drop ( or I win the lottery ) .
Is there a process to remove vdevs from a pool? Lets say I buy 10 x 30 TB drives to replace my existing drives in vdev 1 and vdev 2. And, I want them all in a single vdev. I could upgrade and expand vdev 1 in place, but due to physical limitations ( my server has 15 drive bays ) at least one of the other vdevs would have to be removed. Is that possible without losing data?
Now would be the time to rethink your Raid-Z1 layouts and your VDEVs. Raid-Z1 isn’t recommended with those large drives. It is due to time to resilver when a drive fails and the risk of loosing the entire pool.
Existing data is not rebalanced, and ZFS allocates new writes proportionally to available space so your files are not evenly distributed and you do not get the full performance of a perfectly uniform pool.
With raidz you’re stuck with the number of drives and vdevs you’ve commited to the pool. Cannot remove vdevs. Cannot reduce width.
As @SmallBarky already pointed out, raidz1 is not that safe, and I assume you do not have a backup. Now would be the time to think about your storage requirements and a safer layout.
You can have 6-wide raidz2 (6*18 TB) + 6-wide raidz2 (4*12+2*10) for 4*18+4*10=112 TB of raw space (best use only about 80% of that, etc…). The question is how you get there moving your data.
You create a new pool and replicate to it. Use a temporary enclosure or another server.
This is the best answer so far, so I’ll mark it as the solution.
I recently learned that you can rebalance data across vdevs with the zfs-rewrite utility , which I’m going to do after I move all the data to my expanded pool.
Luckily for me, I have an offsite backup of data that would be difficult to replace. None of it is impossible to replace. The difference is in the level of effort to replace the data.
Good point on using an external server. If drive prices ever go back down, or I decided to pay almost twice what I intended pay when I planned my next upgrade out, I decided that using an external server to at least power the drives in the new pool will work.
My planned upgrade is 10 drives, 28 TB or higher each. I would add another 16 port HBA card to my TN server , attach the drives, create a new pool , and migrate. My current server’s PSU is out of SATA drive power ports so I would need another source. Luckily again, I have a spare server racked right next to my current one that can supply the power! The spare server was a TN server back when I ran 2 of them. It could pull the job again if I power it back on, but I don’t need 2 of them atm.
