Hi All,
Just wanted to confirm this can be done without repercussions. I currently have a storinator that can hold 30 drives and 10 are filled. I have a raidz1 configured on Truenas Scale 25.10.3 - Goldeye and I am reading that now expanding the vdev pool is possible.
Is it really as simple as just importing new disks to the pool and that’s it, expansion begins? I will likely have to add more than 10 disks to the existing pool.
Any guidance is appreciated.
A 10 wide Raid-Z1 already? I don’t think anyone would recommend adding drives to a single VDEV. Adding one or move VDEVs to the pool would be the way. What size drives as Raid-Z1 isn’t recommended with large drives.
Currently the existing pool is 10x10TB drives and we are running out of space.
If expanding the existing pool with individual disks is not an option, whats the next best alternative?
Add another 10 disks in a new raidz1 vdev to the pool
Raid-Z2 or Raid-Z3 VDEVs would be recommended over Raid-Z1 for those drive sizes. The risk of losing the pool during a resilver is the reason. Consider backing up and restoring the data to a new pool configuration.
What are the goals for your storage and do you have a separate backup of all the data?
As an example, you are stuck adding a second VDEV, 10 wide of Raid-Z1 to your current setup as LarsR says. You are also stuck with 10 resilvers per VDEV if you were to replace the 10TB drives with something like 20TB drives to gain more space.
Smaller width of VDEVs could help for flexibility and adding space. You have space for 30 drives? so I would guess and end goal could be something like Raid-Z2 with 5 to 8, maybe 9 drives per VDEV. This would allow two disks to fail in a VDEV without losing the entire pool.
Adding pool layout whitepaper link. This may be a bit older but it was shown as static.
We just need a larger array now that the current pool is almost full. We are at 95% capacity at the moment.
Just create another pool but raidz2 for the new set of drives and just add the new pool to the existing one for expansion?
we are doing an rsync of the data to another location that is off site but that will also need an expansion to accommodate for the increase of data too.
You are really stuck. I think the only option is to create a new pool with enough space to get all your data to move and keep the new pool below 80% usage before verifying and removing the pool with the 10 wide Raid-Z1 layout. If the hardware is about the same it would be something like this.
current_pool is 10x 10TB drives in Raid-Z1 @95% capacity
You create a new_pool with 2 VDEVs of Raid-Z2.
new_pool is VDEV1 (8x 10TB Raid-Z2) & VDEV2 (8x 10TB Raid-Z2). you might be able to do wider but I don’t know what the recommendations on VDEV width were per layout. Hoping others will post a link or something.
You move the data from current_pool to new_pool and verify everything. Once everything is verified, you can decommission the original_pool and reuse those drives in one of the new pool / VDEV layouts on one of your servers.
Posting as much details on what hardware you have to work with and any service window time may help with advice from others.
I don’t know how much of a budget we can get but we will try and aim for another 20 drives if possible.
We have old hardware running this at the moment:
Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
64GB of RAM
with a total of 30 bays and about 20 available
It came from 45 Drives so there should be about 2 LSI cards in there as well. With regards to service window, I don’t know if we will get one any time soon even after purchasing these drives (assuming the green light is given for this purchase).
But why do a new vdev and move the data over?
Four 7-wide raidz2 vdevs and two spares would nicely fill these slots.
Three 10-wide is possible… but then raidz3 over raidz2 for safety. (If safety is not a concern—why ZFS then?— you may try and expand towards 30-wide raidz1… but do not expect to recover when disaster will struck.)
You’re also in territory where dRAID could be considered, but you’d need 30 drives in one go: There is no dRAID expansion—and no way to convert your current raidz1 into dRAID.
Mind the terminology here!
A vdev is a group of disks. (raidz1 ↔ RAID5)
A pool is a group of vdevs. (stripe of raidz1 ↔ RAID50)
If you do not fancy the idea of restoring everything from the remote backup, we suggest creating a new pool with better redundancy, as outlined above by @SmallBarky.
You do NOT want to add a raidz2 vdev in the same pool as your current raidz1 vdev, as you’d be stuck with this configuration forever.
How many drives of which capacity can you afford right now? Drives in a vdev are preferably of the same size, but drives need not be of the same capacity across vdevs
I see, don’t mix vdevs. This is noted.
We are hoping to pitch for another 20 drives of about 10TB each but the pricing of all that might not sit well with our CEO (Thank you AI).
We really just need to minimize the time it takes for the extra capacity to exist that’s why I was looking at just expanding but if this isn’t recommended then we don’t have much of a choice in the matter. Sounds like making 2 new vdev’s and moving the data over to the new one is inevitable.
You might expand your raidz1 one drive at time… but you’d be going deeper into brown stuff.
You may add a second vdev of 10 drives in raidz1 (or even less drives, and expand later until you reach 10). And later add a third 10-wide raidz1. But the day one drive fails you’ll be heading for trouble: Resilver stresses the drives, and wider vdevs makes resilver longer—if one of the 9 survivors does not take it well, the pool is lost. (And if you go for a single 30-wide raidz1 it will be even worse.)
The safer way we advise is to build a new raidz2 (or raidz3) pool. This does not need to be 10 TB drives: If you can find, say 20 TB drives, “just” 7 are enough to make a 7-wide raidz2 (and pool) which can (just) hold your ca. 85 TB of data. Replicate from old pool to new pool, destroy oldpool, and you can reuse 7 of the 10 TB drives from old pool to make a second raidz2 vdev which you can add to new pool (imbalanced capacity but consistent geometry): You then have a pool with 150 TB of raw space—plus 3 “free” 10 TB drives. Your expansion paths are then buying other sets of 7 drives (plus some spares) to add vdevs and end up with 4 vdevs, and replacing all drives in an existing vdev by larger ones.
If you can get 14 (or 15) drives of 12-14 TB, you can make newpool as a stripe of two 7-wide raidz2 (plus spare), replicate and reuse to arrive at a stripe of 7-wide raidz2, one of which is 10 TB and the other are slighly larger. As an additional benefit over the above scenario, you need not consider a rewrite/rebalance to redistribute the data: All the old data is in only two of the three vdevs, but free space is reasonably distributed across, so new data will naturally spread across the whole pool.
Adapt for other sizes, other number of drives, or other target geometries than my suggestion of 4*7z2+2s. The key is that you need to decide upfront on what the endgame looks like: You can add vdevs, you can expand (widen) raidz vdevs, but you can not remove raidz vdevs, you can not shrink raidz (reduce width) and you can not increase raidz level.
I will see if I can pitch for 15 new 16TB drives instead of the 10TB, seems like creating a new vdev and moving the data is inevitable.
If I get 15, keeping that last one as a cold spare is the best route right?
14-wide is not advisable, so it seems that you’re aiming for two 7-wide vdevs.
In that case it is indeed advisable to keep the 15th as spare, cold or hot as you prefer. It seems that in the current market you cannot rely on getting a RMA in time—or even getting a RMA at all.
(No obligation to follow my suggestion for 4*7z2+2s: I just tried to find a good divider for 30 bays.)
Correct, 2 7-wide vdevs is what I am aiming for with the 15th being a spare drive. At least then I can combine those two vdevs into a larger pool and have greater capacity.
The existing vdev of the 10-wide array, if I rebuilt that once the data is moved I can recreate it as a 7-wide array and add that to the new pool I create from the new drives?
Yes, you would be able to add a 7 wide 10TB VDEV to the pool and end up with a three, 7 wide VDEVs in a single pool.
