i built my first NAS earlier this year. it’s running TrueNAS scale, and currently has 2 12 tb drives in a mirror for storage. when i built it, i knew i would need more storage eventually, so my plan was to use ZFS expansion to add more storage once it was roughly 80% full.
it hasn’t hit that point yet, and probably won’t for another year or two. however, i live in the USA, and recent events mean electronics could become much more expensive for me in the near future. so, i decided to buy two more of those same 12 tb drives, so i’d have them when i need them.
and now i need to figure out what to do with them in the meantime.
my first thought was to just test to make sure they’re not defective, and then put them back in their boxes until my current mirror is 80% full. is that reasonable, or is there a good reason to add them earlier?
also, i’ve read that RAIDZ1 is dangerous with drives this large, due to the risk of data loss during re-silvering. Is that true? if so, would it be safe to convert my current pool to RAIDZ2 instead when i add two more drives? or should i just stick with the mirror? i know 4x12 tb drives in a mirror would have the same storage capacity as they would in RAIDZ2, but being RAIDZ2 instead would make adding more drives in the future easier.
Good idea - never hurts to run some badblocks & smart longs to make sure that drives don’t already suck; as always caution warning caution danger, upon reboot sda/sdb/etc can randomly change. If running badblocks in a production system always triple check & compare sda/sdb/etc to the drive SN# before executing the commands. Sometimes easier to run outside of the NAS for sanity & data safety.
Do it well before hitting 80%.
Yes. Resilver stresses the drives. Larger the drives are, longer the resilver, more stress, more chance of things going wrong critically.
Unless you have somewhere to copy your existing data, I’m not sure how you’re thinking of converting.
There is a risky way of doing it of course; manually creating a degraded 4 drive raid z2 pool (it only has 3 drives active, but wants 4 - this’ll require degrading your mirror to 1 drive only as well), copying over the data to the degraded pool, then adding the original drives from your mirror to the raidz2 pool & re-silvering. Now your degraded pool is happy with 4 drives, and your mirror is gone. Not for the faint of heart.
Generally going from RAIDZ1 to RAIDZ2/3 (or RAIDZ2 to 3) is a full start from scratch as far as your pool & data goes.
Since your option to add more volume to your original two disk mirror is add another VDEV of a second mirror pair.
two drives mirror + two drives mirror = POOL
Back up all your data. Recreate your pool with all four drives in Raid-Z2 VDEV
Both options give you the same amount of space. The Z2 configuration is less risky data wise as you can lose ANY two drives and the pool is okay, just degraded.
If you lose two disks in the same VDEV on the mirrors setup, the pool is gone.
You can use ZFS expansion to add another disk to the Z2 setup for 5 disk but the mirrror setup requires adding a VDEV with a 2 disk mirror again.
i was planning to test the drives on my desktop, not the NAS.
i picked 80% because trueNAS Scale has an option to send warnings once a pool is that full. If you think is should add more storage “well before” then, could you give me a different number? 70%? 50%?
maybe i misunderstood, but i thought it was possible to convert a 2 drive mirror to a different topology when adding additional drives. after all, a mirror and raid z1 are effectively the same thing with only two drives.
The mirror setup could have the advantage in the future when larger drives are available. If you got two 24GB drives, you could replace a mirror pair that has 12GB drives, one by one, allowing for resilvering, and get more pool space.
Raid-Z(1,2,3) requires replacing all the drives if you can’t add drives to the pool. For example you would go from 4x 12GB to 4x 24GB. Replacing a single drive at a time, allowing for resilvering, until all drives were replaced.
You can’t convert between mirror, z1, z2 or z3. Pool destruction is required
Sorry, given we’re talking about 12tb I meant ‘well before’ as in space used, not % space used. I’d go for 70% max personally. Earlier means faster resilver/data transfer due to… well less total data.
Sadly you did misunderstand. Somewhere there’ll be posts explaining better than I how data is split between disks to actually build redundancy & why mirror to raidz1/2/3 or raidz1 to 2/3 can’t be done without destroying data.
Your options to resolve this are:
Looking at more mirrors
Copying data somewhere else (likely buying another drive to do so) & creating a raidz2 pool & moving data back
Doing some sketchy stuff so you don’t have to copy anything anywhere (or buy anythign else) else but end up with raidz1/raidz2 pool with future raidz expansion!
Raidz2 with your drive size is the safer way to go & as SmallBarky mentioned, will give you options to expand in a more sane way down the line if you’re on latest TrueNAS version (raidz expansion will let you add additional disks 1 at a time to your raidz1/2/3 pools).
okay, thank you both.i don’t have enough storage space elsewhere to back up the current mirror and build a new proper 4 drive raidz2 pool. before you judge my data backup practices, this nas is just for personal use, and all of the truly life critical stuff on it is backed up elsewhere.
i’m leaning towards adding another mirror VDEV to the pool., since i’m an amateur, and it seems the least risky. plus, while my current pool is roughly 50% full, i expect the rate i’m adding data to it to slow down a lot in the near future, so it’ll likely be a long time before i need even more storage.
however, i just realized there is a way i could get to raid z2 without buying more storage or building a “degraded pool.”
copy all my current data to one of the new drives
create a new 3 drive Raid Z2 pool from the other new drive + both existing drives
copy the data backed up to the first drive to that Z2 pool
then expand the Z2 pool by adding the first drive to it.
does that sound right? and is it reasonably safe?
edit - if it matters, i have about 5.5 tb of data in the mirror right now.
I would stick with just the mirror pairs. I don’t know if you can do the 3 drive Raid-Z2 in the GUI. It might require command line work and if your single data disk goes down in all of this, there goes all your data.
If you could have your current data backed up in two separate places and then convert, I would say go for Raid-Z2, but current state, stick with mirror.
Sounds reasonably safe to me. Just make sure to test the drives first.
My plan is from the old days before raidz expansion - so I forget that this would be a valid path forward, yet still remember to advise of the benefits of raidz expansion.
The way you listed would be good enough for my personal risk tolerances & much less involved then how it used to be done!
huh, seems weird that z2 requires 4 drives. a pool with 3 drives all mirroring each other seems like it could make sense for some situations.
i’ve been daily driving linux on desktop for about a year now. i’ve done plenty of basic command line stuff there, and some in truenas as well. is there a guide on how to create datasets via command line?
thanks. there wasn’t anything like this in the truenas docs, and i didn’t think to google around for zfs stuff elsewhere.
this is definitely on the edges of my command line proficiency. i’ll need to think about it.
in the meantime, if i were to do this, what would be the proper way to do step 1? after hooking up the drive and creating a single disk pool for it, is there a proper way to copy everything over and then test that it worked in the GUI? or would it be better to just do it in the command line with rsync?
If you’re considering manual CLI, my opinion is you’ll have to commit fully to CLI.
I’m not sure what you mean by ‘proper way to do step 1’ or hooking up drives, so I’ll just vomit the steps at you & hope they answer your question.
Keep in mind, if you’re following the guide for your purposes, you have to keep in mind that you’re making some minor very minor tweeks.
You have 2 disks in a mirror & 2 new disks. You’re going to break the mirror, you’re going to create (basically) a fake disk, you’re going to create a raidz2 pool with 3 disks & a fake disk (2 new, 1 recently detached from mirror), you’re doing to replicate from the now degraded mirror to the newly created pool, you’re going to destroy the degraded mirror, you’re going to replace the fake drive with the newly freed disk. Success.
Expect GUI to not enjoy this process.
I can’t think of a way to do a dry or test run unless you got some old disks lying around that you can mess with.
If you don’t feel comfortable, it might be worth considering buying a 5th drive, copying the pool to drive#5, creating a raidz2 pool with 4 drives, copying everything over, then expanding & have a 5 drive raidz2 pool. At least that can all be done with the GUI.
yeah, i am thinking about getting a 5th drive for the NAS, or even just 6 tb of other storage i could back up the NAS’s current contents to temporarily so i could create a 4 drive raid z2 pool.
raid z2 definitely seems like the best option for me long term. switching to it is tricky though, and it will get trickier over time as i get more data. it’s just of question of whether i bite the bullet and do it now, or if i kick the can down the road and settle for doubling the mirror.
Can you use the Best Buy deal? Some people get these for the drive inside, shuck them and use them in a NAS. It would be useful for your temp use and then …
wow, that’s quite the discount. NGL, it’s a lot more tempting that buying a 5th NAS drive right now, since i’m so far away from actually needing a 5th drive in the nas.
The problem with this is that at some point you will have your data only on one drive and if that drive fails, your data is gone.
Here is an alternative approach:
Add 2 new disks.
Create a RAIDZ2 pool with those two disks and a dummy file. Offline and remove the dummy file and you have a degraded RAIDZ2 pool equivalent to RAIDZ1.
Replicate your data from the old pool to the new one.
Remove one mirror from the old pool, and add it to the RAIDZ2 pool to resilver.
Use RAIDZ2 expansion to add the second drive from the old pool to the new pool.
This is technically more complicated, but at no point do you have data only on a single drive.
i can see how that method is safer - since the data is always on 2 disks. could you elaborate on “2. Create a RAIDZ2 pool with those two disks and a dummy file” though? Per SmallBarky’s post above, the GUI won’t allow for a 3 disk RaidZ2 pool, so i assume it wouldn’t allow for a 2 disk one either.
i know creating a degraded Raid Z2 pool is possible in the command line, but given my skill level, any method that requires using rhe command line has an elevated risk of data loss.
