Hello guys,
So, i’m having a NAS which has 6x16TB HDDs and i’ve got 8 more and want to add them to my existing. I’m really not sure how i did i added two more an year ago as i had 4 disk originally.
Can i add the new drives to the existing pool without destroying the pool?
Thanks
What is the output of zpool status? I assume your pool is made of 3x 2-way mirrors.
Will provide the output in a while. There were only 5 disks configured in a Pool #1 and Pool #2 has a single disk as it does not have any important data.
Unfortunately, when i built this, i had no idea about RAID-Z1 that its too dangerous and as i didn’t have any high performance NIC, and wanted maximum speed at that time, so i went for maximum capacity and after testing various RAID-X layouts, i finalized Z-1 and it worked great for like 2yrs and a couple of months ago, a disk in that pool started having bad sectors all of sudden so, want to add more disks to the pool and then resilver. Or should i first move the data to any other NAS and simply destroy the old pool and add the new disks and create new pool?
Plus, is it really possible to add more disks to an existing pool, provided that the disks are of same capacity, model, firmware and brand.
Also, i’ve so much confusion in understanding 2 way, 3way, or say X way mirrors. Can you please help me to understand this?
A X-way mirror is a mirror VDEV composed of X drives.
Suggested reading: ZFS Storage Pool Layout - Resources - TrueNAS Community Forums
Before answering your other questions I have to understand your pool layout, but generally you cannot remove, enlarge, or reduce a RAIDZ VDEV.
Still i get confused when people say X way mirror or X wide.
Sorry to say but the link does not work ;(
Allow me some time to reach my place and then i can provide you the output.
Also, till today, i had this single drive failure, which first started giving bad sectors and then those bad sectors started increasing and one day, the drive motor stopped ;(
Any idea what could have caused that so that i can have some precaution next time?
It means the number of drives in a VDEV: 4-wide RAIDZ1 means 4 drives in a RAIDZ1 VDEV; a 3-way mirror means 3, drives in a mirror VDEV.
Check the first comment, the bot broke the resource by porting it.
Age. Without looking at the smart data your guess is as good as mine.
Now, its clear but need more practice 
Okay!
Well, well. The drive was purchased in 2022 in month of June and then the total data written was like 10TB approx and also, the NAS was not on for 24x7. Only, when i had to write data to it. So, say like in a month, i use to write 3-4 (max) otherwise say a single day. I might have some disk info from the HD Sentinel on Windows. Let me find and attach it here.
All of these drives were Seagate EXOS X16. Even enterprise drives have this much quick failure rate?
I’m clueless why the bad sector appeared all of sudden one day.
Looks like a faulty drive for sure, the warranty should cover this.
You could try a badblock run to fix the sectors. Did you burn in the drive?
Yes, but like i mentioned, it was all good for months and then a couple of months ago, probably, in March, when i started the NAS, i saw errors, to confirm, i booted into Windows i was quite sad as the drive had bad sectors, then these bad sector count increased and then one day when i powered on my NAS, the pool status was degraded and when i diagnosed, i found that the drive is dead (motor was not working, hence the platter was not spinning), no movement, no noise, nothing, like dead drive ;(
What does that mean and does it really fixes the bad sectors and then drive can be used further, with reliability?
You mean stress test?
Drives fail. If it’s in warranty, and if bought new it should be, then return it for another.
You can replace a failing drive, with any drive as long as it is the same or larger.
Do not use SMR drives.
You can add additional Vdevs to your pool to increase size, or replace all the drives in a VDev with larger drives to increase the size of your pool.
A VDev is a virtual device. And is like a virtual drive made of one or more drives using either mirroring or a raidz technique.
I think you should convert to raidz2, but I’d suggest taking some time to figure out what you should do first, and let us help you do that.
Can you run the zpool status command and paste the results inside triple backticks ```
@Davvo @Stux Here is the zpool status
root@Storage[~]# zpool status
pool: Pool 1
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using zpool online’ or replace the device with
‘zpool replace’.
scan: scrub repaired 0B in 07:14:16 with 0 errors on Tue Oct 17 13:12:27 2023
config:NAME STATE READ WRITE CKSUM Pool 1 DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 gptid/8c112XXX-c934-11ec-918c-7c10c967afed REMOVED 0 0 0 gptid/8c25XXX-c934-11ec-918c-7c10c967afed ONLINE 0 0 0 gptid/8c44XXX-c934-11ec-918c-7c10c967afed ONLINE 0 0 0 gptid/8c33XXX-c934-11ec-918c-7c10c967afed ONLINE 0 0 0 gptid/8c6eXXX-c934-11ec-918c-7c10c967afed ONLINE 0 0 0errors: No known data errors
pool: Pool 2
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using ‘zpool upgrade’. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:NAME STATE READ WRITE CKSUM Pool 2 ONLINE 0 0 0 gptid/86cfcXXX-ac30-11ec-a4b2-7c10c967afed ONLINE 0 0 0errors: No known data errors
pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:02 with 0 errors on Mon Jun 17 03:45:02 2024
config:
I see two options for now:
Option 1: Add the new disks i have and create a new pool with those disks and copy the data from the Pool 1 to this New Pool X. And if possible, destroy the pool 1 after verifying the data is moved,add those disks from Pool 1 to the new Pool X.
Option 2: Copy the data from the Pool 1 to a different NAS and verify the data after moving. Once moved, destroy the old pool and add the new disks to the NAS. Create a new pool with the aim for maximum performance+redundancy and then copy that data back to the new pool created with 16 disks.
Let me know what do you guys suggest.
@Davvo Any info on this friend?
First, you are running your main pool in a degraded state: I assume you removed the drive from Pool 1 and used to create a single-drive Pool 2.
Re-establishing Pooln1’s parity drive should be your proprity since right now, if one of the drives experiences a failures, you will lose data.
For expanding the pool, your only non-destructive choice is to add another VDEV to Pool1; it should be symmetrical to the existing one, but that can be addressed later with the RAIDZ expansion that’s going to be introduced in later SCALE versions.
I do not suggest increasing the width of your existing VDEV due to it being only RAIDZ1.
You should consider destroying the pool and re-creating it as a single VDEV in RAIDZ2 or RAIDZ3 using all your drives. This will however require any existing data on the pool to be moved.
So, your analysis is correct: if you are able to, I would pick your option 2… with the caveat that the reccomended maximum VDEV width is 12 drives; 16 can be done if you don’t have much performance requirements, and at that widths there is no other way than going RAIDZ3.
In alternative, you could go with 2x 8-wide RAIDZ2 VDEVS: you will be losing some space to the parity drives, but will get more performance and quite high releability, plus an easier path for future upgrades (adding 8 drives is easier than 16).
No, the drive is out and is for RMA. The other Pool (Pool #2) has always been a single individual pool as where i bought from, the seller shipped it so bad that it has got bit damaged so for testing, i used single drive and its working so far.
Yes, but the main thing is if i replace the disk with the failed one and try to resilver, there is still a chance of failure right? But wait, it will be in RAID-Z1 then right?
This is TrueNAS Core. How would i add another vdev to the existing pool? Sorry, its been a while and i forgot how i added disks to my existing pool a couple of years ago ;(
Umm, can you explain this please?
Yes, that’s what i’m thinking friend
Yes, that’s what i think, easier and faster
After using the NAS for a couple of years, capacity does not matter that much, but performance and redundancy. I would really not prefer a single VDEV with 16 disks in it, when the maximum suggested disks number is 12 as per the ZFS Primer. Plus, if i would have another VDEV, i’ll have more IOPS and performance as well
So, if i understand correctly, the data drives will be 12 (6 from each VDEVs) and 4 parity drives (2 from each VDEVs). So, that means even if 4 disks fails from both the VDEVs (2 from each VDEVs), the pool still will be functional, in degraded state.
What if i create 2x 8-wide Mirrored VDEVs? I guess this can be the maximum redundancy one could go for? This way, the data drives will be 8 (4 from each VDEVs) and 8 parity drives (4 from each VDEVs). So, that means even if 8 disks fails from both the VDEVs (4 from each VDEVs), the pool still will be functional, in degraded state. I’m thinking of it because, how my one of the drive failed, i’m quite scared regarding that because it happened all of sudden and do not want to loose data at all. However, i’m not sure how good is this configuration and whether its practical and reliable or not and also whether it can have a good performance or not. Moreover, Is that a good and safe enough configuration?
Makes sense.
That’s a tricky decision, it depends on how bad was the fault on the drive: a few reallocated sectors? Let it resilver.
5 drives in RAIDZ1, even with a lightly faulted one, is still better than a degraded RAIDZ1 VDEV with only 4 drives and no parity.
RAIDZ1 gives you the ability to withstand a single drive failure amongst the ones in the VDEV: if we increase the number of drives in the VDEV, we increase the chances of more than one drive failing at the same time.
Then you could also go with 3x 5-wide RAIDZ2 VDEVS and a single hotspare. It depends on your requirements.
Basically yes, the pool will be able to withstand up to 2 drive failures per VDEV at the same time.
Each VDEV will be able to withstand up to 7 drive failures, but the available space will only be 1/8 of your drives. At that point, it would likely be better to go with 4x 4-way mirrors if you need the resiliency and performance (do note that this is an incredibily resilient layout, likely eccessive for anything but mission-critical enterprise/government data). If you need a 8-way mirror level of redundancy, you should consider professional alternatives to home storage.
I suggest reading the following resources.
So, you mean currently, if i add the new disk to the Pool 1 which is in degraded state, and start re-silvering the data, i’m still good to go? I mean what if during the resilvering process, the disk fails. If i’m not wrong, until the resilvering is done, it will still be in RAID-Z and if during that process the other (old disk) fails, then its a toast. Now, i understand why people don’t recommend using Z1 in the first place. But also, i’ve done resilvering on Z1 for a couple of times when i replaced the disk initially and it was all good. But now i am aware and therefore more scared.
Honestly, the data is like 20TB+ and the drive which failed had bad sectors and then it started growing and the motor failed one day. So, i’m really not sure how bad was the drive state.
Also, i’ve heard about burn-in tests. Does it mean stress test? If so, do you recommend doing that? As the Hardware Guide says it will take 12hrs to complete and that’s a long time and i doubt for the heat and the stress it puts on the drive. Is it safe to do that? I mean will stressing out the drive that much, will it impact the life anyhow when using the drive for production? Do expert like you do that before the final deployment?
Got it!
Hmm. Need to think. But the 2xRAID-Z2 sounds really good option for me as i can have a total of 4 drives (2 drives from each VDEV) failures and the pool will still be operational.
Oh, damn. I completely forgot this!
So, this means 4 DEVS containing 4 drives in each VDEV and the pool can withstand up to 2 drives in each VDEV i.e. upto 8 disk failures and the pool is still operational in degraded state. Is that correct?
Apart from this, i’ve got a couple of questions:
Lastly, i want to understand about the resilvering and parity calculations. As per my research, the parity hits all disks. So, it means, it will stress out all the disks, so there are chances of failures. How does this risk is mitigated between Z1, Z2, 2x 8-wide Z2 setup?
If i understand correctly, in a RAID-Z2 setup, two disks store data and two disks are for parity. Now, let’s say, if a disk failed during resilvering, there is still one parity disk for the rebuild and it will continue rebuilding. Is that correct?
Also, let’s say in the same RAID-Z2 setup, a disk failed and i replaced the disk and started resilvering and during the resilvering if another disk failed, the pool enters the RAID-Z mode right? Are there chances of any data corruption during this time?
Please read the linked resources as they give you the tools to answer most of your questions or directly address them.
I will respond to a few, not being able to run the numbers for the math required not being home and answering from my mobile.
If the new disk you are using to replace the failed disk fails during resilvering, your pool stays in a degraded state and does not gain back the parity to withstand further failures.
If at any point your degraded RAIDZ1 pool experiences the failure of one of the remaining disks, you lose the pool.
Doesn’t sound like it’s in a good state, but do you not run periodic long smart tests? Those are made to get a picture of the health of the drives and take decisions.
Basically, yes. You stress test the hardware before using it in order to have some degree of assurance it won’t fail on you as soon as you put it inside the system.
I reccomend to always burn-in your hard drives when you get them before putting your data on them.
The drives that should be used inside a NAS are the ones engineered to operate 24/7: if they fail after a few hours, because a few days are just a few hours compared to the 3-5 years they are guaranteed to last, your system dodget a bullet and the manufacturer earned a RMA. If your drives reaches unusually high temperatures during burn-in, that’s usually your signal to increase their cooling.
I burn-in my drives for a few days before attaching them to any pool. Professional folks burn-in them for weeks or even a few months, I am on the “if it did not fail on me after a few days of intensive use, it’s good for my personal use” camp.
Nope, each 4-way mirrored VDEV can withstand up to 3 drive failures before incurring in data loss: this means that a pool composed of 4x 4-way mirrored VDEVs can withstand up to 12 drive failures.
With symmetrical VDEVs performance is calulated per VDEV, then it’s multiplied for the number of VDEVs: assuming the use of EXOS X16 drives with a read speed (meaning streaming reads of big blocks and large files) of 260 MB/s, a single 8-wide RAIDZ2 VDEV will theoretically reach 8 - 2 = 6 * 260 MB/s which means around 1500 MB/s; since we have two VDEVs the pool’s theoretical read speed will be of 3000 MB/s, or around 3 GB/s. At least on paper.
256 GB should be and appropriate number, but the real size of the pool won’t be grater than 160 TB: still grater than 128 GB, meaning you should require 256 GB of RAM due to to the power of two rule. With 128 GB you should still experience great performance, and I guestimate that 32 GB would still be plenty for traditional home network speeds.
From unrelated reasons (ie excluding PSU failures), very low[1]; do note that the chance of failure increases as time passes.
If you mean a physical impact, depending on the failure maybe; an improperly configured HBA (aka non IT-mode) is a different story.
For RAIDZ VDEVs not on CORE, it’s a feature landing in the next versions of SCALE.
You can only add more VDEVs to existing pools.
For MIRROR VDEVs, yes: you can add and remove drives as you require.
Parity data is actually striped across all drives. The concept of parity drives is a semplification.
Having more resiliency means that even if a drive fails, if the stress experienced during resilver brings another down the resilver will be able to continue, establishing again enough parity to withstand further failures. That’s why RAIDZ2 is better than RAIDZ1; read the linked resources for more in depth explainations
Finally, please leave me a heart as I spent several hours digiting on my small mobile keyboard with my big fingers in order to answer you; you are lucky it’s just me, @winnielinnie would have asked for an ethernal monument to immortalize his glory!
0, lots of 0s low. ↩︎
This is true, and more people need to know about his scam.
Give my post a like if you want this important message to reach more people with the algorithm.
click the flag to signal your pleasure?
Hey @Davvo so sorry for being late here
Okay!
Yes, that’s understood!
Umm, how do i do that? Is there any guide for that?
Hmm. Sounds fair
Makes sense
OMG. Few days? Dang. I’m learning…
Oh, i see. Will have to do some practice with this.
Hmm
Have got the sticks
Coo cool
Definitely not going to use the IR mode.
Okay, gotcha!
Do you mean i can extend them as well?
Got it got it.
Yes, definitely. Your help has been great, as ALWAYS 
