Disks and VDEV

TrueNAS General
1

Hello guys,

So, i’m having a TrueNAS server running CORE version and it was all good until a disk failed in the setup. The pool has 5 disks in RAID-Z1 and one of disk has failed which means the data is at risk. If a drive becomes faulty or starts having issues, its all toast. Yes, i know you guys would be like that’s dangerous but i was not aware of this risk when i set this up like 2yrs ago. I was not knowing what mess it could bring. But since i’ve been reading the forum now i know.

At this point, anyone would recommend me to just backup the important data immediately and as fast as possible but its over 20TB. However, the actual issue is i do not have enough spare disks which i can put it in this system or have another NAS i can copy data to and that’s where i’m currently stuck in. I mean i do have a single 10TB spare and a 20TB spare. But i’m too scared by the haunting situation like what if i copy the data from my NAS to the spare 10TB and 20TB but there could be three situations:

  • I copy the data to the 10TB and 20TB disk successfully and no drives failure.
  • I create pool with the required number of drives i.e. 16 in 2-Wide RAID-Z2 and then copy the data back from the 10TB and 20TB. Verify the data copied and erase the data/disks (10TB and 20TB) and relax after its done.
  • The above two scenarios sounds okay but what if copying the data from the NAS to the 10TB and 20TB or vice-versa result in drive failure? Then, its all over which i definitely not want.

Before this failure, i wanted to do the following but i was still worried about moving the data:
Copy the data from the degraded Pool to a different NAS and verify the data after moving. Once moved, destroy the old pool and add the new disks to the NAS (total becomes 16 after adding 12 disk more). Create a new pool with the aim for maximum performance+redundancy i.e. 2xVDEV consisting 8 drives each in RAID-Z2 and then copy that data back to the new pool created with 16 disks. But i’m totally stuck as of now due to the lack of enough disk which can give me some redundancy or another NAS where i can copy the data from this degraded pool.

At this moment, i can only think of one option which is my hope for safeguarding the data from the degraded pool. So, my question is can i do the following:

  1. Add the 8 new disk i’ve lying for the purpose of expansion and this time set it up as RAID-Z2 containing 8 new disks. 2. Create a New Pool and copy the data from the degraded pool to this new RAID-Z2 pool. 3. Verify the data after moving. 4. Once moved and verified, start the resilver/scrub process for the new pool. 5. Once the resilvering/scrub is completed, destroy the degraded pool. (Here, i want to do the resilvering/scrub just for the sake of integrity so that if there is any drive which has potential of failure, i can replace it so that when i add a second VDEV and resilver/scrub the whole pool, the drives do not fail in both the VDEVs at same time.) 6. Create a new VDEV with the 4 (existing)+4 (new), yes not 3 because the drive that failed is out of warranty so i have to add a new disk to make a total of 8. 7. Then add this another VDEV to the existing RAID-Z2 pool.

This is the only option i can think of which gives me a little hope that at least my data would be safe here (after its copied to the new pool in RAID-Z2). Does this plan sounds good?

Now, my question is whether this can be done or not and is there any potential risk and whether its safe to do or not or are there any other way to do this which can minimize the risk of failures and is more easier? In case i follow this strategy, would be the data scattered with the same level on all the disks after adding the another VDEV provided that i do a resilvering of the new RAID-Z2 pool. I’m asking this because, i’ve read about the rebalancing thingy on this forum. Is that required in my case considering i’ll add a second VDEV to the new RAID-Z2 pool?

Lastly, for the adding another VDEV, i’ve two disks less so that means 4 existing disks+10 new disks which makes a total of 14 disks. But if this plan can work, i guess i can add the 2 more disks as soon as i can and then make a new VDEV and add it to the existing new RAID-Z2 pool. At least this will make the data a bit safe for sometime until i’ve the 2 disks more and i think this option is better than the current situation where i have 1 disk failed in a 5 disk RAID-Z setup which is huge risk in terms of data loss.

Let me know what do you guys suggest.

Thanks

2

Why not replace the failed disk with something the same or larger?

3 Likes
3

Well, the problem is it’s in RAID Z1. So, of I add another disk and replace the failed disk with a new or equivalent one, there is chance that during resilvering, if a disk fails, the pool is dead and the whole data gets lost and this is not something I want.

4

The exact thing can happen now:

  • During full pool copy, all data is read, just like a re-silver, same stress as a re-silver.
  • During ZFS Scrubs, (you have them scheduled on a regular bases, yes?), all used blocks are read, just like a re-silver, or full pool copy.
  • During a re-silver, bad blocks on source disks likely cause only specific file(s) to be lost. If you have backups, they can be restored.

The comment “bad blocks on source disks likely cause only specific file(s) to be lost” is precise. ZFS by default, stores 2 copies of standard metadata, like directory entries, and 3 copies of more critical metadata. This is because ZFS does not want loss of single block in a directory entry to cause the loss of a data file or part of a data file. Those can be GigaBytes in size, or even TeraBytes in size.

So your concern that RAID-Z1 only had 1 disk of redundancy is good. Today with HDDs of >20TBytes in size, and less ability to back the data up, using RAID-Z2 or triple Mirrors makes sense.

But, you are stuck with what you have.

If you got this far, the only advice I can give, is to backup the pool in pieces. Meaning the most likely cause of other bad blocks or additional failed disk(s) is stress or heat. Staging backups in pieces may reduce the chances of additional failures.


PS - I have seen ZFS in action for automatic metadata repair. My media server uses a non-redundant pool for the media. (I have good backups...) So when I saw a failed block but no associated need to restore a file, it had to be metadata.
3 Likes
5

In a previous thread, i did indicated that this NAS was running fine for almost 2yrs, of course not 24x7 always, whenever i need to read/write the data as this was kind of archival storage for important things which i need on case to case basis when the work comes up. Of course, as this was not always on but the room had AC and the temps were maintained a bit and they were always under 40c. The average power on time during a month would have been like 2-6hrs. All of sudden a disk started having bad sectors and then before i could replace, the disk died eventually. The last thing i remember when i removed the drive and tested it after failing was when powering on the drive, it spin very little speed (not like the normal working drive) and then it was complete silent and nothing was detected in the system either in Windows or Linux or macOS.

So, i’m not really sure whether it was due to stress or heat. The last resilvering was done like an year ago and it was bit scrubbed as per the TrueNAS automatic feature. These disks were tested well before the deployment and i’m not sure what caused it to fail like this.

Yes, that’s right but i was not aware of the risks when this NAS was setup. I was a complete noob that time, hence the trouble now.

What do you mean by backup the pool in pieces? Do you mean copying the data manually to the spare disks i have?

I think the best would be to create a new pool with the 8disks in RAID-Z2 and copy the data first to safeguard the important data and then add another 8 disks to make another VDEV and add it to the existing new RAID-Z2 pool for further expansion. So the layout becomes 2xVDEV in RAID-Z2 containing 8 disks in each VDEV. As per me this seems to be the safest instead of copying the data from the degraded pool to the spare disks (10TB+20TB) which may fail during copying or moving back to the new pool.
The last option which is replacing the failed disks and do a resilver, sounds quite dangerous to me.

Let me know what do you think.

6

Any option that involves a full reading of data is (more or less) equally dangerous. First thing I would do is to establish the parity again, then plan for the layout change.

2 Likes
7

Hmm. That is right but what if the disk fails during the resilvering if i replace the failed disk?

I know there is still a chance of failure even when i try to copy the data from the degraded pool as there is no redundancy at all as it was configured in RAID-Z1.

So, if i try to create a new pool with the new 8 disks in RAID-Z2 and then copy the data from the degraded pool to this new RAID-Z2 pool. Would that work? If so, i think this seems to be the safest considering the drives will be less stressed out as compared to replacing the failed drive in the degraded pool which will cause resilver to happen. Then, i can also add another 8 disks and do a further expansion by adding one more VDEV to the new RAID-Z2 pool. Would this strategy be any good from all the options i’m left with?

8

You keep thinking that a full copy of the data is less stressful than a re-silver. In theory, it may be less than 1% less stressful than a re-silver because a full copy does not need the read the redundant metadata, (except if the block read was bad). But, from what I know, it should not mater.

What does mater is that a full copy could be more stressful in another way. (Note the emphasis on “could be”.) This is due to the heads of the drives moving alot more during a copy of files verses a more linear re-silver.

Newer versions of ZFS include a feature to perform the re-silver in a different manor than originally designed. This does a linear re-silver starting at the beginning of the disk and working toward the end. Their is a bit of setup that could potentially be more impactful than a full copy.

So, 1/2 dozen of one thing, and 6 of another. Your choice.

1 Like
9

At this point, due to your past choices, every remaining option is linked to a certain amount of sizeable risk.

Pick one, and hope for the best.

4 Likes
10

Another lesson to learn from this: Don’t plan or create a top-heavy NAS storage without a true independent backup.

You’ll find yourself in situations where you hold your irreplaceable data in “limbo”. To try to later resilver or make a backup bumps into one or more of the following hurdles:

  • Cannot afford to purchase large enough capacity to build a redundant backup pool
  • Longer resilver times (without an existing backup) that could lose your data if one disk fails
  • Required to purchase a very large capacity drive (single) in order to do something as simple as temporarily copying your at-risk data in the meantime (which still requires reading the entire degraded pool)

If you can only afford a single 24-TiB pool to house your data, then it means you should plan for 12-TiB and 12-TiB, primary and backup. A 24-TiB storage pool without a backup is inferior to a 12-TiB pool with regular backups, if the data is precious and irreplaceable.

1 Like
11

Yes. Unfortunately that’s the risk you’ve already taken.

The same thing applies when trying to copy the data off. Or in fact just leaving things as they are.

Your quickest, safest and simplest approach is to replace the failed drive as soon as possible.

BUT be extra 100% certain you replace the right disk. Verify by serial.

4 Likes
12

If you can properly identify the drive that has failed, then go through the replace process. It’s simple and takes only a few hours. During which time you can still use your system.

Once completed you COULD migrate the complete contents to another pool. If you’re really smart you will use an HP or DELL server with hot swap capability and I would recommend against using GIGANTIC drives as in RAIDZ2 you loose the space of two drives. I have Six 4T drives in my system for media. I’ve used it a lot and have as much free space as used. So there’s little danger of running out of space.

The HP DL380eG8 14LFF is what I recommend. Gives you lots of options. Install a 320i riser card to get pass through on drives, and let TrueNAS do it’s thing with RAID.

One reporter’s recommendation.
riser_HBA

13

DO NOT use hardware RAID with TrueNAS.

Suggested readings:

3 Likes
14

I agree. All in all this is the easiest safest option. I have done it before with a raidz1 and nothing was lost. And ZFS DOES tell you if a file is lost, and you can restore that one file separately. I would simply replace this disk. This is what raid systems are designed for after all.

1 Like
15

Backup the most critical data NOW. Then replace the failed drive and resilver ASAP! Then add a second (and third) parity drive and do a proper backup of at least the important data.

My thinking is that doing a re-silver will do the same stress as reading data off and backing it up elsewhere, but at internal speeds, so less time for a potential second hdd failure.

I run RAID-Z3, and had a 25TB drive fail just a few months in. I always keep an exact duplicate HDD in cold storage, so I threw that in, and shipped off the bad drive to be replaced, so a few months later now, the NAS is humming along fine, and I have replaced the cold spare.

Just my 2¢

And in the end remember that it’s all ephemera. :wink:

2 Likes
16

Hmm. I think you have a point here.

Damn sick.

So, resilvering has less stress on the disk than copying the files from the pool?

17

Yes, you’re right ;(

18

Yes, i get your point friend.

19

Yes, i understand!

20

I understand it but its RAID-Z1 and one disk has failed so what makes me worried is if i replace the failed disk and start to resilver, either of the disk may fail and then i’m dead.

Yes, i wish i would have used RAID-Z2 when creating the pool. But i was not that aware of things when i created it.

As far as i’m aware, one should not use the RAID but should directly connect the drives without the involvement of Hardware RAID. One can use HBA which will simply passthrough the disk to the TrueNAS and everything can be handled by the TrueNAS.