Hi all. I have a pool of 4 vdevs of 2 disk mirrors. A few days ago, one of the drives failed and changed to “Removed” state. I bought a new drive, swapped out the failed one and clicked “Replace”. The pool started resilvering, but soon I was notified that some sectors were not readable on the other drive in the mirror, and the drive is in “Faulted” state.
Now I have a completely failed drive, a damaged drive with data on it, and a new drive with no data.
After the resilvering finished, the new drive is reporting millions of checksum errors. I guess it means the data is not actually copied onto it? The resilver report says only 18G of data were resilvered, while I clearly have more than that. I tried to scrub the pool but I just see the number of checksum error increasing.
Here is a screenshot of current pool status
How is this system cabled? Is there a chance that when you replaced the failed drive you could have accidentally nudged a cable to this other drive? The situation doesn’t look good atm so I would check cables first.
Thank you for your reply. The system is cabled within a server chassis. The server chassis has two sata backplates, each holds 4 drives in the front panel drive bay. Each pair of mirror is separated into these two backplates. I did pull out the working one and look at the serial number (when the system is powered off), but I don’t think it affects the cables. I’ll try to clean the dust, maybe it will help. Thank you for your suggestion.
What is the next best thing to do in your opinion?
If you are able to access your data back it up asap (if not already) before anything else.
Check the cabling etc.
Run zpool clear and then run a scrub.
If things look better after that try to replace your failed drive again.
Worse case scenario here you have lost one drive in a mirror and the other is having issues. The saving grace is that your smart output for your possible failing drive looks fine to me hence why I’m thinking it could be a cable issue. Needless to say if two drives in a mirror have issues your data is at risk.
Final option and it may not be possible could you move these drives to another system to rule out hardware issues and perform a zpool clear / scrub and resilver in that environment.
Oh, sorry for the confusion. The damaged drive is sdh. Sdc is the new drive that I bought and replaced into the system. The issue is somehow resilvering is not copying the data to it. The original failed drive has been removed from the system.
In that case there is not much else you can do. Essentially you have lost two drives in a mirror and that results in a loss of the pool. Unfortunately we often see this scenario where one drive fails and another drive about the same age fails during resilver.
Can you still access the pool data? Do you have a backup?
I haven’t found any corrupt data yet. That led me to think most of the data may be salvageable. I have backups for the important files, but not everything. The pool is too large (~36TB), and I do not have another machine with equal or greater capacity.
Ok I think it’s safe to say this pool is toast so its now a data recovery situation. Appreciate the limitations in size but you need to take action now and backup everything you can and that you care about. Once complete the pool will need destroying and going again.
Technically as its a pool of mirrors, you can delete a vdev. The trouble is that ZFS will attempt to vacate the data from the vdev by reading and then writing to the other vdevs - which presumably won’t actually work in this situation due to errors.
