I have, in the past, replaced a drive experiencing errors with a new disk using the ZFS replace-and-resilver method. The only problem here is the time it takes to resilver.
However I was wondering whether it is possible to use a hardware disk copier instead, removing the failing drive, performing the disk copy and replacing the new disk into the RAIDZ1 array.
This should be much faster as both disks would be performing sequential IO instead of random IO. In theory, this should be around 20x faster.
Has anybody done this? Is there something I’m missing here with this theory?
I’ve only got one home NAS (production), so I don’t have a dev environment on which to test this.
First, you would have to shutdown the server to quiesce the RAID-Z1 vDev / pool. Next, if there is a single bad block in used space on the old disk, then you absolutely need to run a scrub after to have ZFS fix that bad block. Thus, not saving any time.
Further, if your pool is only 1/2 full, the re-silver only has to read 1/2 of the other disks to bring the new disk up to speed. To be fair, if you are pulling the old disk and putting in the new disk, ZFS does have to read some or all of the other disks to recreate the new disk.
A better option is the replace in place method. This does require the existing disk to be both reasonably good, and have an additional disk port for the new, replacement disk.
ZFS will basically mirror the failing drive and when complete, detach the old drive. This is automatic. Any bad used blocks on the old disk are recovered from any available redundancy, RAID-Z1 in this case. (Or potentially redundant MetaData…) Rough example in the command line, but I think the GUI can do so as well:
zpool replace pool old_disk new_disk
There was a project years ago to perform in sequence scrubs and re-silvers. I don’t recall the status, nor if it is automatically performed. (I think there is a certain case where ZFS won’t use sequential scrubs and re-silvers, but I don’t recall why or when.) In any case, that would be faster than the old walk the tree method.
Thanks for your prompt and insightful reply. Let me give some context to my situation: I have a four-bay NAS with all 4 disks in a RAIDZ1 configuration. I also have a USB connected disk caddy in which I could place a new disk for your replacement method.
With that in mind, could I replace the failing disk with the disk in the caddy, then afterwards power-down the machine, remove the failing disk from the NAS, replace with the new disk and power-up. Would this work?
The point is that you want the most important drives, that is the good disks and the replacement, on the most reliable connections. Leave the dubious drive to the least reliable connection.
Plus, doing it this way will at least save one reboot cycle: When resilver is done you can just unplug the USB caddy.
I agree, this might work. If you can insert a drive in your USB Caddy, and TrueNAS sees it, then you should be able to do what @Arwen is suggesting. I never thought about moving the failing drive to an external caddy, but it makes more sense to me after reading it.
When I say failing, I don’t wait until I get ZFS errors, I change them out based upon smart thresholds being exceeded.
So just to confirm, I don’t add any disks to the pool do I? I don’t want to inadvertently expand the array to 5 disks! Presumably, ZFS will find the failing disk in the usb caddy based upon the disk identifier?
I am probably not teaching you anything here but for some readers, it will help them.
SMART is what you should use, just as you are doing to determine a failed or failing drive.
A ZFS error means there is data corruption at some level. A lot of people think one is the same as the other. I’m not saying that a person couldn’t have both a SMART error and ZFS error, that does happen as well, they just should not be assumed to be the same cause of a problem. That is why we like to request a SMART Long test to see if it passes or not. If it passes and there are no outlying SMART values, then we assume the drive is good.
