Zfs error replacing faulty disk

[lepinaydavid]

Hello,

I have a zfs pool with 4 disks, one of them was faulty.
I’ve replaced it physically.
It seems that, instead of replacing the faulty one by the new one,I’ve extended the pool…

pool: Data
id: 14766178653660046557
state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
see:
config:

    Data                                      UNAVAIL  insufficient replicas
      raidz1-0                                DEGRADED
        5edbe71c-1154-4d81-95b8-39f88b6e367e  ONLINE
        19659ea4-cefc-4aa7-84fb-cb9eab7ce9fc     ONLINE
        5139077374136145411                                UNAVAIL
        4f915d28-fe69-485c-9f9a-bfce0631f2f8       ONLINE
      89f7804c-cd58-4ab2-a11e-058d511541d8    UNAVAIL

the last one seems to be the new disk, 89f7804c-cd58-4ab2-a11e-058d511541d8
Do I have a way to save my data ?
Thanks all for your help

Regards

[etorix]

@lepinaydavid

It seems that, instead of replacing the faulty one by the new one,I’ve extended the pool…

That’s it, and you’re in serious trouble… Do you have a backup?
Plug back the drive you’ve added. The pool absolutely requires it now. Do not reformat it, or attempt to use it for replacement again: That would destroy your pool for good.
What’s the condition of the drive you removed? Does it still spin when powered? Can it be plugged back?

What’s your hardware?
If you’re short of SATA ports, you need a HBA to be able to add more drives.
And you’ll need more drives: At least one more to actually replace the failing drive. And then even more drives to backup your pool completely, destroy it and recreate it with a sane geometry.
(Not to mention that if you cannot get the pool back, your last hope to recover something will be Klennet ZFS Recovery, and this will require disk space to recover data to.)

[lepinaydavid]

The faulty disk is having very very strange spinning noise, and this one is not correctly detected now…
I will try to plug it back at the end of my holidays but…
I’ve read the documentation, asked ChatGPT and as I see it’s not possible for me to recover my data

[etorix]

So ‘Faulty’ is likely dead. The new drive HAS to come online for you to do anything with ZFS.

Failing that, your last line of hope is Klennet ZFS Recovery, running from Windows. Good points: Scanning is free, and it possibly does not need the new drive to recover what was in the raidz1 vdev before the new drive was added. Bad points: If Klennet can recover something, you’ll need sufficient free space (new drives) and pay the $399 license.

[lepinaydavid]

Thanks for your help Etorix
I will try this software Klennet ZFS Recovery

[winnielinnie]

I really want to write up a good feature request for integrating pool checkpoints into the GUI, and triggered automatically before certain operations.

Need to find the motivation to do it. I just don’t feel like putting so much effort into creating a feature request, justifying it, and giving examples on how to implement it, only for it to die on the vine.

[Johnny_Fartpants]

@winnielinnie

I really want to write up a good feature request for integrating pool checkpoints into the GUI, and triggered automatically before certain operations.

Problem is you can’t currently replace a drive in TrueNAS when a checkpoint exists so dude would have still have had to discard the checkpoint before the disk replacement went bad and would still be in trouble. Caveat you may be able to do an in place replacement but I’d imagine it’s more common that users have a faulted/offline/remove first.

[winnielinnie]

@Johnny_Fartpants

Problem is you can’t currently replace a drive in TrueNAS when a checkpoint exists

That’s not what happened. @lepinaydavid accidentally extended the pool. In this situation, a checkpoint would automatically be made before the operation. They would then rewind to the checkpoint, back to a time before the new vdev or extension happened.

If they actually (correctly) did replace a faulty drive, then no automatic checkpoint would be made.

The safety trigger would happen only for adding new vdevs. Then it could be automatically discarded the next day, maybe.

Win-win.

[Johnny_Fartpants]

Can you extend a pool with a checkpoint in place? I’ve never tried it.

[winnielinnie]

Should be able to.

You make a checkpoint.

You add a new vdev to the pool.

If you rewind the pool back to the checkpoint, it’s like the new vdev was never added.

If you saved any new data on the pool, you’ll of course lose that data, which was written after the checkpoint’s creation.

[Johnny_Fartpants]

I feel like you can’t but I’d need to check to be sure. If it works then it’s a neat idea.

[Johnny_Fartpants]

Yeah no joys. I think this happens for the same reason hot-spares don’t work and you can’t replace a drive that has gone offline when a checkpoint exists.

[winnielinnie]

I think you misread my post?

@winnielinnie

That’s not what happened. @lepinaydavid accidentally extended the pool.

@winnielinnie

If they actually (correctly) did replace a faulty drive, then no automatic checkpoint would be made.

@winnielinnie

The safety trigger would happen only for adding new vdevs. Then it could be automatically discarded the next day, maybe.

Adding a new vdev to a pool is not the same as replacing a drive in an existing vdev.

@lepinaydavid intended to replace a failing drive, but they accidentally added a new vdev. This is where an automatic checkpoint trigger could have been a safety net.

[Johnny_Fartpants]

@winnielinnie

That’s not what happened. @lepinaydavid accidentally extended the pool. In this situation, a checkpoint would automatically be made before the operation. They would then rewind to the checkpoint, back to a time before the new vdev or extension happened.

It appears that you can’t extend a pool while a checkpoint is in place. Therefore the extend would fail and the checkpoint would be of no use.

[winnielinnie]

Your example made it seem like you tried to replace a drive, not add a new vdev? Unless I read it wrong?

@Johnny_Fartpants

It appears that you can’t extend a pool while a checkpoint is in place.

The GUI is ambiguous. The correct term is “add”, as in zpool add, even if the GUI might label it as “extend”. You’re not actually extending an existing vdev. You’re adding a new vdev entirely. (Which is what the OP did.) A checkpoint would reverse this, beginning the pool back to the state before a new vdev was added.

[Johnny_Fartpants]

@winnielinnie

Your example made it seem like you tried to replace a drive, not add a new vdev? Unless I read it wrong?

No sorry I tried an extend and it failed.

[winnielinnie]

“Extend” an existing vdev or “add” a new vdev to the pool?

TrueNAS terminology differs from ZFS terminology. (Still don’t know why they use different words.)

[Johnny_Fartpants]

Extend a vdev.

[winnielinnie]

Right. That won’t work with checkpoints.

What the OP did was add a whole new vdev (single-drive stripe) to the pool.

Their pool went from 1 vdev (RAIDZ only) to 2 vdevs (RAIDZ and single-drive stripe).

If they had a checkpoint before the new vdev was added to the pool, they could rewind back to the state where only 1 vdev (RAIDZ) existed.

[Johnny_Fartpants]

Ah I get you. I heard extend and assumed extend. Silly me

Adding a new stripe vdev to an existing RAIDZ1 pool via the UI is impressive. I can’t even figure out how to do that yet as it really doesn’t want me to shoot myself.