Spare disk unavailable

[RickyIT]

I am showing my spare disk unavailable.

I can see it is sdb - and is fine in the DRAC.

I think all is ok - but the GUI doesn’t seem to reflect it.

Can anyone point me in the right direction to fix?

If I try and replace with one of the 3 x unassigned disks, it errors “device is reserved as a hot spare”

Ok in DRAC:

Kind regards,

Ricky

[nite_route]

yeah that gui “unavailable” on a spare while drac still sees the drive is usually the pool still holding the old spare reservation, not the disk being dead.

stuff that usually clears it for me:

1. zpool status -v and look for the spare path / guid under spares
2. 2. if the old spare is still listed, remove it first: zpool remove POOL <old-spare-guid-or-path> then add the replacement
3. 3. the “device is reserved as a hot spare” error on unassigned disks usually means that guid is still marked spare in the pool label. zpool labelclear -f /dev/disk/by-id/... on the NEW candidate only after you are sure it is empty, then try attach again from the ui
      paste zpool status -v for that pool plus ls -l /dev/disk/by-id | grep -i sdb (or the serial from drac).

[RickyIT]

Hi Nite_route, I’m struggling with this one.

My guess is sdn showed 7 errors, so the spare swapped in for it.

If so, how do I put that right?

[SmallBarky]

Post your detailed hardware listing and how the drives are attached. Your latest screenshot isn’t showing the spare in use from your zpool status -v results. Do you have a plain HBA?

What's all the noise about HBA's, and why can't I use a RAID controller?

[RickyIT]

Hi, it is a Dell PE R740xd with a load of disks hanging off a PERC controller…

I have had a similar issue before, but the disks show up as DEGRADED or REMOVED. Then I can go and REPLACE the failed one, and re-silver.

But this time it is showing all 15 x disks all fine - but somehow my spare UNAVAIL - so I would guess it has swapped itself in for another disk. But which one?

This is a screenshot from before:

And now:

[SmallBarky]

Dell PERC 730 Mini is a RAID controller and is not recommended for use with TrueNAS. It may be the cause of some issues as ZFS (TrueNAS) needs direct access to the disks. The article or post I linked in a previous post should explain it a bit. RAID controllers also can cause issues with moving the pool (disks) to another system or just recovery efforts.

[RickyIT]

Maybe I explained it wrong. Although the server does have a RAID controller - the disks used for the TrueNAS are all Non-RAID. So TrueNAS formats them etc.

I have 2 x Dell PE and both have been running fine for 8 years.

Hope that makes more sense!

[RickyIT]

Maybe I need to kick off a re-silver? or maybe just reboot the NAS?

I’m getting desperate, but I am RAIDZ2, 3 x 5wide VDEVs, so I believe I am safe unless another 2 x disks fail ?

[SmallBarky]

Try rebooting.
You really need to read the HBA post. Your controller is presenting the disks as individuals but it is not the same as a plain HBA, as highly recommended. Your controller also lists a cache and that is another red flag issue.

The online documentation also has warning about your controller setup.

Storage Controllers - TrueNAS Hardware Guide

[nite_route]

looking at that zpool status shot, i dont see a spare marked INUSE replacing a vdev member. if the spare had actually swapped in for sdn, youd usually see that disk FAULTED/REMOVED and the spare sitting ONLINE in its place. UNAVAIL under spares with every data disk looking ONLINE usually means the spare reservation itself is sick, not that a silent replace already finished.

on a PERC 730 that gets messier because TrueNAS is not talking to the drives the way ZFS wants. SmallBarky’s HBA note is the real long term fix. short term, paste a fresh zpool status -v (full text) and the by-id path for the UNAVAIL spare serial from DRAC so we can see whether remove + re-add is even safe. dont labelclear anything that might still hold pool data.

[RickyIT]

Thanks for your help nite_route. You are correct, last time it swapped in a drive, and showed which one had failed/degraded etc. This time it is showing all 15 disks in the pool are fine, just the spare UNAVAILABLE.

I audited the disks, and can tell you the spare is disk 0:1:1 which is showing as fine and online in DRAC. It is disk sdb in TrueNAS.

Hope that helps.

[nite_route]

yeah if DRAC shows 0:1:1 / sdb healthy and every data disk is ONLINE, i would not kick a resilver. that only helps when a vdev member is actually FAULTED or DEGRADED.

what i would try next:

1. zpool status -v and copy the full spares section (guid + path)
2. zpool remove POOL <that-spare-guid> to drop the UNAVAIL spare reservation
3. re-add the same disk by-id as spare from the UI or zpool add POOL spare /dev/disk/by-id/...

if remove says busy / in use, reboot once (PERC Non-RAID can leave stale state), then retry remove before re-adding.

do not labelclear the live spare unless remove already succeeded and you are re-adding a blank candidate. paste the exact zpool remove error if it refuses.

[RickyIT]

Thanks Nite_route. I will schedule in some downtime this week – and run through this. I have a hunch a reboot of the NAS will update the stale state.

I’ll keep you updated.

[RickyIT]

A reboot did remove all the VDEV errors:

However the spare still says UNAVAIL:

But I have a feeling maybe another reboot will make that disk available again, but for now I am happy with it, as there is a green tick next to VDEVs!

Thanks all for the help.