I have RaidZ2 with two hot spares, Looks like i have three faulted drives I added a third drive as another space… Now what? It’s running a scrub. 77% ish.
I can’t take any of the faulted disks offline atm. I’m also scared I don’t know what drives are being used, are the spares actually working? Has it rebuilt?
Not sure where to go next with this. There is a total of 11 disks (I justed added one more)
all are 18TB except the last one i added which is 20TB.
What does zpool status -v show ?
It will also tell us if the hotspares kicked in, because 3 failed drives in a RAIDZ2 otherwise means that the pool is gone.
Please describe your server (motherboard, CPU, RAM, HBA, how everything is wired and powered…) and repost the output of zpool status -v as formatted text (</> button, or paste the whole thing between two lines of three backquotes) so we can read it with the proper indent. Use a SSH session to have a clean output.
```
text goes here
```
As this seems to be CORE, you may want to add camcontrol devlist
and whatever is appropriate among sas2flash -list sas3flash -list storcli /c0 show all
Do not take any drive offline anyway.
I am confused by your flat pasting of zpool status. My guess is that spare da0 kicked in to replace da9 but also failed with read errors, then da2 further failed but new spare(?) da10 did not kick in and you’re standing on the edge with two failed members in a raidz2. But I hate guessing.
Do you have backups and/or cold spares?
data is all working as expected.
raidz2 and it had two hot spares.
Looked to me like it had failed Three drives so I put another spare online.
I’m just confused now cause I don’t know where I am in the replacement.
pool: boot-pool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using ‘zpool upgrade’. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:00:03 with 0 errors on Mon Oct 19 03:45:03 2026
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
nvd0p2 ONLINE 0 0 0
errors: No known data errors
pool: tank
pool: tank
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use ‘zpool clear’ to mark the device
repaired.
scan: scrub in progress since Fri Oct 23 11:57:13 2026
105T scanned at 583M/s, 104T issued at 577M/s, 121T total
0B repaired, 86.11% done, 08:28:50 to go
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/dc9edc24-c4e1-11ec-863a-3cecef618610 ONLINE 0 0 0
gptid/dca052ba-c4e1-11ec-863a-3cecef618610 ONLINE 0 0 0
gptid/dcd889c0-c4e1-11ec-863a-3cecef618610 FAULTED 94 0 0 too many errors
gptid/dca18675-c4e1-11ec-863a-3cecef618610 ONLINE 0 0 0
gptid/dccd5882-c4e1-11ec-863a-3cecef618610 ONLINE 0 0 0
gptid/dca42960-c4e1-11ec-863a-3cecef618610 ONLINE 0 0 0
spare-6 UNAVAIL 96 0 0 insufficient replicas
gptid/dd132db0-c4e1-11ec-863a-3cecef618610 FAULTED 1.33K 0 0 too many errors
gptid/dcd5e195-c4e1-11ec-863a-3cecef618610 FAULTED 97 0 0 too many errors
gptid/dce68a55-c4e1-11ec-863a-3cecef618610 ONLINE 0 0 0
gptid/dcf782bb-c4e1-11ec-863a-3cecef618610 ONLINE 0 0 0
spares
gtpid/dcd5e195-c4e1-11ec-863a-3cecef618610 Avail
gptid/b897c94c-cf0c-11f1-81fd-3cecef618610 Avail
Engaged spares normally show “IN USE” in all caps. I’m bit puzzled by this “Avail” with small caps and the labels showing with negative indent spares. (“spares” aligns with “tank” but should really align with “raidz2-0”.)
Otherwise it seems to fit my guess, so I’d suggest to use GUI to Replace gptid/dcd889c0… (da2) by gptid/b897c94c… (da10) so you hopefully get one degree of redundancy back and then rush to send all three faulted drives for RMA.
I’m not a big fan of hot-spares precisely for this reason. It can add a lot of confusion to an already stressful situation.
My guess is that the first disk failed and the hot-spare kicked in but failed during resilver. Another disk has subsequently failed which for whatever reason hasn’t triggered your second spare.
It should be said that your data is now at huge risk because if another disk fails you will lose your pool. I’d suggest you backup important data if you haven’t already and then start the replacement process.
Cold spares work just as well so long as you are regularly monitoring your system.
Ah sounds like instead of replacing a failed drive you just added another spare right? Unfortunately adding a spare doesn’t fix existing failed drives in a pool. They just sit there waiting for another drive to fail which in your case would be disaster.
Wow what drives you using?
Those are hot-spares waiting for another failure to happen.
Once you have backed up your important data I would be inclined to remove those hot-spares from within the GUI and use them to replace your failed disks one at a time.
Using zpool status isn’t intuitive if there is a hot spare currently being used in place of a failed disk. The layout of the text doesn’t make it clear what is going on, unless you’re familiar with how zpool status presents hot spares.
There’s also the misconception that hot spares are the same as cold spares, but automatic. A hot spare doesn’t automatically “replace” a failed disk, unless you manually intervene and tell it to take the place of the former disk. If you don’t, then it remains as a hot spare, while temporarily providing a resilvered and working disk in place of the failed disk. If you “replace” the failed disk with a new one, it must be resilvered from scratch, which means that the fully resilvered hot spare is now redundant and returns to being an inactive hot spare. (This comes out to two full resilvers to replace a single failed disk.)
AFAIK, ZFS doesn’t have a way to configure a hot spare to act like an automatic cold spare to permanently replace a failed disk, which is what I believe most new users expect to happen.
