Issue: Spare drive seems like put in service, but the pool is degraded with errors
h/w:
Intel(R) Core™ i5-3330 CPU @ 3.00GHz
Gigabyte GA-Z77X-UD5H Motherboard
32GB RAM
RAIDZ2 Pool with a spare drive (6+1, 20TB drives):
ada0: ST20000NT001-3MB101 EN01 ACS-5 ATA SATA 3.x device 19074048MB (39063650304 512 byte sectors)
ada1: TOSHIBA MG10ACA20TE 0104 ACS-4 ATA SATA 3.x device 19074048MB (39063650304 512 byte sectors)
ada2: TOSHIBA MG10ACA20TE 0104 ACS-4 ATA SATA 3.x device 19074048MB (39063650304 512 byte sectors)
ada3: WDC WD201KFGX-68BKJN0 83.00A83 ACS-5 ATA SATA 3.x device 19074048MB (39063650304 512 byte sectors)
ada4: WDC WD201KFGX-68BKJN0 83.00A83 ACS-5 ATA SATA 3.x device 19074048MB (39063650304 512 byte sectors)
ada6: ST20000NM007D-3DJ103 SN05 ACS-4 ATA SATA 3.x device 19074048MB (39063650304 512 byte sectors)
ada7: WDC WUH722020BLE6L4 PQGNW540 ACS-5 ATA SATA 3.x device 19074048MB (39063650304 512 byte sectors) <SPARE>
ISSUE: I received an email with POOL DEGRADED message in Jan (I was on travel). So, I had the NAS shutdown. Upon powering up, I see the following:
Current status on the GUI is:
Pool Status RESILVER
Status: FINISHED
Errors: 7
Date: 2026-01-11 01:14:20
Name
Read
Write
Checksum
Status
/mnt/storage
0
0
0
DEGRADED
RAIDZ2
0
0
0
DEGRADED
ada3
0
0
0
DEGRADED
ada4
0
0
0
DEGRADED
SPARE
0
0
0
DEGRADED
ada1
0
0
0
DEGRADED
ada7
0
0
0
ONLINE
ada0
0
0
0
DEGRADED
ada2
0
0
0
DEGRADED
ada6
0
0
0
DEGRADED
spare
ada7
0
0
0
UNAVAIL
However, checking the zpool status shows:
# zpool status -v storage
pool: storage
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: resilvered 5.53T in 11:19:14 with 7 errors on Sun Jan 11 12:33:34 2026
config:
NAME STATE READ WRITE CKSUM
storage DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
(ada3) gptid/95632bf3-c2d5-11e5-9d80-94de80a78ddd DEGRADED 0 0 0 too many errors
(ada4) gptid/610faf79-c2d5-11e5-9d80-94de80a78ddd DEGRADED 0 0 0 too many errors
spare-2 DEGRADED 0 0 0
(ada1) gptid/5bdb0168-c232-11e5-8fb6-94de80a78ddd DEGRADED 0 0 0 too many errors
(ada7) gptid/c97af9d7-11dd-11f0-b286-94de80a78ddd ONLINE 0 0 0
(ada0) gptid/7c62ff80-c232-11e5-8fb6-94de80a78ddd DEGRADED 0 0 0 too many errors
(ada2) gptid/75efd457-c2d5-11e5-9d80-94de80a78ddd DEGRADED 0 0 0 too many errors
(ada6) gptid/43837a79-071d-11e5-9ef5-94de80a78ddd DEGRADED 0 0 0 too many errors
spares
(ada7) gptid/c97af9d7-11dd-11f0-b286-94de80a78ddd INUSE currently in use
errors: Permanent errors have been detected in the following files
I checked the smart status of each of the drives, they seemed okay (please see attached) smart_disks.txt (53.0 KB)
.
Please help, I’m confused as to how to proceed next.
Thanks so much
Ravi
Chipset:
1. 2 x SATA 6Gb/s connectors (SATA3 0/SATA3 1) supporting up to 2 SATA 6Gb/s devices
2. 4 x SATA 3Gb/s connectors (SATA2 2~5) supporting up to 4 SATA 3Gb/s devices
…
2 x Marvell 88SE9172 chips:
1. 3 x SATA 6Gb/s connectors (GSATA3 6/7/8) supporting up to 3 SATA 6Gb/s devices
2. 1 x eSATA 6Gb/s connector on the back panel supporting up to 1 SATA 6Gb/s device
Which pool drive(s) hang on the Marvell controllers?
Gemini, gave me this explanation:
Your hardware configuration immediately explains the root cause of the ZFS pool failure. Your Gigabyte GA-Z77X-UD5H motherboard does not have enough native Intel SATA ports to support your pool, forcing your drives onto lower-tier secondary controllers that are notoriously unstable under ZFS storage loads.
When your TrueNAS system performed a heavy disk operation (like the 5.53TB resilver shown in your first message), the weak Marvell controllers became overloaded, timed out, and briefly dropped the connection to those drives. ZFS interpreted this split-second hardware drop as a disk failure, resulting in the “too many errors” pool degradation.
(but, AI gets knowledge only because of experienced forumers like you, thank you)
Gemini also told me to move the boot drive (ada5) out of Z77 Sata controller to Marvell and move the pool-drive ada0 from Marvell to Z77 Sata controller.
Thus, keep all 6 pool-drives (ada0, 2, 3, 4, 6, 7) on Z77 Sata controller. And the boot-drive ada5 and now-spare drive ada1 on Marvell.
With that, there are 7 errors left. Hopefully they are in real files and you have backups, or you may take the loss. Deal with these, and scrub the pool.
If there are errors in ZFS metadata (“file” looks like “<0x0>:<0xdeadbeef>”), you still have an issue.
No didn’t find any ZFS metadata error, here is the full output:
root@jyoti-nas[/var/log]# zpool status -v
pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:05 with 0 errors on Tue Jun 2 03:45:05 2026
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
errors: No known data errors
pool: storage
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: resilvered 5.53T in 11:19:14 with 7 errors on Sun Jan 11 12:33:34 2026
config:
NAME STATE READ WRITE CKSUM
storage DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/95632bf3-c2d5-11e5-9d80-94de80a78ddd DEGRADED 0 0 0 too many errors
gptid/610faf79-c2d5-11e5-9d80-94de80a78ddd DEGRADED 0 0 0 too many errors
spare-2 DEGRADED 0 0 0
gptid/5bdb0168-c232-11e5-8fb6-94de80a78ddd DEGRADED 0 0 0 too many errors
gptid/c97af9d7-11dd-11f0-b286-94de80a78ddd ONLINE 0 0 0
gptid/7c62ff80-c232-11e5-8fb6-94de80a78ddd DEGRADED 0 0 0 too many errors
gptid/75efd457-c2d5-11e5-9d80-94de80a78ddd DEGRADED 0 0 0 too many errors
gptid/43837a79-071d-11e5-9ef5-94de80a78ddd DEGRADED 0 0 0 too many errors
spares
gptid/c97af9d7-11dd-11f0-b286-94de80a78ddd INUSE currently in use
errors: Permanent errors have been detected in the following files:
storage/.system/rrd-6e20a1c70a684291a4fa4d3197a67324:/localhost/nfsstat-server/nfsstat-symlink.rrd
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNCamera/2014-11-23/00010.MTS
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNCamera/2014-11-16/00001.MTS
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNCamera/2018-08-20 Sw_Mit/2018-08-20/00002.MTS
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNCamera/2016-02-28 JT Yatra/00005.MTS
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNCamera/2021-06-20_Grad/00013.MTS
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNAudio/BV year 2016-17/R_20170224-110113pm.wav
I rearranged the Drives - moved ada0 to Z77 SATA Controller (all 6 drives are on Z77), moved boot-drive and spare-drive on each Marvell SATA Controllers.
Can I now: clear the errors (zpool clear storage), delete the 2022 snapshot (zfs destroy storage/Media@storage-manual-2022-10-23_09-53) and detach the failed drive (zpool detach storage gptid/5bdb0168-…)?
Interesting thing happened, once I did ‘zpool clear storage’ and ‘zfs destroy snapshot’ here is the status ouput:
root@jyoti-nas[~]# zpool clear storage
root@jyoti-nas[~]# zpool status -v storage
pool: storage
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: resilvered 5.53T in 11:19:14 with 7 errors on Sun Jan 11 12:33:34 2026
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/95632bf3-c2d5-11e5-9d80-94de80a78ddd ONLINE 0 0 0
gptid/610faf79-c2d5-11e5-9d80-94de80a78ddd ONLINE 0 0 0
gptid/5bdb0168-c232-11e5-8fb6-94de80a78ddd ONLINE 0 0 0
gptid/7c62ff80-c232-11e5-8fb6-94de80a78ddd ONLINE 0 0 0
gptid/75efd457-c2d5-11e5-9d80-94de80a78ddd ONLINE 0 0 0
gptid/43837a79-071d-11e5-9ef5-94de80a78ddd ONLINE 0 0 0
spares
gptid/c97af9d7-11dd-11f0-b286-94de80a78ddd AVAIL
errors: Permanent errors have been detected in the following files:
storage/.system/rrd-6e20a1c70a684291a4fa4d3197a67324:/localhost/nfsstat-server/nfsstat-symlink.rrd
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNCamera/2014-11-23/00010.MTS
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNCamera/2014-11-16/00001.MTS
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNCamera/2018-08-20 Sw_Mit/2018-08-20/00002.MTS
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNCamera/2016-02-28 JT Yatra/00005.MTS
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNCamera/2021-06-20_Grad/00013.MTS
storage/Media@storage-manual-2022-10-23_09-53:/Video/PNAudio/BV year 2016-17/R_20170224-110113pm.wav
root@jyoti-nas[~]# zfs destroy storage/Media@storage-manual-2022-10-23_09-53
root@jyoti-nas[~]# zpool status -v storage
pool: storage
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: resilvered 5.53T in 11:19:14 with 7 errors on Sun Jan 11 12:33:34 2026
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/95632bf3-c2d5-11e5-9d80-94de80a78ddd ONLINE 0 0 0
gptid/610faf79-c2d5-11e5-9d80-94de80a78ddd ONLINE 0 0 0
gptid/5bdb0168-c232-11e5-8fb6-94de80a78ddd ONLINE 0 0 0
gptid/7c62ff80-c232-11e5-8fb6-94de80a78ddd ONLINE 0 0 0
gptid/75efd457-c2d5-11e5-9d80-94de80a78ddd ONLINE 0 0 0
gptid/43837a79-071d-11e5-9ef5-94de80a78ddd ONLINE 0 0 0
spares
gptid/c97af9d7-11dd-11f0-b286-94de80a78ddd AVAIL
errors: Permanent errors have been detected in the following files:
storage/.system/rrd-6e20a1c70a684291a4fa4d3197a67324:/localhost/nfsstat-server/nfsstat-symlink.rrd
<0x1c9cb>:<0xf06>
<0x1c9cb>:<0xe43>
<0x1c9cb>:<0x34151>
<0x1c9cb>:<0x21a65>
<0x1c9cb>:<0x5a47d>
<0x1c9cb>:<0x257de>
Looks like original drive ada1 was put back in service and original spare-drive ada7 is in place!
Is this normal?
)