Pool degraded, device not healthy

Hello,

Is anyone able to assist please? I am getting the following Alert.

Preformatted text#### Pool Media1 state is DEGRADED: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state.
The following devices are not healthy:

  • Disk WDC_WD40EFRX-68WT0N0 WD-WCC4E2VZ20NK is FAULTEDPreformatted text

I have followed the TrueNAS guide “Hard Drive Troubleshooting Guide (All Versions of FreeNAS)”.

I ran multiple SMART tests, with differing results, however none ever pull up an error on ID 5, 197 or 198. I am yet to check the cables as it says in the guide but am happy to do so if that is the next step?

I am running TrueNAS 25.04.0 with RAIDZ1 using 4 disks total (WD Red 4TB) currently at 90% capacity.

Here are my SMART results

Preformatted textroot@freenas[~]# smartctl -a /dev/sdb
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.15-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke,

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (CMR)
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E2VZ20NK
LU WWN Device Id: 5 0014ee 20d839bc8
Firmware Version: 82.00A82
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database 7.3/5706
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri May 30 20:58:31 2025 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 249) Self-test routine in progress…
90% of test remaining.
Total time to complete Offline
data collection: (53280) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 532) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 120
3 Spin_Up_Time 0x0027 214 172 021 Pre-fail Always - 6258
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 153
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 001 001 000 Old_age Always - 77489
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 153
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 111
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 2141
194 Temperature_Celsius 0x0022 116 104 000 Old_age Always - 36
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 1

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 11953 -

2 Short offline Completed without error 00% 11933 -

3 Short offline Completed without error 00% 11909 -

4 Short offline Completed without error 00% 11885 -

5 Short offline Completed without error 00% 11861 -

6 Short offline Completed without error 00% 11837 -

7 Extended offline Completed without error 00% 11823 -

8 Short offline Completed without error 00% 11789 -

9 Short offline Completed without error 00% 11765 -

#10 Short offline Completed without error 00% 11741 -
#11 Short offline Completed without error 00% 11717 -
#12 Short offline Completed without error 00% 11693 -
#13 Short offline Completed without error 00% 11669 -
#14 Extended offline Completed without error 00% 11655 -
#15 Short offline Completed without error 00% 11621 -
#16 Extended offline Completed without error 00% 11604 -
#17 Short offline Completed without error 00% 11595 -
#18 Short offline Completed without error 00% 11587 -
#19 Short offline Completed without error 00% 11574 -
#20 Short offline Completed: read failure 60% 11573 1450168
#21 Extended offline Completed: read failure 90% 11571 1450168
2 of 2 failed self-tests are outdated by newer successful extended offline self-test # 7

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The above only provides legacy SMART information - try ‘smartctl -x’ for morePreformatted text

And here is a pool status

Preformatted textroot@freenas[~]# zpool status -v
pool: Media1
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use ‘zpool clear’ to mark the device
repaired.
scan: scrub repaired 0B in 10:18:27 with 0 errors on Thu May 15 12:18:28 2025
config:

    NAME                                                STATE     READ WRITE CKSUM
    Media1                                              DEGRADED     0     0     0
      raidz1-0                                          DEGRADED     0     0     0
        sdd2                                            ONLINE       0     0     0
        ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5HF6LHA-part2  ONLINE       0     0     0
        sda2                                            ONLINE       0     0     0
        ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E2VZ20NK-part2  FAULTED     39     0     0  too many errors

errors: No known data errors

pool: freenas-boot
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using ‘zpool upgrade’. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:03:26 with 0 errors on Mon May 26 03:48:35 2025
config:

    NAME          STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0    ONLINE       0     0     0
        sdf2      ONLINE       0     0     0
        sde2      ONLINE       0     0     0

errors: No known data errorsPreformatted text

Sorry if I am missing anything obvious or any useful information.
Thank you in advance!

Sorry for the poor formatting! I’ll try again…

Here are my SMART results:

root@freenas[~]# smartctl -a /dev/sdb     
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.15-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red (CMR)
Device Model:     WDC WD40EFRX-68WT0N0
Serial Number:    WD-WCC4E2VZ20NK
LU WWN Device Id: 5 0014ee 20d839bc8
Firmware Version: 82.00A82
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database 7.3/5706
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri May 30 20:58:31 2025 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.
Total time to complete Offline 
data collection:                (53280) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 532) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       120
  3 Spin_Up_Time            0x0027   214   172   021    Pre-fail  Always       -       6258
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       153
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   001   001   000    Old_age   Always       -       77489
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       153
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       111
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2141
194 Temperature_Celsius     0x0022   116   104   000    Old_age   Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     11953         -
# 2  Short offline       Completed without error       00%     11933         -
# 3  Short offline       Completed without error       00%     11909         -
# 4  Short offline       Completed without error       00%     11885         -
# 5  Short offline       Completed without error       00%     11861         -
# 6  Short offline       Completed without error       00%     11837         -
# 7  Extended offline    Completed without error       00%     11823         -
# 8  Short offline       Completed without error       00%     11789         -
# 9  Short offline       Completed without error       00%     11765         -
#10  Short offline       Completed without error       00%     11741         -
#11  Short offline       Completed without error       00%     11717         -
#12  Short offline       Completed without error       00%     11693         -
#13  Short offline       Completed without error       00%     11669         -
#14  Extended offline    Completed without error       00%     11655         -
#15  Short offline       Completed without error       00%     11621         -
#16  Extended offline    Completed without error       00%     11604         -
#17  Short offline       Completed without error       00%     11595         -
#18  Short offline       Completed without error       00%     11587         -
#19  Short offline       Completed without error       00%     11574         -
#20  Short offline       Completed: read failure       60%     11573         1450168
#21  Extended offline    Completed: read failure       90%     11571         1450168
2 of 2 failed self-tests are outdated by newer successful extended offline self-test # 7

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The above only provides legacy SMART information - try 'smartctl -x' for more

And here is a pool status

root@freenas[~]# zpool status -v
  pool: Media1
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
  scan: scrub repaired 0B in 10:18:27 with 0 errors on Thu May 15 12:18:28 2025
config:

        NAME                                                STATE     READ WRITE CKSUM
        Media1                                              DEGRADED     0     0     0
          raidz1-0                                          DEGRADED     0     0     0
            sdd2                                            ONLINE       0     0     0
            ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5HF6LHA-part2  ONLINE       0     0     0
            sda2                                            ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E2VZ20NK-part2  FAULTED     39     0     0  too many errors

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:03:26 with 0 errors on Mon May 26 03:48:35 2025
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            sdf2      ONLINE       0     0     0
            sde2      ONLINE       0     0     0

errors: No known data errors

The WD-WCC4E2VZ20NK drive is failing.
Also, it looks like you’re not running any smart tests on it, and haven’t done so for 60000 hours. I suspect it will fail at least a long smart test. /Edit: Hmm, unless it loops around at around 65k? That does sound familiar.

Do you have a working backup? If not, make a copy of any important data now.

Either way I recommend you to get a replacement disk ready as soon as possible. Since you have a RAIDZ1 you are close to losing your data.

Thank you neofusion.
I have now succesfully replaced the drive.

1 Like