Unreadable/ uncorrectable sectors after self-test

TrueNAS General
1

Hi,
My TrueNAS just ran into a bunch of problems and I am not quite sure what to do/ how to act or what I should prepare for, and how. Essentially, it reported that /dev/ada2 has 8 offline uncorrectable sectors:

saturn% sudo zpool status -v

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Password:
  pool: boot-pool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
  scan: scrub repaired 0B in 00:00:03 with 0 errors on Thu Mar 27 03:45:03 2025
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   DEGRADED     0     0     0
          mirror-0  DEGRADED     0     0     0
            da0p2   UNAVAIL      0     0     0  cannot open
            nvd0p2  ONLINE       0     0     0

errors: No known data errors

  pool: core
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 03:37:20 with 0 errors on Sat Mar  1 10:37:20 2025
config:

        NAME                                                STATE     READ WRITE CKSUM
        core                                                ONLINE       0     0     0
          raidz2-0                                          ONLINE       0     0     0
            gptid/196085e2-1152-11ea-957b-3cecef012038.eli  ONLINE       0     0     0
            gptid/1a3b1dad-1152-11ea-957b-3cecef012038.eli  ONLINE       0     0     0
            gptid/1b1ee326-1152-11ea-957b-3cecef012038.eli  ONLINE       0     0     0
            gptid/1be79663-1152-11ea-957b-3cecef012038.eli  ONLINE       0     0     0

errors: No known data errors

  pool: jail
 state: ONLINE
  scan: scrub repaired 0B in 00:00:21 with 0 errors on Tue Apr  1 00:00:21 2025
config:

        NAME        STATE     READ WRITE CKSUM
        jail        ONLINE       0     0     0
          nvd0p3    ONLINE       0     0     0

errors: No known data errors


saturn% sudo smartctl -a /dev/ada2
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZDH71EQQ
LU WWN Device Id: 5 000c50 0b66abe62
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Apr  1 10:47:24 2025 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 116) The previous self-test completed having
                                        the read element of the test failed.
Total time to complete Offline
data collection:                (  591) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 670) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   078   064   044    Pre-fail  Always       -       65498192
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       244
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   090   060   045    Pre-fail  Always       -       1118270736
  9 Power_On_Hours          0x0032   057   057   000    Old_age   Always       -       38112 (166 230 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       143
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   072   057   040    Old_age   Always       -       28 (Min/Max 25/31)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       143
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       582
194 Temperature_Celsius     0x0022   028   043   000    Old_age   Always       -       28 (0 17 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       8
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       8
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       37351h+40m+06.262s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       42134266433
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       124152133710

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       40%     38110         -
# 2  Short offline       Completed without error       00%     38104         -
# 3  Short offline       Completed without error       00%     38080         -
# 4  Short offline       Completed without error       00%     38056         -
# 5  Short offline       Completed without error       00%     38033         -
# 6  Short offline       Completed without error       00%     38009         -
# 7  Short offline       Completed without error       00%     37985         -
# 8  Short offline       Completed without error       00%     37961         -
# 9  Short offline       Completed without error       00%     37937         -
#10  Short offline       Completed without error       00%     37913         -
#11  Short offline       Completed without error       00%     37889         -
#12  Short offline       Completed without error       00%     37865         -
#13  Short offline       Completed without error       00%     37841         -
#14  Short offline       Completed without error       00%     37817         -
#15  Short offline       Completed without error       00%     37793         -
#16  Short offline       Completed without error       00%     37769         -
#17  Short offline       Completed without error       00%     37745         -
#18  Short offline       Completed without error       00%     37721         -
#19  Short offline       Completed without error       00%     37697         -
#20  Short offline       Completed without error       00%     37673         -
#21  Short offline       Completed without error       00%     37649         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

How to interpret all of this?

Thank you for your feedback.

2

Hi and welcome to the forums.

These are the values you are referring to and that TrueNAS has alerted you for.

These values are generally a good indicator that a drive is failing.

Do you have a spare to replace it? If not then I would suggest you get one ordered.

Your pool is a 4 disk Z2 so nothing major to worry about atm as you can afford to lose two drives before losing data but nevertheless get that disk replaced.

3 Likes
3

Not to forget:

Replace or RMA as soon as possible.

And one of your mirrored boot drives is unavailable, but this poses no safety risk. Always keep a recent copy of the configuration file at hand.

1 Like
4

Thank you both for coming back to me so quickly. I really appreciate the feedback. I guess I need to replace the drive asap.

In that regard: I recently read a lot about manipulated serviced/refurbished Seagate drives which are being sold under false label. I am wondering whether I should shy away from those offers at all and purchase new a new Seagate drive?

Is there maybe any recommendation for reliable yet affordable drives?

Also, if I would want to upgrade storage capacity, I would just swap in the bigger sized one and after replacing the last smaller one, the pool size will be increased automatically, right?

5

Iā€™m afraid not. Manipulated drives are certainly NOT confined to Seagate, so either you buy new or you have to trust your seller on any ā€œrefurbishedā€ drive (and trust that seller himself was not scammed by its supplierā€¦).

Yes, CORE shall just do the right thing.
Although I hate to say that but, starting with 22 TB drives and a 4-wide raidz2, your easiest path to increase storage is to sidegrade to SCALE and use raidz expansion to go wider. (Of course, if you have a backup, you could just do the old-fashioned backup-destroy-restore, avoid expansion shenanigans, and stay with CORE.)

1 Like
6

That makes complete sense. I think I will just look for a good deal on a new drive instead of buying ā€˜somethingā€™.

Well, that is another thought I had in the past already, migrate to SCALE. My pool is encrypted which makes moving it fairly difficult. There is a way to remove the encryption via console but I am not sure about this.
Especially because my backup strategy is ā€¦ letā€™s say improvable and I am not sure how to do it otherwise. So yeah, ideally I would create a backup and move to scale directly.

Hmm that makes me think: how would I create a backup in the first place? Rsynch from old to new pool? Maybe I could make best use of my current situation, add some extra work and be on SCALE.