Truenas scale disk errors and pool missing

[smic717394]

@Protopia

smartctl -x /dev/sdX

Here are the smartctl -x for the 4 drives.

First Disk

sda

root@HomeServer[~]#  smartctl -x /dev/sda 
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.32-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFZX-68AWUN0
Serial Number:    WD-WX22DC1RNK8N
LU WWN Device Id: 5 0014ee 2bfab56ec
Firmware Version: 81.00B81
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Nov 23 01:13:41 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (42600) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 452) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   241   221   021    -    2950
  4 Start_Stop_Count        -O--CK   100   100   000    -    65
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   072   072   000    -    20879
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    64
192 Power-Off_Retract_Count -O--CK   200   200   000    -    33
193 Load_Cycle_Count        -O--CK   200   200   000    -    69
194 Temperature_Celsius     -O---K   119   093   000    -    31
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    1
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      78  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 4
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 4 [3] occurred at disk power-on lifetime: 16244 hours (676 days + 20 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 80 00 00 a3 a4 35 da 40 00  Error: UNC at LBA = 0xa3a435da = 2745447898

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 58 00 68 00 00 a3 a4 40 d8 40 08     00:13:34.700  READ FPDMA QUEUED
  60 03 18 00 60 00 00 a3 a4 3d c0 40 08     00:13:34.700  READ FPDMA QUEUED
  60 07 e8 00 40 00 00 a3 a4 35 d8 40 08     00:13:34.700  READ FPDMA QUEUED
  ea 00 00 00 00 00 00 00 00 00 00 a0 08     00:13:34.676  FLUSH CACHE EXT
  60 02 68 00 20 00 00 a3 a4 33 70 40 08     00:13:34.666  READ FPDMA QUEUED

Error 3 [2] occurred at disk power-on lifetime: 7728 hours (322 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 01 af dd cc 48 40 00  Error: IDNF at LBA = 0x1afddcc48 = 7245515848

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 08 00 b0 00 01 ba 46 a4 d8 40 08 31d+14:23:48.066  WRITE FPDMA QUEUED
  61 00 08 00 38 00 01 af dd cc 48 40 08 31d+14:23:48.066  WRITE FPDMA QUEUED
  ea 00 00 00 00 00 00 00 00 00 00 a0 08 31d+14:23:43.369  FLUSH CACHE EXT
  61 00 08 00 30 00 01 d1 c0 bd 98 40 08 31d+14:23:43.368  WRITE FPDMA QUEUED
  61 00 08 00 28 00 01 d1 c0 bb 98 40 08 31d+14:23:43.368  WRITE FPDMA QUEUED

Error 2 [1] occurred at disk power-on lifetime: 7727 hours (321 days + 23 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 01 ba 43 89 c0 40 00  Error: IDNF at LBA = 0x1ba4389c0 = 7419955648

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 08 00 98 00 01 b7 0e 59 18 40 08 31d+13:32:41.695  WRITE FPDMA QUEUED
  61 00 08 00 20 00 01 ba 43 89 c0 40 08 31d+13:32:41.695  WRITE FPDMA QUEUED
  ea 00 00 00 00 00 00 00 00 00 00 a0 08 31d+13:32:37.191  FLUSH CACHE EXT
  61 00 08 00 80 00 01 d1 c0 bd e8 40 08 31d+13:32:37.191  WRITE FPDMA QUEUED
  61 00 08 00 78 00 01 d1 c0 bb e8 40 08 31d+13:32:37.191  WRITE FPDMA QUEUED

Error 1 [0] occurred at disk power-on lifetime: 3998 hours (166 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 01 c6 c1 a1 98 40 00  Error: IDNF at LBA = 0x1c6c1a198 = 7629545880

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 10 00 28 00 01 c6 c1 a1 98 40 08  4d+21:32:31.932  WRITE FPDMA QUEUED
  61 00 10 00 e0 00 01 c6 43 6a c8 40 08  4d+21:32:31.932  WRITE FPDMA QUEUED
  61 00 10 00 d8 00 01 c6 42 a6 68 40 08  4d+21:32:31.932  WRITE FPDMA QUEUED
  61 00 08 00 20 00 01 c5 c3 72 00 40 08  4d+21:32:31.932  WRITE FPDMA QUEUED
  61 00 08 00 18 00 01 c0 44 4f 60 40 08  4d+21:32:31.932  WRITE FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     20830         -
# 2  Short offline       Completed without error       00%     20821         -
# 3  Short offline       Completed without error       00%     20820         -
# 4  Extended offline    Completed without error       00%     16260         -
# 5  Short offline       Completed without error       00%     16201         -
# 6  Short offline       Completed without error       00%       579         -

Selective Self-tests/Logging not supported

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
Device State:                        Active (0)
Current Temperature:                    31 Celsius
Power Cycle Min/Max Temperature:     20/37 Celsius
Lifetime    Min/Max Temperature:     18/57 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/65 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (399)

Index    Estimated Time   Temperature Celsius
 400    2024-11-22 17:16    31  ************
 ...    ..( 47 skipped).    ..  ************
 448    2024-11-22 18:04    31  ************
 449    2024-11-22 18:05    32  *************
 ...    ..(173 skipped).    ..  *************
 145    2024-11-22 20:59    32  *************
 146    2024-11-22 21:00    33  **************
 147    2024-11-22 21:01    32  *************
 ...    ..( 22 skipped).    ..  *************
 170    2024-11-22 21:24    32  *************
 171    2024-11-22 21:25    33  **************
 ...    ..(  3 skipped).    ..  **************
 175    2024-11-22 21:29    33  **************
 176    2024-11-22 21:30    32  *************
 177    2024-11-22 21:31    32  *************
 178    2024-11-22 21:32    33  **************
 ...    ..( 10 skipped).    ..  **************
 189    2024-11-22 21:43    33  **************
 190    2024-11-22 21:44    32  *************
 191    2024-11-22 21:45    32  *************
 192    2024-11-22 21:46    33  **************
 193    2024-11-22 21:47    32  *************
 194    2024-11-22 21:48    32  *************
 195    2024-11-22 21:49    32  *************
 196    2024-11-22 21:50    33  **************
 197    2024-11-22 21:51    33  **************
 198    2024-11-22 21:52    32  *************
 ...    ..(155 skipped).    ..  *************
 354    2024-11-23 00:28    32  *************
 355    2024-11-23 00:29    31  ************
 ...    ..(  4 skipped).    ..  ************
 360    2024-11-23 00:34    31  ************
 361    2024-11-23 00:35    32  *************
 362    2024-11-23 00:36    31  ************
 ...    ..( 36 skipped).    ..  ************
 399    2024-11-23 01:13    31  ************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              64  ---  Lifetime Power-On Resets
0x01  0x010  4           20879  ---  Power-on Hours
0x01  0x018  6     34560115016  ---  Logical Sectors Written
0x01  0x020  6      1620968669  ---  Number of Write Commands
0x01  0x028  6     72299307123  ---  Logical Sectors Read
0x01  0x030  6       161249870  ---  Number of Read Commands
0x01  0x038  6      2149955968  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4           20754  ---  Spindle Motor Power-on Hours
0x03  0x010  4           20739  ---  Head Flying Hours
0x03  0x018  4             103  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               1  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               0  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4              33  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               4  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              31  ---  Current Temperature
0x05  0x010  1              31  ---  Average Short Term Temperature
0x05  0x018  1              32  ---  Average Long Term Temperature
0x05  0x020  1              57  ---  Highest Temperature
0x05  0x028  1              23  ---  Lowest Temperature
0x05  0x030  1              53  ---  Highest Average Short Term Temperature
0x05  0x038  1              30  ---  Lowest Average Short Term Temperature
0x05  0x040  1              48  ---  Highest Average Long Term Temperature
0x05  0x048  1              32  ---  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              65  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4             418  ---  Number of Hardware Resets
0x06  0x010  4             199  ---  Number of ASR Events
0x06  0x018  4               1  ---  Number of Interface CRC Errors
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            8  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           10  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       212637  Vendor specific

Second Disk

sdb

root@HomeServer[~]#  smartctl -x /dev/sdb
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.32-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFZX-68AWUN0
Serial Number:    WD-WX22DC1RND3T
LU WWN Device Id: 5 0014ee 26a5555e1
Firmware Version: 81.00B81
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Nov 23 01:16:23 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (42600) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 452) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   245   221   021    -    2725
  4 Start_Stop_Count        -O--CK   100   100   000    -    59
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   072   072   000    -    20872
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    58
192 Power-Off_Retract_Count -O--CK   200   200   000    -    27
193 Load_Cycle_Count        -O--CK   200   200   000    -    64
194 Temperature_Celsius     -O---K   118   094   000    -    32
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    1
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      78  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 3
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 3 [2] occurred at disk power-on lifetime: 16237 hours (676 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 80 00 00 a3 a4 27 1a 40 00  Error: UNC at LBA = 0xa3a4271a = 2745444122

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 07 e8 00 e8 00 00 a3 a4 31 b8 40 08     00:13:34.928  READ FPDMA QUEUED
  60 07 e8 00 18 00 00 a3 a4 29 d0 40 08     00:13:34.927  READ FPDMA QUEUED
  60 07 e8 00 10 00 00 a3 a4 21 e8 40 08     00:13:34.924  READ FPDMA QUEUED
  60 07 e8 00 e0 00 00 a3 a4 1a 00 40 08     00:13:34.924  READ FPDMA QUEUED
  60 07 e8 00 38 00 00 a3 a4 0a 30 40 08     00:13:34.916  READ FPDMA QUEUED

Error 2 [1] occurred at disk power-on lifetime: 7720 hours (321 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 01 ba 47 08 c8 40 00  Error: IDNF at LBA = 0x1ba4708c8 = 7420184776

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 08 00 70 00 01 b0 74 bb 18 40 08 31d+14:30:19.613  WRITE FPDMA QUEUED
  61 00 08 00 68 00 01 ba 47 08 c8 40 08 31d+14:30:19.613  WRITE FPDMA QUEUED
  ea 00 00 00 00 00 00 00 00 00 00 a0 08 31d+14:30:16.568  FLUSH CACHE EXT
  61 00 08 00 48 00 01 d1 c0 be 08 40 08 31d+14:30:16.566  WRITE FPDMA QUEUED
  61 00 08 00 40 00 01 d1 c0 bc 08 40 08 31d+14:30:16.566  WRITE FPDMA QUEUED

Error 1 [0] occurred at disk power-on lifetime: 3998 hours (166 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 01 c7 c0 dd 18 40 00  Error: IDNF at LBA = 0x1c7c0dd18 = 7646272792

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 08 00 90 00 01 c7 c0 dd 18 40 08  4d+21:32:30.434  WRITE FPDMA QUEUED
  61 00 08 00 68 00 01 c7 45 0f 48 40 08  4d+21:32:30.434  WRITE FPDMA QUEUED
  61 00 10 00 88 00 01 c6 c2 61 50 40 08  4d+21:32:30.434  WRITE FPDMA QUEUED
  61 00 10 00 e0 00 01 c6 43 6a c8 40 08  4d+21:32:30.434  WRITE FPDMA QUEUED
  61 00 10 00 40 00 01 c9 46 97 10 40 08  4d+21:32:30.434  WRITE FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     20831         -
# 2  Short offline       Completed without error       00%     20815         -
# 3  Extended offline    Interrupted (host reset)      90%     16240         -
# 4  Short offline       Completed without error       00%     16194         -
# 5  Short offline       Completed without error       00%       579         -

Selective Self-tests/Logging not supported

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
Device State:                        Active (0)
Current Temperature:                    33 Celsius
Power Cycle Min/Max Temperature:     20/36 Celsius
Lifetime    Min/Max Temperature:     18/56 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/65 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (460)

Index    Estimated Time   Temperature Celsius
 461    2024-11-22 17:19    32  *************
 ...    ..( 29 skipped).    ..  *************
  13    2024-11-22 17:49    32  *************
  14    2024-11-22 17:50    33  **************
 ...    ..(190 skipped).    ..  **************
 205    2024-11-22 21:01    33  **************
 206    2024-11-22 21:02    34  ***************
 ...    ..(  8 skipped).    ..  ***************
 215    2024-11-22 21:11    34  ***************
 216    2024-11-22 21:12    33  **************
 217    2024-11-22 21:13    34  ***************
 ...    ..( 30 skipped).    ..  ***************
 248    2024-11-22 21:44    34  ***************
 249    2024-11-22 21:45    33  **************
 250    2024-11-22 21:46    34  ***************
 ...    ..( 10 skipped).    ..  ***************
 261    2024-11-22 21:57    34  ***************
 262    2024-11-22 21:58    33  **************
 263    2024-11-22 21:59    33  **************
 264    2024-11-22 22:00    34  ***************
 265    2024-11-22 22:01    34  ***************
 266    2024-11-22 22:02    34  ***************
 267    2024-11-22 22:03    33  **************
 ...    ..(163 skipped).    ..  **************
 431    2024-11-23 00:47    33  **************
 432    2024-11-23 00:48    32  *************
 433    2024-11-23 00:49    33  **************
 434    2024-11-23 00:50    32  *************
 ...    ..(  3 skipped).    ..  *************
 438    2024-11-23 00:54    32  *************
 439    2024-11-23 00:55    33  **************
 440    2024-11-23 00:56    33  **************
 441    2024-11-23 00:57    32  *************
 442    2024-11-23 00:58    33  **************
 443    2024-11-23 00:59    33  **************
 444    2024-11-23 01:00    32  *************
 445    2024-11-23 01:01    33  **************
 446    2024-11-23 01:02    32  *************
 447    2024-11-23 01:03    32  *************
 448    2024-11-23 01:04    33  **************
 ...    ..(  5 skipped).    ..  **************
 454    2024-11-23 01:10    33  **************
 455    2024-11-23 01:11    32  *************
 ...    ..(  4 skipped).    ..  *************
 460    2024-11-23 01:16    32  *************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              58  ---  Lifetime Power-On Resets
0x01  0x010  4           20872  ---  Power-on Hours
0x01  0x018  6     34690364214  ---  Logical Sectors Written
0x01  0x020  6      1619211431  ---  Number of Write Commands
0x01  0x028  6     72302221588  ---  Logical Sectors Read
0x01  0x030  6       162749786  ---  Number of Read Commands
0x01  0x038  6      2124755968  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4           20748  ---  Spindle Motor Power-on Hours
0x03  0x010  4           20741  ---  Head Flying Hours
0x03  0x018  4              92  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               0  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4              27  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               3  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              32  ---  Current Temperature
0x05  0x010  1              32  ---  Average Short Term Temperature
0x05  0x018  1              34  ---  Average Long Term Temperature
0x05  0x020  1              56  ---  Highest Temperature
0x05  0x028  1              23  ---  Lowest Temperature
0x05  0x030  1              52  ---  Highest Average Short Term Temperature
0x05  0x038  1              30  ---  Lowest Average Short Term Temperature
0x05  0x040  1              48  ---  Highest Average Long Term Temperature
0x05  0x048  1              34  ---  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              65  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4             411  ---  Number of Hardware Resets
0x06  0x010  4             199  ---  Number of ASR Events
0x06  0x018  4               1  ---  Number of Interface CRC Errors
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            8  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           10  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       212799  Vendor specific

Third Disk

sdc

root@HomeServer[~]#  smartctl -x /dev/sdc
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.32-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFZX-68AWUN0
Serial Number:    WD-WX22DC1RNZ4D
LU WWN Device Id: 5 0014ee 215002892
Firmware Version: 81.00B81
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Nov 23 01:21:18 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (42900) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 455) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   233   223   021    -    3308
  4 Start_Stop_Count        -O--CK   100   100   000    -    59
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   072   072   000    -    20833
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    58
192 Power-Off_Retract_Count -O--CK   200   200   000    -    26
193 Load_Cycle_Count        -O--CK   200   200   000    -    72
194 Temperature_Celsius     -O---K   117   095   000    -    33
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    2
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      78  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 1
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 [0] occurred at disk power-on lifetime: 3998 hours (166 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 01 c6 43 6a c8 40 00  Error: IDNF at LBA = 0x1c6436ac8 = 7621274312

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 10 00 40 00 01 c6 43 6a c8 40 08  4d+21:32:29.436  WRITE FPDMA QUEUED
  61 00 10 00 38 00 01 c6 42 a6 68 40 08  4d+21:32:29.436  WRITE FPDMA QUEUED
  61 00 08 00 30 00 01 c5 c3 72 00 40 08  4d+21:32:29.436  WRITE FPDMA QUEUED
  61 00 08 00 80 00 01 c0 44 4f 60 40 08  4d+21:32:29.436  WRITE FPDMA QUEUED
  61 00 10 00 78 00 01 c9 46 97 08 40 08  4d+21:32:29.436  WRITE FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     20801         -
# 2  Short offline       Completed without error       00%     20775         -
# 3  Short offline       Completed without error       00%     16149         -
# 4  Short offline       Completed without error       00%     15715         -
# 5  Short offline       Completed without error       00%      6351         -
# 6  Short offline       Completed without error       00%       579         -

Selective Self-tests/Logging not supported

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
Device State:                        Active (0)
Current Temperature:                    33 Celsius
Power Cycle Min/Max Temperature:     20/36 Celsius
Lifetime    Min/Max Temperature:     18/55 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/65 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (22)

Index    Estimated Time   Temperature Celsius
  23    2024-11-22 17:24    33  **************
 ...    ..( 74 skipped).    ..  **************
  98    2024-11-22 18:39    33  **************
  99    2024-11-22 18:40    34  ***************
 ...    ..(  3 skipped).    ..  ***************
 103    2024-11-22 18:44    34  ***************
 104    2024-11-22 18:45    33  **************
 ...    ..(  5 skipped).    ..  **************
 110    2024-11-22 18:51    33  **************
 111    2024-11-22 18:52    34  ***************
 ...    ..(278 skipped).    ..  ***************
 390    2024-11-22 23:31    34  ***************
 391    2024-11-22 23:32    33  **************
 ...    ..(108 skipped).    ..  **************
  22    2024-11-23 01:21    33  **************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              58  ---  Lifetime Power-On Resets
0x01  0x010  4           20833  ---  Power-on Hours
0x01  0x018  6     33650421625  ---  Logical Sectors Written
0x01  0x020  6      1579500897  ---  Number of Write Commands
0x01  0x028  6     70096659113  ---  Logical Sectors Read
0x01  0x030  6       156581257  ---  Number of Read Commands
0x01  0x038  6      1984355968  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4           20707  ---  Spindle Motor Power-on Hours
0x03  0x010  4           20692  ---  Head Flying Hours
0x03  0x018  4              99  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               1  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               0  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4              26  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               1  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              33  ---  Current Temperature
0x05  0x010  1              32  ---  Average Short Term Temperature
0x05  0x018  1              34  ---  Average Long Term Temperature
0x05  0x020  1              55  ---  Highest Temperature
0x05  0x028  1              23  ---  Lowest Temperature
0x05  0x030  1              52  ---  Highest Average Short Term Temperature
0x05  0x038  1              31  ---  Lowest Average Short Term Temperature
0x05  0x040  1              48  ---  Highest Average Long Term Temperature
0x05  0x048  1              34  ---  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              65  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4             412  ---  Number of Hardware Resets
0x06  0x010  4             198  ---  Number of ASR Events
0x06  0x018  4               2  ---  Number of Interface CRC Errors
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            8  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           10  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       213096  Vendor specific
```
root@HomeServer[~]# 

Forth Disk

Summary

root@HomeServer[~]#  smartctl -x /dev/sdd
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.32-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFZX-68AWUN0
Serial Number:    WD-WX22DC1DL4D6
LU WWN Device Id: 5 0014ee 2bfab52f8
Firmware Version: 81.00B81
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Nov 23 01:23:22 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (42900) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 455) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   232   221   021    -    3391
  4 Start_Stop_Count        -O--CK   100   100   000    -    58
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   072   072   000    -    20882
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    57
192 Power-Off_Retract_Count -O--CK   200   200   000    -    25
193 Load_Cycle_Count        -O--CK   200   200   000    -    72
194 Temperature_Celsius     -O---K   117   095   000    -    33
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    2
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      78  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 1
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 [0] occurred at disk power-on lifetime: 15719 hours (654 days + 23 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 94 16 97 b8 40 00  Error: UNC at LBA = 0x941697b8 = 2484508600

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 20 00 a8 00 01 c5 40 99 b0 40 08  8d+10:07:02.787  READ FPDMA QUEUED
  61 00 08 00 a0 00 01 bf 2e c8 00 40 08  8d+10:07:02.293  WRITE FPDMA QUEUED
  61 00 08 00 98 00 01 cb 1d 0d 48 40 08  8d+10:07:02.293  WRITE FPDMA QUEUED
  60 00 38 00 90 00 01 08 cd b1 28 40 08  8d+10:06:59.378  READ FPDMA QUEUED
  60 00 38 00 08 00 01 98 40 7f 08 40 08  8d+10:06:59.250  READ FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     20859         -
# 2  Short offline       Completed without error       00%     20824         -
# 3  Short offline       Completed without error       00%     16197         -
# 4  Short offline       Completed without error       00%      4293         -
# 5  Short offline       Completed without error       00%       579         -

Selective Self-tests/Logging not supported

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
Device State:                        Active (0)
Current Temperature:                    33 Celsius
Power Cycle Min/Max Temperature:     20/36 Celsius
Lifetime    Min/Max Temperature:     18/55 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/65 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (53)

Index    Estimated Time   Temperature Celsius
  54    2024-11-22 17:26    32  *************
 ...    ..( 20 skipped).    ..  *************
  75    2024-11-22 17:47    32  *************
  76    2024-11-22 17:48    33  **************
 ...    ..(194 skipped).    ..  **************
 271    2024-11-22 21:03    33  **************
 272    2024-11-22 21:04    34  ***************
 ...    ..( 75 skipped).    ..  ***************
 348    2024-11-22 22:20    34  ***************
 349    2024-11-22 22:21    33  **************
 ...    ..(175 skipped).    ..  **************
  47    2024-11-23 01:17    33  **************
  48    2024-11-23 01:18    32  *************
 ...    ..(  4 skipped).    ..  *************
  53    2024-11-23 01:23    32  *************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              57  ---  Lifetime Power-On Resets
0x01  0x010  4           20882  ---  Power-on Hours
0x01  0x018  6     33855739859  ---  Logical Sectors Written
0x01  0x020  6      1586265262  ---  Number of Write Commands
0x01  0x028  6     71588762464  ---  Logical Sectors Read
0x01  0x030  6       166367554  ---  Number of Read Commands
0x01  0x038  6      2160755968  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4           20755  ---  Spindle Motor Power-on Hours
0x03  0x010  4           20740  ---  Head Flying Hours
0x03  0x018  4              98  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               5  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               0  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4              25  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               1  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              33  ---  Current Temperature
0x05  0x010  1              32  ---  Average Short Term Temperature
0x05  0x018  1              34  ---  Average Long Term Temperature
0x05  0x020  1              55  ---  Highest Temperature
0x05  0x028  1              23  ---  Lowest Temperature
0x05  0x030  1              52  ---  Highest Average Short Term Temperature
0x05  0x038  1              32  ---  Lowest Average Short Term Temperature
0x05  0x040  1              47  ---  Highest Average Long Term Temperature
0x05  0x048  1              34  ---  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              65  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4             413  ---  Number of Hardware Resets
0x06  0x010  4             199  ---  Number of ASR Events
0x06  0x018  4               2  ---  Number of Interface CRC Errors
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            8  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           10  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       213223  Vendor specific

[Protopia]

I can’t see any issues with these drives based on a quick review. No reallocated sectors, few CRC and other errors all of which are old.

I would recommend that you schedule a SMART short test weekly and a SMART Long test quarterly or monthly (and these can run on all disks at the same time).

I also recommend that you implement @joeschmuck’s Multi-Report script.

[smic717394]

@Protopia

I can’t see any issues with these drives based on a quick review. No reallocated sectors, few CRC and other errors all of which are old.

Thank you friend. Can it ve something else, like the power supply, cables or maybe the connectors on the case where the drives slide in?

I sometimes get errors the ups, thar recover quick.

Summary

The following alert has been cleared: UPS ups is on battery power.

UPS Statistics: 'ups'

Statistics recovered:

1) Battery charge (percent)
    battery.charge: 100

2) Battery level remaining (percent) when UPS switches to Low Battery (LB)
    battery.charge.low: 20

3) Battery runtime (seconds)
    battery.runtime: 2250

[smic717394]

@Protopia

I cannot find the MB you specified - is it A1SRi-2758F or something else?

Yes A1SRi-2758F

@Protopia

What exact Atom processor do you have? C2758?

Yes 2758

@Protopia

Are you using the PCIe slot(s)?

No, I use the 4 sata2 ports for the 4 drive

@Protopia

You do realise that whilst these MBs have 6x SATA ports, only 2 are SATA 3 with the other 4 being SATA 2? Since you are using RAIDZ and I/O goes to all ports, this is effectively equivalent to all ports being SATA2.

I use the 4 drives in the 4 sata 2. The 2 sata 3 are free. The boot pool, i have in on a nvmi in a enclosure connected to the internal usb3 port.

@Protopia

I am a bit worried by some of what you said: “Im running the latest version of truenas scale” suggests you are running 24.10

I’m on Dragonfish-24.04.2.5

@Protopia

Finally, and most importantly, do you have a backup of your system configuration file? If not take one now!!

I do have, the one I made not long when i updated the os.

@Protopia

P.S. Once we have it back up and running you should think about a small SSD to hold your apps and their data.

I like this idea, shoud i get 2 and mirror them? How easy is to move the apps to the new ssd pool?

I’m sure that sata 2 its ok fot the mechanical drive, I do have a IBM HBA/LSI SAS2 6 Gigabit pci card that i dont use, if this is the problem, i could use the pci card .
The case its a chep Eolize SVD-NC11-4 Mini ITX, I think was 120€ with the PSU 250W 80 plus bronze includes.
Fot the cables I bouth Corsair Premium Sleeved SATA 6Gbps 30cm, they were expensive.

[Protopia]

@smic717394

Thank you friend. Can it ve something else, like the power supply, cables or maybe the connectors on the case where the drives slide in?

Possibly - but I suspect that these would likely show up as CRC errors, and you don’t seem to having frequent enough problems and you are using quality cables and a PSU that should easily be big enough.

@smic717394

I sometimes get errors the ups, thar recover quick.

If you have a UPS and have a short power brownout or interruption, the UPS should be continuing to provide good quality power to the server - that is its purpose.

It is possible that if the UPS isn’t doing its job correctly that my create these symptoms, but unless you can correlate disk errors to UPS cutting in, then it could easily be something else.

[smic717394]

@Protopia

I also recommend that you implement @joeschmuck’s Multi-Report script.

I install the multi report and the basic configuration, email and path and look great, I got the inicial tests show smart test results pass on all drives but last test age show different some yellow, i guess because i run the test different days.

A bit scarry receiving every day emais starting warning, but…

This script doesn’t actually run any test, but just email the smart data right?

I have to schedule a short test weekly ang long monthly.

I set the cron for the script as every day, is point to have this run daily if the smart runs weekly and monthly?

[Protopia]

Yes you need to schedule the smart tests weekly and monthly separately, and run this script daily.

Once you are comfortable with it you can get it to email you once a week and also if there is anything going wrong.

There are tons of settings - and one of them is how long between SMART tests.

[smic717394]

@Protopia

nce you are comfortable with it you can get it to email you once a week and also if there is anything going wrong.

There are tons of settings - and one of them is how long between SMART tests

This topic has been resolved now. I just have to figure out how to run the cron for long smart every sunday once a month. i tried few things I found on the forum, no vail. For now I set every 1 of every month, I dont like it much, because may coincide with the scrub task that I it runs every sunday 12:00AM with a Threshold Days 35 days.

Thank you very much @Farout , @Protopia and @etorix for your help and patience and time.

[etorix]

@smic717394

I dont like it much, because may coincide with the scrub task that I it runs every sunday 12:00AM with a Threshold Days 35 days.

Run the long SMART on Saturday 12:00am with 35 day threshold?

[smic717394]

@etorix

threshold

No, there is no threshold on smart, I run the scrub task sunday at 12 AM with a 35 day threshold.
I have the multi-report/multi_report.sh everyday at 2:00 AM
I have Short Smart test - at 02:00, only on Monday (weekly)
I have Long Smart Disk - at 01:00, on day 28 of the month

[smic717394]

Hi guys, it happened again, since November, no problem with the pool or drives, I had installed Multi-Report and so far, no problem. Few days ago I got another warning email that the pool was degraded, this time disc sdc and sdd IO errors, last time was sda and sdb, I know this drive name can change between the drive, but I have checked the serial number and they not the same.

I had to turn off, reconnect all the cables on the back of the each of the 4 hot swap connectors, turn it back on follow again all this steps and got it back to work. It’s been a week and so far ok.

But this time I got one of the drive sbc with a Pending Sector Error. I did run long smart test on every drive, 3 times during 3 days on the sdc and it passes the tests. But now I get every few hours an alert that this drive has a pending error on the log.

I don’t know if the drive is bad, if the connectors on the case, or because I have the nas on the desk and maybe movements ot vibrations affect it, but the drives have only 3 years and not that much use really, backup nextcloud, and plex really.

Question, is there any way of clearing this error o relocate this sector. The drives are WD40EFZX. I’m thinking to get one drive to have on stand by, but I don’t know if to get segate or wd again. I have this kind or errors basically after one year of use, usually turn it off removing the drive and reconnect them and then turn it back on clear the errors.

Critical

Device: /dev/sdc [SAT], 1 Currently unreadable (pending) sectors.

I´m thinking to remove the hot swap UNas connectors and connect the sata and power directly to the drive for a period of time, just to confirm the problem are not the bays, or move the nas from the desk.

[etorix]

This is an internal error of the drive; moving the drive around will not help.

[smic717394]

I know this pending error is drive related. But this happend after the las io errors. Every six month more or less i used to get io error on 2 deives, usually sda and sdb.

The error usually would be clear by power cycle or removeing and reinserting rhe drive . This time I got this pending sector, althow the drive is passing the smart test.

I was saying to move things around because I belive this IO errors are triggered by something else, I dont think the drive are the culpid.

Can I relocate this drive I know in windows you run a chkdsk an will relocate the sector.

If I need to change the drive, should I go with WD again or change to Segate. I read the Segate support support is bad and they fail more than WD. Toshiba also kook good, peopleday they ladt longer.