SMART errors on multiple disks

DanP · December 10, 2024, 2:00pm

Hi,

in the recent months I started get SMART errors.

Device: /dev/da2 [SAT], Self-Test Log error count increased from 4 to 5.
Device: /dev/da5 [SAT], Self-Test Log error count increased from 0 to 1.

I have a RAIDZ2 with 6 x 6TB WD Red disks.

First da5 started with log erros. It had around 35000 hrs. I replaced it with a brand new one (tested for bad blocks before).
Then da2 started as well with erros (also ~35k hrs)
And now the new da5 disk (3236 hrs) had it’s first error as well.

Storage is around 80%.

Could this be jsut a bad coincidence with the disks? Or could it be the HBA controller (Dell H310), or cable as well?

Thanks

NugentS · December 10, 2024, 5:07pm

Exactly what disk models do you have?
Have you run long smart tests on the drives recently and what are the results?
Post your hardware please - so we know what we are dealing with

etorix · December 10, 2024, 6:02pm

Questions for the usual suspects:
zpool status -v
camcontrol devlist
sas2flash -list
smartctl -a /dev/da2 and da5… and all others as well; we want to know which kind of error it was

Results as preformatted text, please. Use </> button or paste the outputs between lines of three backquotes:
```
output
```

DanP · December 11, 2024, 8:27am

Thanks for the quick replies, I am back in the office now.

System spec:

Supermicro X9SCM-F
Xeon E3-1230v2
Kingston 4x8 GB ECC
Dell H310 1x
Boot: 2 x Crucial BX500 120 GB
Storage:
– 5 x WD60EFRX
– 1 x WD60EFZX (new disk)

I run short smart tests every 7 days and long smart tests every 14 days.

zpool status -v

root@truenas[~]# zpool status -v
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
	The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:08 with 0 errors on Fri Dec  6 03:45:08 2024
config:

	NAME        STATE     READ WRITE CKSUM
	boot-pool   ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    ada0p2  ONLINE       0     0     0
	    ada1p2  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub repaired 0B in 12:32:17 with 0 errors on Sun Dec  8 12:32:21 2024
config:

	NAME                                            STATE     READ WRITE CKSUM
	tank                                            ONLINE       0     0     0
	  raidz2-0                                      ONLINE       0     0     0
	    gptid/038117ff-8ceb-11eb-bb67-002590aaacc7  ONLINE       1     0     0
	    gptid/0487cea5-8ceb-11eb-bb67-002590aaacc7  ONLINE       0     1     0
	    gptid/04ae7f80-8ceb-11eb-bb67-002590aaacc7  ONLINE       0     0     0
	    gptid/5b2a6c96-4fff-11ef-8285-002590aaacc7  ONLINE       0     0     0
	    gptid/054422f3-8ceb-11eb-bb67-002590aaacc7  ONLINE       1     0     0
	    gptid/055f51ed-8ceb-11eb-bb67-002590aaacc7  ONLINE       0     1     0

errors: No known data errors

camcontrol devlist

root@truenas[~]# camcontrol devlist
<ATA WDC WD60EFRX-68M 0A82>        at scbus0 target 0 lun 0 (pass0,da0)
<ATA WDC WD60EFRX-68M 0A82>        at scbus0 target 2 lun 0 (pass1,da1)
<ATA WDC WD60EFRX-68M 0A82>        at scbus0 target 3 lun 0 (pass2,da2)
<ATA WDC WD60EFRX-68M 0A82>        at scbus0 target 4 lun 0 (pass3,da3)
<ATA WDC WD60EFRX-68M 0A82>        at scbus0 target 5 lun 0 (pass4,da4)
<ATA WDC WD60EFZX-68B 0A81>        at scbus0 target 7 lun 0 (pass5,da5)
<CT120BX500SSD1 M6CR013>           at scbus1 target 0 lun 0 (ada0,pass6)
<CT120BX500SSD1 M6CR013>           at scbus2 target 0 lun 0 (ada1,pass7)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus7 target 0 lun 0 (ses0,pass8)

sas2flash -list

root@truenas[~]# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

	Adapter Selected is a LSI SAS: SAS2008(B2)

	Controller Number              : 0
	Controller                     : SAS2008(B2)
	PCI Address                    : 00:01:00:00
	SAS Address                    : 5000000-0-1cda-2f71
	NVDATA Version (Default)       : 14.01.00.08
	NVDATA Version (Persistent)    : 14.01.00.08
	Firmware Product ID            : 0x2213 (IT)
	Firmware Version               : 20.00.07.00
	NVDATA Vendor                  : LSI
	NVDATA Product ID              : SAS9211-8i
	BIOS Version                   : N/A
	UEFI BSD Version               : N/A
	FCODE Version                  : N/A
	Board Name                     : 6Gbps SAS HBA
	Board Assembly                 : N/A
	Board Tracer Number            : N/A

	Finished Processing Commands Successfully.
	Exiting SAS2Flash.

SMART outputs:
da2:

root@truenas[~]# smartctl -a /dev/da2
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD60EFRX-68MYMN1
Serial Number:    WD-WX21DC4D5ND6
LU WWN Device Id: 5 0014ee 2b646e5c4
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Dec 11 09:18:12 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection: 		( 3824) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 692) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   195   192   021    Pre-fail  Always       -       9241
  4 Start_Stop_Count        0x0032   099   099   000    Old_age   Always       -       1535
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   052   052   000    Old_age   Always       -       35294
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       689
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       106
193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       3854
194 Temperature_Celsius     0x0022   124   093   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     35205         -
# 2  Short offline       Completed without error       00%     35144         -
# 3  Short offline       Completed without error       00%     34976         -
# 4  Extended offline    Completed without error       00%     34822         -
# 5  Short offline       Completed without error       00%     34641         -
# 6  Extended offline    Completed: read failure       90%     34473         402424368
# 7  Short offline       Completed without error       00%     34305         -
# 8  Short offline       Completed without error       00%     34136         -
# 9  Extended offline    Completed: read failure       90%     34065         402424368
#10  Short offline       Completed without error       00%     33969         -
#11  Short offline       Completed without error       00%     33801         -
#12  Extended offline    Completed: read failure       90%     33729         402424368
#13  Short offline       Completed without error       00%     33633         -
#14  Short offline       Completed without error       00%     33465         -
#15  Extended offline    Completed: read failure       90%     33346         402424368
#16  Short offline       Completed without error       00%     33297         -
#17  Short offline       Completed without error       00%     33130         -
#18  Extended offline    Completed: read failure       90%     33010         402424368
#19  Short offline       Completed without error       00%     32962         -
#20  Short offline       Completed without error       00%     32794         -
#21  Short offline       Completed without error       00%     32626         -
5 of 5 failed self-tests are outdated by newer successful extended offline self-test # 1

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

da5:

root@truenas[~]# smartctl -a /dev/da5
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD60EFZX-68B3FN0
Serial Number:    WD-C80NDUNG
LU WWN Device Id: 5 0014ee 2be72f7da
Firmware Version: 81.00A81
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5640 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Dec 11 09:19:15 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 114)	The previous self-test completed having
					the read element of the test failed.
Total time to complete Offline
data collection: 		(61020) seconds.
Offline data collection
capabilities: 			 (0x11) SMART execute Offline immediate.
					No Auto Offline data collection support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					No Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 645) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   188   188   021    Pre-fail  Always       -       7600
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       7
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   096   096   000    Old_age   Always       -       3330
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       7
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       2
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       11
194 Temperature_Celsius     0x0022   125   112   000    Old_age   Always       -       27
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       20%      3236         925823104
# 2  Short offline       Completed without error       00%      3180         -
# 3  Short offline       Completed without error       00%      3012         -
# 4  Extended offline    Completed without error       00%      2856         -
# 5  Short offline       Completed without error       00%      2676         -
# 6  Extended offline    Completed without error       00%      2520         -
# 7  Short offline       Completed without error       00%      2341         -
# 8  Short offline       Completed without error       00%      2172         -
# 9  Extended offline    Completed without error       00%      2111         -
#10  Short offline       Completed without error       00%      2004         -
#11  Short offline       Completed without error       00%      1836         -
#12  Extended offline    Completed without error       00%      1776         -
#13  Short offline       Completed without error       00%      1669         -
#14  Short offline       Completed without error       00%      1501         -
#15  Extended offline    Completed without error       00%      1392         -
#16  Short offline       Completed without error       00%      1333         -
#17  Short offline       Completed without error       00%      1165         -
#18  Extended offline    Completed without error       00%      1056         -
#19  Short offline       Completed without error       00%       997         -
#20  Short offline       Completed without error       00%       829         -
#21  Short offline       Completed without error       00%       662         -

Selective Self-tests/Logging not supported

ALL others:

root@truenas[~]# smartctl -a /dev/da0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD60EFRX-68MYMN1
Serial Number:    WD-WX21DC4D5ARA
LU WWN Device Id: 5 0014ee 260f14415
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Dec 11 09:21:50 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection: 		( 4784) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 702) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   204   203   021    Pre-fail  Always       -       8800
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2231
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   049   049   000    Old_age   Always       -       37948
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       966
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       121
193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       5342
194 Temperature_Celsius     0x0022   123   095   000    Old_age   Always       -       29
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     37859         -
# 2  Short offline       Completed without error       00%     37798         -
# 3  Short offline       Completed without error       00%     37631         -
# 4  Extended offline    Completed without error       00%     37476         -
# 5  Short offline       Completed without error       00%     37295         -
# 6  Extended offline    Completed without error       00%     37141         -
# 7  Short offline       Completed without error       00%     36959         -
# 8  Short offline       Completed without error       00%     36791         -
# 9  Extended offline    Completed without error       00%     36732         -
#10  Short offline       Completed without error       00%     36623         -
#11  Short offline       Completed without error       00%     36455         -
#12  Extended offline    Completed without error       00%     36396         -
#13  Short offline       Completed without error       00%     36287         -
#14  Short offline       Completed without error       00%     36119         -
#15  Extended offline    Completed without error       00%     36012         -
#16  Short offline       Completed without error       00%     35951         -
#17  Short offline       Completed without error       00%     35784         -
#18  Extended offline    Completed without error       00%     35676         -
#19  Short offline       Completed without error       00%     35616         -
#20  Short offline       Completed without error       00%     35448         -
#21  Short offline       Completed without error       00%     35280         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@truenas[~]# smartctl -a /dev/da1
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD60EFRX-68MYMN1
Serial Number:    WD-WX11DC449271
LU WWN Device Id: 5 0014ee 2b6420d91
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Dec 11 09:22:03 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection: 		( 2984) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 684) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   198   195   021    Pre-fail  Always       -       9075
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2781
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   046   046   000    Old_age   Always       -       39961
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1090
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       58
193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       6093
194 Temperature_Celsius     0x0022   122   100   000    Old_age   Always       -       30
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     39871         -
# 2  Short offline       Completed without error       00%     39811         -
# 3  Short offline       Completed without error       00%     39643         -
# 4  Extended offline    Completed without error       00%     39488         -
# 5  Short offline       Completed without error       00%     39307         -
# 6  Extended offline    Completed without error       00%     39152         -
# 7  Short offline       Completed without error       00%     38972         -
# 8  Short offline       Completed without error       00%     38803         -
# 9  Extended offline    Completed without error       00%     38744         -
#10  Short offline       Completed without error       00%     38635         -
#11  Short offline       Completed without error       00%     38467         -
#12  Extended offline    Completed without error       00%     38408         -
#13  Short offline       Completed without error       00%     38299         -
#14  Short offline       Completed without error       00%     38131         -
#15  Extended offline    Completed without error       00%     38024         -
#16  Short offline       Completed without error       00%     37964         -
#17  Short offline       Completed without error       00%     37796         -
#18  Extended offline    Completed without error       00%     37688         -
#19  Short offline       Completed without error       00%     37628         -
#20  Short offline       Completed without error       00%     37460         -
#21  Short offline       Completed without error       00%     37292         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@truenas[~]# smartctl -a /dev/da3
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD60EFRX-68MYMN1
Serial Number:    WD-WX11DA4DJ8C6
LU WWN Device Id: 5 0014ee 20b43b8ae
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Dec 11 09:22:34 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection: 		( 7784) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 731) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   197   194   021    Pre-fail  Always       -       9133
  4 Start_Stop_Count        0x0032   097   097   000    Old_age   Always       -       3132
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   044   044   000    Old_age   Always       -       41490
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1202
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       124
193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       3488
194 Temperature_Celsius     0x0022   124   102   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     41401         -
# 2  Short offline       Completed without error       00%     41340         -
# 3  Short offline       Completed without error       00%     41172         -
# 4  Extended offline    Completed without error       00%     41018         -
# 5  Short offline       Completed without error       00%     40836         -
# 6  Extended offline    Completed without error       00%     40682         -
# 7  Short offline       Completed without error       00%     40501         -
# 8  Short offline       Completed without error       00%     40332         -
# 9  Extended offline    Completed without error       00%     40274         -
#10  Short offline       Completed without error       00%     40164         -
#11  Short offline       Completed without error       00%     39996         -
#12  Extended offline    Completed without error       00%     39938         -
#13  Short offline       Completed without error       00%     39828         -
#14  Short offline       Completed without error       00%     39661         -
#15  Extended offline    Completed without error       00%     39554         -
#16  Short offline       Completed without error       00%     39493         -
#17  Short offline       Completed without error       00%     39325         -
#18  Extended offline    Completed without error       00%     39218         -
#19  Short offline       Completed without error       00%     39157         -
#20  Short offline       Completed without error       00%     38989         -
#21  Short offline       Completed without error       00%     38821         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@truenas[~]# smartctl -a /dev/da4
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD60EFRX-68MYMN1
Serial Number:    WD-WX11DC4FKD23
LU WWN Device Id: 5 0014ee 2b642329d
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Dec 11 09:23:19 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection: 		( 6164) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 715) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   196   192   021    Pre-fail  Always       -       9200
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2842
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   046   046   000    Old_age   Always       -       39959
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1161
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       130
193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       6416
194 Temperature_Celsius     0x0022   123   102   000    Old_age   Always       -       29
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
ATA Error Count: 2
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2 occurred at disk power-on lifetime: 8216 hours (342 days + 8 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 10 f8 84 20 e3  Error: UNC 16 sectors at LBA = 0x032084f8 = 52462840

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 10 f8 84 20 e3 08   1d+09:26:39.523  READ DMA
  e5 00 00 00 00 00 00 08   1d+09:26:39.392  CHECK POWER MODE
  ca 00 01 a0 2d 10 e0 08   1d+09:26:29.702  WRITE DMA

Error 1 occurred at disk power-on lifetime: 6070 hours (252 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  10 51 01 a0 2d 10 e0  Error: IDNF at LBA = 0x00102da0 = 1060256

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ca 00 01 a0 2d 10 e0 08   1d+13:04:01.653  WRITE DMA
  e5 00 00 00 00 00 00 08   1d+13:04:01.518  CHECK POWER MODE

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     39870         -
# 2  Short offline       Completed without error       00%     39809         -
# 3  Short offline       Completed without error       00%     39641         -
# 4  Extended offline    Completed without error       00%     39487         -
# 5  Short offline       Completed without error       00%     39306         -
# 6  Extended offline    Completed without error       00%     39152         -
# 7  Short offline       Completed without error       00%     38970         -
# 8  Short offline       Completed without error       00%     38801         -
# 9  Extended offline    Completed without error       00%     38743         -
#10  Short offline       Completed without error       00%     38634         -
#11  Short offline       Completed without error       00%     38466         -
#12  Extended offline    Completed without error       00%     38407         -
#13  Short offline       Completed without error       00%     38298         -
#14  Short offline       Completed without error       00%     38130         -
#15  Extended offline    Completed without error       00%     38023         -
#16  Short offline       Completed without error       00%     37962         -
#17  Short offline       Completed without error       00%     37794         -
#18  Extended offline    Completed without error       00%     37687         -
#19  Short offline       Completed without error       00%     37627         -
#20  Short offline       Completed without error       00%     37459         -
#21  Short offline       Completed without error       00%     37291         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

etorix · December 11, 2024, 10:42am

HBA on IT firmware, CMR Red Plus drives, raidz2, regular SMART tests.
This looks like da2 and da5 had errors in close succession in what is otherwise a healthy setup. Failed SMART tests justify replacing these drives—act fast if warranty is still valid. Run a scrub, and clear errors if all if fine.
(da4 had a SMART error a long time ago but passed its tests; it should be fine.)

DanP · December 11, 2024, 11:30am

Thanks @etorix.

I setup the NAS in 2021. Since then it ran fine.
I am surprised that 3 disks started failing within the last few months.
Especially the new disk da5 with only ~ 3000hrs. The first disk I replaced was da5 as well.
Unfortunately no more warranty, because the new disk has been has been lying around since 2021 as backup

Do you think it could be HBA or cables? I don’t wanna swap da5 and have the new disk failing soon as well.

etorix · December 11, 2024, 12:03pm

Failing a SMART test is an internal drive issue.
An overheating controller would give errors on all drives. You may change the cable if you have a doubt, or swap cables and see if issues follow the cable. But it looks like a coincidence in driv failures to me.