Feedback on faulted disk

[Fr33dan]

@joeschmuck

That is the only easy option I can see when someone has a huge amount of data.

So an idea I have but might be crazy (and definitely expensive) to reogranize without having to destroy and re-create everything is to do 2 expansions at once. Add 2 of the 4 drive VDEV’s I mentioned before, do a zpool remove on the 8 drive VDEV and let it evacuate the data to the new VDEVS and remaining space on the existing 4 drive VDEV. Then re-add the old drives in a 2x4 configuration. The end result would be a uniform pool of 4 drive VDEVS, but it just feels kind of hacky and off. Although the increased space kicks the expansion can much further down the road.

The big thing now is that I need more drive capacity as my MD1200 is full.

[etorix]

You can only remove mirror vdevs, not raidz.
So the only way to reconfigure the pool is Backup-Destroy-Restore.

[joeschmuck]

@etorix

Could it be that #198 is for WRITE errors, while these events were the mythical Uncorrectable READ Error, of “RAID 5 is dead” fame?

Very possible. I have not found any documentation that specifies if this is read/write/ or both. Your answer sounds very plausible.

[Fr33dan]

Oh dang, I see now in the man page how you can’t remove a top-level vdev if the primary pool has any top-level raidz vdev’s. Guess I missed that before.

[Fr33dan]

Just for a little closure, got the new drive in yesterday. Resilver finished overnight last night. Pool back to being healthy and I’m getting a multi-report once a day now.

Thanks y’all.

[bouncybrian]

Anything to do do with this?

“Clearly predatory”: Western Digital sparks panic, anger for age-shaming HDDs

[etorix]

The heading says it all: “Another NAS-ty WD controversy”.
Maybe a rebrand is in order? I propose: Wilfuly Deceptive. Or Wrong Drive.

[genfoch01]

My understanding is when a pre fail attribute reports over threshold then the disk is bad.

 ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
   1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
   3 Spin_Up_Time            0x0027   219   219   021    Pre-fail  Always       -       4050
   4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       56
   5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always   

[Stux]

@Fr33dan

The checksum errors are a longstanding issue that predates the HBA. I’ve always attributed the checksum errors to some kind of race condition where Steam updated a file in the middle of the scrub checking that specific file

The checksum issues should not happen. They indicate a hardware issue.

The same hardware issue could explain the drive issue.

Zfs should not have race conditions like that.

BUT, I wouldn’t 100% trust the uuid to drive mapping in the gui. It can be wrong when a drive is offline, as it uses cached values.

I suspect it may be that a different drive will show the smart failure you’re looking for.

[Fr33dan]

@genfoch01

My understanding is when a pre fail attribute reports over threshold then the disk is bad

I’m certainly not the expert, but I believe these numbers go down to indicate issue instead of up.

@Stux

I wouldn’t 100% trust the uuid to drive mapping in the gui.

That is good to know, luckily I did confirm it using lsblk before taking action.

@Stux

The same hardware issue could explain the drive issue.
[…]
I suspect it may be that a different drive will show the smart failure you’re looking for.

I starting writing this reply feel pretty confident that the two things are seperate but still thinking there’s potentially something else going on. The fact that it only occurs in this dataset as a game is updating feels like it can’t be a coincidence.

Maybe just a some piece of hardware flaking out when the number of IO operations peaks? It would have to predate the HBA, but also be in the mix with the HBA which rules out most components of the MOBO/CPU it’s a completely different path through to the CPU. Could be those things, but most like either memory or the drives. I did test the memory when I got it, but haven’t since. What if the drive was a lemon the whole time and only had issues under max load until total failure? Maybe I’ve talked myself into thinking it’s one issue after all.

The smart reports for all the other drives look very similar to the failed drive. Here is the text of the latest multi-report (although I have smart tests scheduled only once a week so the test info is a few days old):

Multi-Report Text Section
1) External Configuration File (Present) dtd:2025-02-25 
2) Statistical Data Log (Present) @ (/mnt/fr33dan-raid/Personal/Joseph/multi_report/statisticalsmartdata.csv)
 
Attachments:
1) TrueNAS Configuration File (Mon) - (Enabled)
2) Multi Report Configuration File (Mon) - (Enabled)
3) Statistical Log (Mon) - (Enabled)
4) HDD/SSD Partition Backup (Mon) - (Enabled)
5) Sendemail (v0.16)
 
Checks/Tests:
1) SMR Checking - (Enabled) - No Errors Detected
2) Partition Check - (Enabled) - No Errors Detected
3) Spencer - (Disabled)
4) S.M.A.R.T. Testing External File (v1.04) - (Enabled)
   a) Short Test Authorized Test Days (Mon, Tue, Wed, Thu, Fri, Sat, Sun) (~1 Drive(s) per day)
      (none)
   b) Long Test Authorized Test Days (Mon, Tue, Wed, Thu, Fri, Sat, Sun) (~1 Drive(s) per day)
      (none)
   c) A SCRUB or RESILVER is NOT in progress.
5) Seagate Drive SCAM Check - (Enabled)

WARNING LOG FILE
Drive: WD-WX42DB16NKJU - Test Age = 4 Days
Drive: WD-WX82DA1L5RZE - Test Age = 4 Days
Drive: WD-WX42DB1H9CC4 - Test Age = 4 Days
Drive: WD-WX32DB1ATDX9 - Test Age = 4 Days
Drive: WD-WX52DA3K3998 - Test Age = 5 Days
Drive: WD-WX72DA12ZH7V - Test Age = 4 Days
Drive: WD-WX52DA3KS5AV - Test Age = 5 Days
Drive: WD-WX82DA1PFFPL - Test Age = 5 Days
Drive: WD-WX82DA1PFZJ8 - Test Age = 5 Days
Drive: WD-WX12DA3KUDNS - Test Age = 5 Days
Drive: WD-WX12DA35LH51 - Test Age = 5 Days
Drive: AA000000000000001504 - Test Age = 4 Days


END

########## ZPool status report for boot-pool ##########
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
	The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:02:41 with 0 errors on Fri Feb 28 03:47:43 2025
config:

	NAME        STATE     READ WRITE CKSUM
	boot-pool   ONLINE       0     0     0
	  sdm3      ONLINE       0     0     0

errors: No known data errors

Drives for this pool are listed below:
0a56b40d-78f4-435a-802b-48ae9276833f -> sdm3 -> S/N:AA000000000000001504 


########## ZPool status report for fr33dan-raid ##########
  pool: fr33dan-raid
 state: ONLINE
  scan: scrub repaired 0B in 12:08:31 with 0 errors on Fri Feb 28 20:13:10 2025
config:

	NAME                                      STATE     READ WRITE CKSUM
	fr33dan-raid                              ONLINE       0     0     0
	  raidz1-0                                ONLINE       0     0     0
	    987c9bf6-56b2-4b30-9c12-55109955bb55  ONLINE       0     0     0
	    9f34b8c6-1841-4840-925e-3920fbe820de  ONLINE       0     0     0
	    ea31863d-b313-4776-940e-be6f730a8116  ONLINE       0     0     0
	    48b88771-4980-4d67-acb2-36c98b09f69a  ONLINE       0     0     0
	    4e3dd146-a7ac-4f6d-a7b6-036004da98b7  ONLINE       0     0     0
	    5bbcbc88-a406-4c6b-b391-a4202cebfc6b  ONLINE       0     0     0
	    8418a2e6-7498-449b-90a8-194573422efe  ONLINE       0     0     0
	    bace6a63-bc18-4657-a8bf-e7fdb05afa37  ONLINE       0     0     0
	  raidz1-1                                ONLINE       0     0     0
	    59eab28c-20b1-4311-948e-4ee246067284  ONLINE       0     0     0
	    45a83ba4-6121-49c9-b79a-0be2f58b3082  ONLINE       0     0     0
	    e3f0750f-2388-47dd-811b-71e185fb1a15  ONLINE       0     0     0
	    513179e8-2bd2-482f-b2cc-b20f20ea3b95  ONLINE       0     0     0

errors: No known data errors

Drives for this pool are listed below:
bace6a63-bc18-4657-a8bf-e7fdb05afa37 -> sda2 -> S/N:WD-WX42DB16NKJU 
5bbcbc88-a406-4c6b-b391-a4202cebfc6b -> sdb2 -> S/N:WD-WX82DA1L5RZE 
48b88771-4980-4d67-acb2-36c98b09f69a -> sdc2 -> S/N:WD-WX42DB1H9CC4 
ea31863d-b313-4776-940e-be6f730a8116 -> sdd2 -> S/N:WD-WX32DB1ATDX9 
59eab28c-20b1-4311-948e-4ee246067284 -> sde1 -> S/N:WD-WX52DA3K3998 
8418a2e6-7498-449b-90a8-194573422efe -> sdf1 -> S/N:WD-WX52D546H4VF 
4e3dd146-a7ac-4f6d-a7b6-036004da98b7 -> sdg2 -> S/N:WD-WX72DA12ZH7V 
513179e8-2bd2-482f-b2cc-b20f20ea3b95 -> sdh1 -> S/N:WD-WX52DA3KS5AV 
9f34b8c6-1841-4840-925e-3920fbe820de -> sdi2 -> S/N:WD-WX82DA1PFFPL 
987c9bf6-56b2-4b30-9c12-55109955bb55 -> sdj2 -> S/N:WD-WX82DA1PFZJ8 
e3f0750f-2388-47dd-811b-71e185fb1a15 -> sdk1 -> S/N:WD-WX12DA3KUDNS 
45a83ba4-6121-49c9-b79a-0be2f58b3082 -> sdl1 -> S/N:WD-WX12DA35LH51 


########## SMART status report for sda drive (WDC WD40EFZX-68AWUN0 : WD-WX42DB16NKJU) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   223   222   021    Pre-fail  Always       -       3825
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       56
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   065   065   000    Old_age   Always       -       25767
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       55
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       30
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       80
194 Temperature_Celsius     0x0022   124   108   000    Old_age   Always       -       26
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 25648 -
# 4 Extended offline Completed without error 00% 25282 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdb drive (WDC WD40EFZX-68AWUN0 : WD-WX82DA1L5RZE) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   222   222   021    Pre-fail  Always       -       3858
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       56
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   065   065   000    Old_age   Always       -       25767
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       55
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       30
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       82
194 Temperature_Celsius     0x0022   123   109   000    Old_age   Always       -       27
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 25648 -
# 4 Extended offline Completed without error 00% 25282 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdc drive (WDC WD40EFZX-68AWUN0 : WD-WX42DB1H9CC4) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   220   220   021    Pre-fail  Always       -       3966
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       56
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   065   065   000    Old_age   Always       -       25764
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       55
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       30
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       81
194 Temperature_Celsius     0x0022   124   106   000    Old_age   Always       -       26
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 25645 -
# 4 Extended offline Completed without error 00% 25279 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdd drive (WDC WD40EFZX-68AWUN0 : WD-WX32DB1ATDX9) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   224   224   021    Pre-fail  Always       -       3783
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       56
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   065   065   000    Old_age   Always       -       25763
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       55
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       30
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       81
194 Temperature_Celsius     0x0022   122   105   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 25644 -
# 4 Extended offline Completed without error 00% 25279 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sde drive (WDC WD40EFPX-68C6CN0 : WD-WX52DA3K3998) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       2
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       8518
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       3
194 Temperature_Celsius     0x0022   122   113   000    Old_age   Always       -       25
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 8398 -
# 4 Extended offline Completed without error 00% 8033 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdf drive (WDC WD40EFPX-68C6CN0 : WD-WX52D546H4VF) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       2
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       77
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   122   111   000    Old_age   Always       -       25
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
0x07 GPL R/O 1 Extended self-test log
Short self-test routine


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdg drive (WDC WD40EFZX-68AWUN0 : WD-WX72DA12ZH7V) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   220   219   021    Pre-fail  Always       -       3991
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       56
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   065   065   000    Old_age   Always       -       25767
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       55
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       30
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       83
194 Temperature_Celsius     0x0022   124   106   000    Old_age   Always       -       26
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 25648 -
# 4 Extended offline Completed without error 00% 25282 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdh drive (WDC WD40EFPX-68C6CN0 : WD-WX52DA3KS5AV) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       2
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       8518
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       3
194 Temperature_Celsius     0x0022   120   111   000    Old_age   Always       -       27
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 8398 -
# 4 Extended offline Completed without error 00% 8032 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdi drive (WDC WD40EFZX-68AWUN0 : WD-WX82DA1PFFPL) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   221   220   021    Pre-fail  Always       -       3941
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       56
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   065   065   000    Old_age   Always       -       25764
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       55
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       30
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       79
194 Temperature_Celsius     0x0022   123   107   000    Old_age   Always       -       27
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 25644 -
# 4 Extended offline Completed without error 00% 25278 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdj drive (WDC WD40EFZX-68AWUN0 : WD-WX82DA1PFZJ8) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   222   222   021    Pre-fail  Always       -       3866
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       56
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   065   065   000    Old_age   Always       -       25764
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       55
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       31
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       81
194 Temperature_Celsius     0x0022   123   109   000    Old_age   Always       -       27
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 25644 -
# 4 Extended offline Completed without error 00% 25278 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdk drive (WDC WD40EFPX-68C6CN0 : WD-WX12DA3KUDNS) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       1
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       8518
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   123   119   000    Old_age   Always       -       27
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 8398 -
# 4 Extended offline Completed without error 00% 8032 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdl drive (WDC WD40EFPX-68C6CN0 : WD-WX12DA35LH51) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       1
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       8518
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   123   119   000    Old_age   Always       -       27
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 8398 -
# 4 Extended offline Completed without error 00% 8032 -


SCT Error Recovery Control:  Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)


########## SMART status report for sdm drive (Dogfish SSD 64GB : AA000000000000001504) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   100   100   050    Old_age   Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   050    Old_age   Always       -       25608
 12 Power_Cycle_Count       0x0032   100   100   050    Old_age   Always       -       64
160 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       0
161 Unknown_Attribute       0x0033   100   100   050    Pre-fail  Always       -       100
163 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       10
164 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       764519
165 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       752
166 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       557
167 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       586
168 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       1500
169 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       61
175 Program_Fail_Count_Chip 0x0032   100   100   050    Old_age   Always       -       0
176 Erase_Fail_Count_Chip   0x0032   100   100   050    Old_age   Always       -       0
177 Wear_Leveling_Count     0x0032   100   100   050    Old_age   Always       -       175441
178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   050    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   050    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       36
194 Temperature_Celsius     0x0022   100   100   050    Old_age   Always       -       40
195 Hardware_ECC_Recovered  0x0032   100   100   050    Old_age   Always       -       2420912
196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       0
232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       100
241 Total_LBAs_Written      0x0030   100   100   050    Old_age   Offline      -       357884
242 Total_LBAs_Read         0x0030   100   100   050    Old_age   Offline      -       123336
245 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       1146778

Warning: ATA error count 0 inconsistent with error log pointer 1

ATA Error Count: 0
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error -4 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 40  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d0 01 00 4f c2 00 08      00:00:00.000  SMART READ DATA
  b0 d1 01 01 4f c2 00 08      00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
  b0 da 00 00 4f c2 00 08      00:00:00.000  SMART RETURN STATUS
  b0 d5 01 00 4f c2 00 08      00:00:00.000  SMART READ LOG
  b0 d5 01 01 4f c2 00 08      00:00:00.000  SMART READ LOG

Most recent Short & Extended Tests - Listed by test number
# 1 Short offline Completed without error 00% 25490 -
# 4 Extended offline Completed without error 00% 25122 -


SCT Error Recovery Control:  SCT Commands not supported


End of data section

I did run a scrub after the resilver and everything came up fine. I’m would love to solve it definitively but I don’t know where to go from here.