I had my first failed drive today. I have an HDD in a drawer I keep for these scenarios so I wasn’t concerned.
First, as soon as the pool was degraded, I shutdown the server, since I wasn’t going to address the issue immediately
Plugged an HDD that holds the backups for this pool (data backed up with replication, it’s a completely different pool)
Initiated a replication task from degraded pool to backup pool. This completed in less than a minute. My rationale is: the backup holds almost all the data, so the replication is incredibly short and makes sure all the data is safe, which drastically reduces the chances of killing the remaining HDD in the pool
Pool is now in healthy state, the faulted HDD is ready for analysis
Now, the last smart test for this HDD is healthy. I need to prove it’s broken so I can get warranty for it.
My server is a SuperMicro CSE-846 X9DRi-F 4U Server, so enterprise quality equipment, I don’t think cabling are the issue, but of course it could be (and the drives are connected to the backplane!)
Suggestions on how to proceed?
My idea is to go with the following:
Stux
September 17, 2024, 4:36am
2
Was the last smart test a long one?
Post results of smartctl -a for failed device
No it was a short one. I’m running a full burn-in test now. So short + badblocks + long + I don’t remember. Will let you know once is done.
root@zeus[~]# smartctl -a /dev/da7
smartctl 7.4 2023-08-01 r5530 [FreeBSD 13.3-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate IronWolf Pro
Device Model: ST4000NE001-2MA101
Serial Number: WS24GXN1
LU WWN Device Id: 5 000c50 0e5ea575f
Firmware Version: EN01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5528
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Sep 16 22:27:09 2024 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 559) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 362) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x50bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 083 064 044 Pre-fail Always - 195401984
3 Spin_Up_Time 0x0003 091 091 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 18
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 088 060 045 Pre-fail Always - 623136913
9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15668
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 18
18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 064 055 040 Old_age Always - 36 (Min/Max 29/37)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 16
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 801
194 Temperature_Celsius 0x0022 036 045 000 Old_age Always - 36 (0 25 0 0 0)
195 Hardware_ECC_Recovered 0x001a 083 064 000 Old_age Always - 195401984
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 15548h+04m+44.107s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 65325398572
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 437961274569
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 15666 -
# 2 Short offline Completed without error 00% 15653 -
# 3 Short offline Completed without error 00% 15652 -
# 4 Short offline Completed without error 00% 15580 -
# 5 Short offline Completed without error 00% 15556 -
# 6 Short offline Completed without error 00% 15532 -
# 7 Short offline Completed without error 00% 15508 -
# 8 Short offline Completed without error 00% 15484 -
# 9 Extended offline Completed without error 00% 15442 -
#10 Short offline Completed without error 00% 15412 -
#11 Short offline Completed without error 00% 15388 -
#12 Short offline Completed without error 00% 15364 -
#13 Short offline Completed without error 00% 15340 -
#14 Short offline Completed without error 00% 15316 -
#15 Short offline Completed without error 00% 15292 -
#16 Short offline Completed without error 00% 15268 -
#17 Short offline Completed without error 00% 15245 -
#18 Short offline Completed without error 00% 15172 -
#19 Short offline Completed without error 00% 15148 -
#20 Short offline Completed without error 00% 15124 -
#21 Short offline Completed without error 00% 15100 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
The above only provides legacy SMART information - try 'smartctl -x' for more
EDIT: I just notied I have an hdd which has been going for 70K hours
dan
September 17, 2024, 11:05am
4
What makes you think that drive has “failed”?
Got a critical alert from truenas that said the drive is faulted due to repeated unrecoverable error and the pool was degraded.
I fixed the pool by replacing the faulted drive
I run the burnin test and it turned back with no errors.
What could cause an HDD to be in faulted state while the hardware is healthy?
Sata wire not fully seated, HBA overheating, low quality sata data wire, HBA not flashed to IT, poor choice of hardware (port multiplier, usb adapter, etc.), memory instability…
I mean I can think of many ways something could go wrong, but it is hard to say which of the 1,000 possible quirks happened to you & if any of them are actually preventable or if you just had bad luck.
Since this is a seagate drive mind giving us translated values? Something like
smartctl -a -v 1,raw48:54 /dev/*DRIVENAMEHERE* -v 7,raw48:54 -v 195,raw48:54
should translate the following into something readable
7 Seek_Error_Rate 0x000f 088 060 045 Pre-fail Always - 623136913
195 Hardware_ECC_Recovered 0x001a 083 064 000 Old_age Always - 195401984
Sorry for the long wait, I’m re-running the whole burnin testing. This is the result of the previous command:
# smartctl -a -v 1,raw48:54 /dev/da7 -v 7,raw48:54 -v 195,raw48:54
smartctl 7.4 2023-08-01 r5530 [FreeBSD 13.3-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate IronWolf Pro
Device Model: ST4000NE001-2MA101
Serial Number: WS24GXN1
LU WWN Device Id: 5 000c50 0e5ea575f
Firmware Version: EN01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5528
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Sep 22 10:46:47 2024 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 249) Self-test routine in progress...
90% of test remaining.
Total time to complete Offline
data collection: ( 559) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 362) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x50bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 083 064 044 Pre-fail Always - 0
3 Spin_Up_Time 0x0003 091 091 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 18
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 088 060 045 Pre-fail Always - 0
9 Power_On_Hours 0x0032 082 082 000 Old_age Always - 15801
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 18
18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 064 055 040 Old_age Always - 36 (Min/Max 29/37)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 16
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 905
194 Temperature_Celsius 0x0022 036 045 000 Old_age Always - 36 (0 25 0 0 0)
195 Hardware_ECC_Recovered 0x001a 083 064 000 Old_age Always - 0
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 15631h+14m+59.999s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 124429581228
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 499360871339
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Self-test routine in progress 90% 15771 -
# 2 Short offline Completed without error 00% 15759 -
# 3 Short offline Completed without error 00% 15746 -
# 4 Short offline Completed without error 00% 15722 -
# 5 Short offline Completed without error 00% 15710 -
# 6 Short offline Completed without error 00% 15675 -
# 7 Short offline Completed without error 00% 15666 -
# 8 Short offline Completed without error 00% 15653 -
# 9 Short offline Completed without error 00% 15652 -
#10 Short offline Completed without error 00% 15580 -
#11 Short offline Completed without error 00% 15556 -
#12 Short offline Completed without error 00% 15532 -
#13 Short offline Completed without error 00% 15508 -
#14 Short offline Completed without error 00% 15484 -
#15 Extended offline Completed without error 00% 15442 -
#16 Short offline Completed without error 00% 15412 -
#17 Short offline Completed without error 00% 15388 -
#18 Short offline Completed without error 00% 15364 -
#19 Short offline Completed without error 00% 15340 -
#20 Short offline Completed without error 00% 15316 -
#21 Short offline Completed without error 00% 15292 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
The above only provides legacy SMART information - try 'smartctl -x' for more
EDIT: Reading the output, there is NOTHING that suggests this drive is dying.
Well we got 10% of the long test remaining, so always a chance something goes wrong… but I’m going to have to agree with you that the drive looks fine.
If it was me I’d toss it back onto a pool (assuming it is raid z2 or higher) and send it until it errors out again.
I only have mirror vdrives (1 disk + a mirror of that disk), does your suggestion still stands?
Ehh - anything single disk until failure I wouldn’t be so confident. I see no major faults with a third mirror other than now having 3 disks worth of equipment for 1 disk worth of space; but at least now that HDD is put back into production.
Stux
September 22, 2024, 9:51pm
12
Pity.
Burn in testing will reset the pending blocks. Thus erasing one of the signs that the drive is dying.
It should have been sufficient to just run a smart long test, which is non destructive.
In the post above you still have 90% of the long test remaining, so wait for that to finish.
If there’s no obvious faults, then put back in a wait for an issue to occur again.
I document this sort of thing in the disk description field.
I wasn’t aware of that, thank you. Will do next time. Yeah I’ll wait that test, but I already executed a whole burnin, that long test remaining is one that triggered “accidentally” with the auto smart test in truenas
No issues. Followed your suggestion, let’s see what happens.
