Darpa
December 20, 2024, 3:00pm
1
Hey!
Since a few weeks i get the following SMART check error:
Critical
Device: /dev/sda [SAT], FAILED SMART self-check. BACK UP DATA NOW!.
2024-12-20 15:18:55
sda is my boot disk.
I checked all SMART reports related to this disk and the STATUS on all of them is: SUCCESS
If i run smartctl -a /dev/sda i get the following:
admin@truenas[~]$ sudo smartctl -a /dev/sda
[sudo] password for admin:
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.32-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: INTENSO SSD
Serial Number: 1782403007000013
Firmware Version: W1025B0
User Capacity: 250,059,350,016 bytes [250 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available
Device is: Not in smartctl database 7.3/5528
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Dec 20 16:31:44 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
No failed Attributes found.
General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 10) minutes.
SCT capabilities: (0x0001) SCT Status supported.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x0032 100 100 050 Old_age Always - 0
5 Reallocated_Sector_Ct 0x0032 100 100 050 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 050 Old_age Always - 3089
12 Power_Cycle_Count 0x0032 100 100 050 Old_age Always - 27
160 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 0
161 Unknown_Attribute 0x0033 100 100 050 Pre-fail Always - 100
163 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 9
164 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 3536
165 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 20
166 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 1
167 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 14
168 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 5050
169 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 100
175 Program_Fail_Count_Chip 0x0032 100 100 050 Old_age Always - 0
176 Erase_Fail_Count_Chip 0x0032 100 100 050 Old_age Always - 0
177 Wear_Leveling_Count 0x0032 100 100 050 Old_age Always - 0
178 Used_Rsvd_Blk_Cnt_Chip 0x0032 100 100 050 Old_age Always - 0
181 Program_Fail_Cnt_Total 0x0032 100 100 050 Old_age Always - 0
182 Erase_Fail_Count_Total 0x0032 100 100 050 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 050 Old_age Always - 25
194 Temperature_Celsius 0x0022 100 100 050 Old_age Always - 33
195 Hardware_ECC_Recovered 0x0032 100 100 050 Old_age Always - 0
196 Reallocated_Event_Count 0x0032 100 100 050 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 050 Old_age Always - 0
198 Offline_Uncorrectable 0x0032 100 100 050 Old_age Always - 0
232 Available_Reservd_Space 0x0032 100 100 050 Old_age Always - 100
241 Total_LBAs_Written 0x0030 100 100 050 Old_age Offline - 41035
242 Total_LBAs_Read 0x0030 100 100 050 Old_age Offline - 1823
245 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 91575
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 3084 -
# 2 Short offline Completed without error 00% 3060 -
# 3 Short offline Completed without error 00% 3036 -
# 4 Short offline Completed without error 00% 3012 -
# 5 Short offline Completed without error 00% 2988 -
# 6 Short offline Completed without error 00% 2964 -
# 7 Short offline Completed without error 00% 2940 -
# 8 Short offline Completed without error 00% 2916 -
# 9 Short offline Completed without error 00% 2892 -
#10 Short offline Completed without error 00% 2868 -
#11 Short offline Completed without error 00% 2844 -
#12 Short offline Completed without error 00% 2820 -
#13 Short offline Completed without error 00% 2796 -
#14 Short offline Completed without error 00% 2772 -
#15 Short offline Completed without error 00% 2748 -
#16 Short offline Completed without error 00% 2725 -
#17 Short offline Completed without error 00% 2701 -
#18 Short offline Completed without error 00% 2677 -
#19 Short offline Completed without error 00% 2653 -
#20 Short offline Completed without error 00% 2629 -
#21 Short offline Completed without error 00% 2605 -
Selective Self-tests/Logging not supported
The above only provides legacy SMART information - try 'smartctl -x' for more
Am I overlooking something? For me everything looks pretty OK but i still get critical alert.
I am running:
Thanks for your help!
It’s easy to change the Boot drive.
dak180
December 20, 2024, 6:35pm
3
First, if you have not already done so, save a copy of your config file (full).
Next run smartctl -AxHi --log="xerror,error" --log="xselftest,selftest" --log="devstat" --log="ssd" --log="farm" "/dev/sda" and paste the output here.
Darpa
December 20, 2024, 10:05pm
4
First of all I want to thank you for your help. It’s much appreciated.
I have saved my config file and pasted the command you suggested. Here is the outcome:
admin@truenas[~]$ sudo smartctl -AxHi --log="xerror,error" --log="xselftest,selftest" --log="devstat" --log="ssd" --log="farm" "/dev/sda"
[sudo] password for admin:
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.32-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: INTENSO SSD
Serial Number: 1782403007000013
Firmware Version: W1025B0
User Capacity: 250,059,350,016 bytes [250 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available
Device is: Not in smartctl database 7.3/5528
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Dec 20 22:54:55 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Disabled
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Unavailable
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
No failed Attributes found.
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 10) minutes.
SCT capabilities: (0x0001) SCT Status supported.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate -O--CK 100 100 050 - 0
5 Reallocated_Sector_Ct -O--CK 100 100 050 - 0
9 Power_On_Hours -O--CK 100 100 050 - 3084
12 Power_Cycle_Count -O--CK 100 100 050 - 27
160 Unknown_Attribute -O--CK 100 100 050 - 0
161 Unknown_Attribute PO--CK 100 100 050 - 100
163 Unknown_Attribute -O--CK 100 100 050 - 9
164 Unknown_Attribute -O--CK 100 100 050 - 3536
165 Unknown_Attribute -O--CK 100 100 050 - 20
166 Unknown_Attribute -O--CK 100 100 050 - 1
167 Unknown_Attribute -O--CK 100 100 050 - 14
168 Unknown_Attribute -O--CK 100 100 050 - 5050
169 Unknown_Attribute -O--CK 100 100 050 - 100
175 Program_Fail_Count_Chip -O--CK 100 100 050 - 0
176 Erase_Fail_Count_Chip -O--CK 100 100 050 - 0
177 Wear_Leveling_Count -O--CK 100 100 050 - 0
178 Used_Rsvd_Blk_Cnt_Chip -O--CK 100 100 050 - 0
181 Program_Fail_Cnt_Total -O--CK 100 100 050 - 0
182 Erase_Fail_Count_Total -O--CK 100 100 050 - 0
192 Power-Off_Retract_Count -O--CK 100 100 050 - 25
194 Temperature_Celsius -O---K 100 100 050 - 30
195 Hardware_ECC_Recovered -O--CK 100 100 050 - 0
196 Reallocated_Event_Count -O--CK 100 100 050 - 0
197 Current_Pending_Sector -O--CK 100 100 050 - 0
198 Offline_Uncorrectable -O--CK 100 100 050 - 0
232 Available_Reservd_Space -O--CK 100 100 050 - 100
241 Total_LBAs_Written ----CK 100 100 050 - 40693
242 Total_LBAs_Read ----CK 100 100 050 - 1769
245 Unknown_Attribute -O--CK 100 100 050 - 91575
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 1 Comprehensive SMART error log
0x03 GPL R/O 1 Ext. Comprehensive SMART error log
0x04 GPL,SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x24 GPL R/O 88 Current Device Internal Status Data log
0x25 GPL R/O 32 Saved Device Internal Status Data log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 3084 -
# 2 Short offline Completed without error 00% 3060 -
# 3 Short offline Completed without error 00% 3036 -
# 4 Short offline Completed without error 00% 3012 -
# 5 Short offline Completed without error 00% 2988 -
# 6 Short offline Completed without error 00% 2964 -
# 7 Short offline Completed without error 00% 2940 -
# 8 Short offline Completed without error 00% 2916 -
# 9 Short offline Completed without error 00% 2892 -
#10 Short offline Completed without error 00% 2868 -
#11 Short offline Completed without error 00% 2844 -
#12 Short offline Completed without error 00% 2820 -
#13 Short offline Completed without error 00% 2796 -
#14 Short offline Completed without error 00% 2772 -
#15 Short offline Completed without error 00% 2748 -
#16 Short offline Completed without error 00% 2725 -
#17 Short offline Completed without error 00% 2701 -
#18 Short offline Completed without error 00% 2677 -
#19 Short offline Completed without error 00% 2653 -
Selective Self-tests/Logging not supported
SCT Status Version: 3
SCT Version (vendor specific): 0 (0x0000)
Device State: Active (0)
Current Temperature: 30 Celsius
Power Cycle Min/Max Temperature: 30/30 Celsius
Lifetime Min/Max Temperature: 24/43 Celsius
Specified Max Operating Temperature: 100 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Data Table command not supported
SCT Error Recovery Control command not supported
Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 27 --- Lifetime Power-On Resets
0x01 0x010 4 3084 --- Power-on Hours
0x01 0x018 6 2666897926 --- Logical Sectors Written
0x01 0x020 6 131676124 --- Number of Write Commands
0x01 0x028 6 115939394 --- Logical Sectors Read
0x01 0x030 6 1002548 --- Number of Read Commands
0x07 ===== = = === == Solid State Device Statistics (rev 1) ==
0x07 0x008 1 0 --- Percentage Used Endurance Indicator
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 4 0 Command failed due to ICRC error
0x0002 4 0 R_ERR response for data FIS
0x0005 4 1 R_ERR response for non-data FIS
0x000a 4 2 Device-to-host register FISes sent due to a COMRESET
FARM log (GP Log 0xa6) not supported for non-Seagate drives
Still looks strange since all RAW data looks pretty OK. But with these cheap drives who knows what all this Unknown_Attributes are …
Meanwhile I replaced the SATA cable with a brand new one, same outcome whatsoever…
Best,
Darpa:
sda is my boot disk.
Did you possibly reboot after the alert was triggered but before you started investigating the smart status? If so, the device id may have changed and sda might actually refer to one of your other drives.
Device IDs can change across reboots, this is normal.
dak180
December 21, 2024, 1:45pm
6
In addition to double checking this please run a long smart test (I do not see one listed) and then rerun the smart output. Also, did anything of note (power outage, hardware change, etc.) happen just before the error started being reported?
Darpa
December 23, 2024, 2:15pm
7
I also checked the id of my disks multiple times, after multiple reboots and also trough the shell. The affected disk is without doubt the boot disk.
I have never seen this message generated before however it appears the drive power on self tests have failed. It could be bad ram, bad controller, something that failed. My advice, replace the drive and RMA the drive.
I appreciate you posting this data as i said, i have never seen one before.
Sorry for your loss.
Darpa
December 23, 2024, 3:14pm
9
dak180:
In addition to double checking this please run a long smart test (I do not see one listed) and then rerun the smart output. Also, did anything of note (power outage, hardware change, etc.) happen just before the error started being reported?
I made multiple LONG SMART test on the sda (boot) disk. I do not really understand why they are not stated under smartctl . But they were listed in the Tasks list and the Status was always: Success.
A few weeks before all of this started i had a power outage (I don’t own a UPS, but will buy one within the next 6 months). I can’t really pin it down as the reason, but it definitely did not help. Within a period of 4 weeks I had first a non-responding server and had to reboot, which kinda fixed it. 2 weeks later (now) I had the SMART alert.
Before I went out and bought a new disk I tried to reinstall Truenas Scale on the affected disk. After selecting of the disk as the boot disk I was presented with the following error:
Command sgdisk -a4096 -n1:0:+1024K -t1:EF02 -A1:set:2
/deu/sda failed:
Could not create partition 1 from 34 to 2081
Something is seriously wrong with this disk, even tho all SMART data shows otherwise. So, after all that struggle I bought a new disk… I went for a Samsung 870 evo, so I went from the lowest shelf to the pretty much highest shelf. Hope the price is justified and it will last for at least a few years. The Intenso one (the one which failed) lasted from middle of June till now. A lifespan of 6 months till total failure. Quite a disaster.
I was thinking a cheap SSD will be good enough for the sistem. I rather spend my money on the storage disks. But since all of this happened, I don’t know anymore…
dak180
December 23, 2024, 6:39pm
10
So long as they are available, I would likely be using a 32gb optane drive for boot.
