4 disk died in 5 days and took entire pool with them

HemaN · August 27, 2024, 8:08pm

I know that is too late for rescue any files, but want learn something from this lesson and know what I did wrong.

Below are NAS specs:

Xeon X3430
Supermicro X8SIE-F
4x8GB DDR3 UDIMM ECC memory
Avago 9211i HBA card in IT mode
SilverStone ET550 550W 80 Plus Bronze
4 x ST4000VN006 connected through HBA card bought in February, so it only had about 4000 hours of mileage
2 x ST6000VN001 connected directly to motherboard with sata cables
Truenas 13.1 U6.1

I built NAS server for my colleagues. I’ve created two pools on that server, one through Avago 9211i HBA in IT mode with 4 disks connected with mini SAS SFF-8087 cable. And there is second pool with two 6 TB drives in mirror connected directly to motherboard with SATA cables. I wanted to add 2 more drives to that second pool and change it to RAID-Z2. So I added 2 new drives, added my own 4 TB drive as a dummy, and detached one drive from mirror (earlier performed backup of data to an USB drive). Then from those drives created RAID-Z2, copied all data from degradaded mirror, then detach 4 TB dummy and attach last disk from old mirror. After powering on server after I added two new drives and my 4 TB dummy I’ve noticed that first pool connected to HBA is degradaded, status of one disk was unknown. „Another dead new Seagate” I thought and ignored the danger and focused on second pool I was working on. Second thing that lulled my vigilance was fact, that HBA pool drives are in different compartment of server case and I didn’t physically touch any part of that pool. But to address that issue I started firstly short SMART test (didn’t find any problems) and then long SMART test on that faulty drive while I was copying data on second pool. The result was:

Device: /dev/da0 [SAT], Read SMART Error Log Failed,
Device: /dev/da0 [SAT], Read SMART Self-Test Log Failed,
Device: /dev/da0 [SAT], failed to read SMART Attribute Data,
Device: /dev/da0 [SAT], not capable of SMART self-check.

After I powered server on to remove my 4 TB dummy things get worst. First disk changed it status to disconnected, and second disk chaneged status to unknown. And that was my last chance to do a backup from that pool. Instead of that i powered off server and started checking cables, especially nimi sas cable, then tried to connect it to second output in HBA card, but it didn’t help. While I rebootet server few times I’ve noticed that after pressing power buton strange sound was coming from HBA disks, like there was something wrong with disks heads. And after I chacked every cable and powered server again HBA pool was offline. My last attempt was to connect all drives directly to motherboard, but it didn’t change anything. So I disconnected all HBA drives and sent it today for warranty data recovery and hopefully they recover the data. Second pool was working well all the time and it’s working now normally.

Did I bought drives from faulty production batch that they died after half a year of non demanding work? But 4 drives in just few days? Do HBA card can cause drive death? I wouldn’t be supprised data corruption, but not so quick drives dying. Strange sounds when powering on suggest rather power supply, but like I said, second pool and whole serwer is working fine, and PSU have single 12V rail, so it can’t be a PSU.

I have some old drives with some small SMART errors and I will connect them to HBA and create RAID-Z pool just to check wether they will die too. I will receive new drive from warranty, but I’m now affraiding to connect them to HBA.

It looked like powering server on damaging the drives, but only those connected to HBA.

Could someone tell me what just happend here?

Okedokey · August 28, 2024, 2:37pm

Just a thought, could temperature be a factor?

HemaN · August 28, 2024, 7:06pm

No, for sure temperature wasn’t a problem. Server is in basement where temperature is about 19 - 20 *C throughout the year. ST4000VN006 have a thin case and there is a lot of space between disk mounted in cage.

Stux · August 28, 2024, 9:16pm

The smart test tells you the temperature of the drives, and sometimes the maximum reached too.

Even in air conditioned offices, drives can reach 50+C under load with insufficient cooling.

HemaN · August 31, 2024, 3:03pm

OK, maybe there is something to it. But since I sent all 4 drives for recovery I’m not able to view it’s temp log. Is there a way to show historic temp logs of disk that are disconnected?

I checked other 6 disk (2 SSDs with Truenas operating system and 4 of second pool I was upgrading) that are now working in server - highest temp which is recoreded is 39*C.

joeschmuck · August 31, 2024, 5:00pm

Post the entire output of smartctl -x /dev/da0 and for each of the other drives. I find it difficult to believe so many drives failed near the same time. My bet is system stabiliy issue. Post that drive data. Let us tell you what we see.

Also, after that start a RAM test, like Memtest86, run it for a day or two. After that, a CPU stress test as well.

Some HBA cards demand high airflow. Just worth noting.

HemaN · September 2, 2024, 10:10am

Ram was checked with memtest in February when I was upgrading from 16 to 32 GB of RAM and it took 4 full loops without any errors. I know, that HBA radiators are symbolic, which is why I set an 120 mm fan blowing on HBA. When I’m touching HBA radiator it’s warm, but not burning. I estimating it has around 40*C. I made smartctl printouts, but I have to find how to put it on this post. Maybe I can sent you a TXT file?

essinghigh · September 2, 2024, 1:03pm

I have absolutely no idea why we aren’t allowed to upload txt files here, considering we can upload sh files just fine. Maybe some misconfiguration.

If it’s too long to put inside triple backticks (```), then attaching screenshots should be fine too (or just rename it to .sh I suppose ).

HemaN · September 2, 2024, 2:44pm

I would like to attach .sh file, but “new users can not upload attachements”. Ok, this will be pain to see, but I’m putting first 3 disk printout in this post, next 3 disks in next post:

ada0-ada2

root@truenas[~]# smartctl -x /dev/ada0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke,

=== START OF INFORMATION SECTION ===
Device Model: SSDPR-CL100-120-G3
Serial Number: R0E011940
Firmware Version: U1013A0
User Capacity: 120,034,123,776 bytes [120 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Sep 2 11:21:24 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Disabled
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Unavailable

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate -O–CK 100 100 050 - 0
5 Reallocated_Sector_Ct -O–CK 100 100 050 - 0
9 Power_On_Hours -O–CK 100 100 050 - 8117
12 Power_Cycle_Count -O–CK 100 100 050 - 51
160 Unknown_Attribute -O–CK 100 100 050 - 0
161 Unknown_Attribute PO–CK 100 100 050 - 100
163 Unknown_Attribute -O–CK 100 100 050 - 11
164 Unknown_Attribute -O–CK 100 100 050 - 8907
165 Unknown_Attribute -O–CK 100 100 050 - 30
166 Unknown_Attribute -O–CK 100 100 050 - 1
167 Unknown_Attribute -O–CK 100 100 050 - 9
168 Unknown_Attribute -O–CK 100 100 050 - 5050
169 Unknown_Attribute -O–CK 100 100 050 - 100
175 Program_Fail_Count_Chip -O–CK 100 100 050 - 0
176 Erase_Fail_Count_Chip -O–CK 100 100 050 - 0
177 Wear_Leveling_Count -O–CK 100 100 050 - 0
178 Used_Rsvd_Blk_Cnt_Chip -O–CK 100 100 050 - 0
181 Program_Fail_Cnt_Total -O–CK 100 100 050 - 0
182 Erase_Fail_Count_Total -O–CK 100 100 050 - 0
192 Power-Off_Retract_Count -O–CK 100 100 050 - 29
194 Temperature_Celsius -O—K 100 100 050 - 40
195 Hardware_ECC_Recovered -O–CK 100 100 050 - 0
196 Reallocated_Event_Count -O–CK 100 100 050 - 0
197 Current_Pending_Sector -O–CK 100 100 050 - 0
198 Offline_Uncorrectable -O–CK 100 100 050 - 0
199 UDMA_CRC_Error_Count -O–CK 100 100 050 - 0
232 Available_Reservd_Space -O–CK 100 100 050 - 100
241 Total_LBAs_Written ----CK 100 100 050 - 7006
242 Total_LBAs_Read ----CK 100 100 050 - 8270
245 Unknown_Attribute -O–CK 100 100 050 - 4589
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning

General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 1 Comprehensive SMART error log
0x03 GPL R/O 1 Ext. Comprehensive SMART error log
0x04 GPL,SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xde GPL VS 8 Device vendor specific log

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 8106 -

2 Extended offline Completed without error 00% 8082 -

3 Short offline Completed without error 00% 8058 -

4 Short offline Completed without error 00% 8034 -

5 Short offline Completed without error 00% 8010 -

6 Short offline Completed without error 00% 7986 -

7 Short offline Completed without error 00% 7962 -

8 Short offline Completed without error 00% 7939 -

9 Extended offline Completed without error 00% 7916 -

#10 Short offline Completed without error 00% 7892 -
#11 Short offline Completed without error 00% 7868 -
#12 Short offline Completed without error 00% 7844 -
#13 Short offline Completed without error 00% 7820 -
#14 Short offline Completed without error 00% 7796 -
#15 Short offline Completed without error 00% 7772 -
#16 Extended offline Completed without error 00% 7748 -
#17 Short offline Completed without error 00% 7724 -
#18 Short offline Completed without error 00% 7700 -
#19 Short offline Completed without error 00% 7676 -

Selective Self-tests/Logging not supported

SCT Commands not supported

Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 51 — Lifetime Power-On Resets
0x01 0x010 4 8117 — Power-on Hours
0x01 0x018 6 459187466 — Logical Sectors Written
0x01 0x020 6 725111 — Number of Write Commands
0x01 0x028 6 542039834 — Logical Sectors Read
0x01 0x030 6 108241 — Number of Read Commands
0x07 ===== = = === == Solid State Device Statistics (rev 1) ==
0x07 0x008 1 0 — Percentage Used Endurance Indicator
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

================================================================================================================

root@truenas[~]# smartctl -x /dev/ada1
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke,

=== START OF INFORMATION SECTION ===
Device Model: SSDPR-CL100-120-G3
Serial Number: G2B008903
Firmware Version: SBFM61.5
User Capacity: 120,034,123,776 bytes [120 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Sep 2 11:32:16 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Unavailable

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (65535) seconds.
Offline data collection
capabilities: (0x79) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 30) minutes.
Conveyance self-test routine
recommended polling time: ( 6) minutes.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate PO-R-- 100 100 050 - 0
9 Power_On_Hours -O–C- 100 100 000 - 8051
12 Power_Cycle_Count -O–C- 100 100 000 - 49
168 Unknown_Attribute -O–C- 100 100 000 - 0
170 Unknown_Attribute PO---- 100 100 000 - 21
173 Unknown_Attribute -O–C- 100 100 000 - 393231
192 Power-Off_Retract_Count -O–C- 100 100 000 - 24
194 Temperature_Celsius PO—K 067 067 000 - 33 (Min/Max 33/33)
218 Unknown_Attribute PO-R-- 100 100 050 - 0
231 Unknown_SSD_Attribute PO–C- 100 100 000 - 99
241 Total_LBAs_Written -O–C- 100 100 000 - 211
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning

General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 51 Comprehensive SMART error log
0x03 GPL R/O 64 Ext. Comprehensive SMART error log
0x04 GPL,SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log

SMART Extended Comprehensive Error Log Version: 1 (64 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 8039 -

2 Extended offline Completed without error 00% 8016 -

3 Short offline Completed without error 00% 7991 -

4 Short offline Completed without error 00% 7967 -

5 Short offline Completed without error 00% 7943 -

6 Short offline Completed without error 00% 7919 -

7 Short offline Completed without error 00% 7895 -

8 Extended offline Completed without error 00% 7878 -

9 Short offline Completed without error 00% 7854 -

#10 Short offline Completed without error 00% 7830 -
#11 Short offline Completed without error 00% 7806 -
#12 Short offline Completed without error 00% 7782 -
#13 Short offline Completed without error 00% 7758 -
#14 Short offline Completed without error 00% 7734 -
#15 Extended offline Completed without error 00% 7710 -
#16 Short offline Completed without error 00% 7686 -
#17 Short offline Completed without error 00% 7662 -
#18 Short offline Completed without error 00% 7638 -
#19 Short offline Completed without error 00% 7614 -

SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Commands not supported

Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 49 — Lifetime Power-On Resets
0x01 0x010 4 8051 — Power-on Hours
0x01 0x018 6 442683857 — Logical Sectors Written
0x01 0x028 6 502774404 — Logical Sectors Read
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 — Number of Reported Uncorrectable Errors
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 33 — Current Temperature
0x05 0x020 1 33 — Highest Temperature
0x05 0x028 1 33 — Lowest Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 352 — Number of Hardware Resets
0x06 0x018 4 0 — Number of Interface CRC Errors
0x07 ===== = = === == Solid State Device Statistics (rev 1) ==
0x07 0x008 1 0 — Percentage Used Endurance Indicator
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries

=================================================================================================================================

root@truenas[~]# smartctl -x /dev/ada2
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke,

=== START OF INFORMATION SECTION ===
Device Model: OOS6000G
Serial Number: 000DGEVA
LU WWN Device Id: 5 000c50 074472857
Firmware Version: OOS1
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Sep 2 11:49:25 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Disabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Write SCT (Get) Feature Control Command failed: Input/output error
Wt Cache Reorder: Unknown (SCT Feature Control command failed)

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 567) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 529) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 077 064 044 - 49051112
3 Spin_Up_Time PO---- 093 092 000 - 0
4 Start_Stop_Count -O–CK 100 100 020 - 21
5 Reallocated_Sector_Ct PO–CK 100 100 010 - 0
7 Seek_Error_Rate POSR-- 068 060 045 - 5858687
9 Power_On_Hours -O–CK 100 100 000 - 192
10 Spin_Retry_Count PO–C- 100 100 097 - 0
12 Power_Cycle_Count -O–CK 100 100 020 - 20
18 Unknown_Attribute PO-R-- 100 100 050 - 0
187 Reported_Uncorrect -O–CK 100 100 000 - 0
188 Command_Timeout -O–CK 100 100 000 - 0
190 Airflow_Temperature_Cel -O—K 065 058 000 - 35 (Min/Max 33/42)
192 Power-Off_Retract_Count -O–CK 100 100 000 - 8
193 Load_Cycle_Count -O–CK 100 100 000 - 1344
194 Temperature_Celsius -O—K 035 042 000 - 35 (0 27 0 0 0)
197 Current_Pending_Sector -O–C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
200 Multi_Zone_Error_Rate PO—K 100 100 001 - 0
240 Head_Flying_Hours ------ 100 100 000 - 87 (120 120 0)
241 Total_LBAs_Written ------ 100 253 000 - 3394258148
242 Total_LBAs_Read ------ 100 253 000 - 3252267459
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning

General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 5 Ext. Comprehensive SMART error log
0x04 GPL R/O 256 Device Statistics log
0x04 SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x0a GPL R/W 8 Device Statistics Notification
0x0c GPL R/O 2048 Pending Defects log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x13 GPL R/O 1 SATA NCQ Send and Receive log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x24 GPL R/O 768 Current Device Internal Status Data log
0x2f GPL - 1 Set Sector Configuration
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa1 GPL,SL VS 160 Device vendor specific log
0xa2 GPL VS 16320 Device vendor specific log
0xa4 GPL,SL VS 160 Device vendor specific log
0xa6 GPL VS 192 Device vendor specific log
0xa8-0xa9 GPL,SL VS 136 Device vendor specific log
0xab GPL VS 1 Device vendor specific log
0xad GPL VS 16 Device vendor specific log
0xb1 GPL,SL VS 160 Device vendor specific log
0xb6 GPL VS 1920 Device vendor specific log
0xbe-0xbf GPL VS 65535 Device vendor specific log
0xc1 GPL,SL VS 8 Device vendor specific log
0xc3 GPL,SL VS 24 Device vendor specific log
0xc6 GPL VS 5184 Device vendor specific log
0xc7 GPL,SL VS 8 Device vendor specific log
0xc9 GPL,SL VS 8 Device vendor specific log
0xca GPL,SL VS 16 Device vendor specific log
0xcd GPL,SL VS 1 Device vendor specific log
0xce GPL VS 1 Device vendor specific log
0xcf GPL VS 512 Device vendor specific log
0xd1 GPL VS 656 Device vendor specific log
0xd2 GPL VS 10256 Device vendor specific log
0xd4 GPL VS 2048 Device vendor specific log
0xda GPL,SL VS 1 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 181 -

2 Extended offline Completed without error 00% 165 -

3 Short offline Completed without error 00% 133 -

4 Short offline Completed without error 00% 109 -

5 Short offline Completed without error 00% 85 -

6 Short offline Completed without error 00% 61 -

7 Short offline Completed without error 00% 37 -

8 Short offline Completed without error 00% 13 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version: 3
SCT Version (vendor specific): 522 (0x020a)
Device State: Active (0)
Current Temperature: 34 Celsius
Power Cycle Min/Max Temperature: 34/42 Celsius
Lifetime Min/Max Temperature: 27/42 Celsius
Under/Over Temperature Limit Count: 0/8
SMART Status: 0xc24f (PASSED)
Vendor specific:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version: 2
Temperature Sampling Period: 4 minutes
Temperature Logging Interval: 59 minutes
Min/Max recommended Temperature: 10/40 Celsius
Min/Max Temperature Limit: 5/60 Celsius
Temperature History Size (Index): 128 (103)

Index Estimated Time Temperature Celsius
104 2024-08-28 06:23 34 ***************
105 2024-08-28 07:22 34 ***************
106 2024-08-28 08:21 35 ****************
107 2024-08-28 09:20 34 ***************
108 2024-08-28 10:19 35 ****************
… …( 3 skipped). … ****************
112 2024-08-28 14:15 35 ****************
113 2024-08-28 15:14 ? -
114 2024-08-28 16:13 34 ***************
115 2024-08-28 17:12 ? -
116 2024-08-28 18:11 35 ****************
117 2024-08-28 19:10 35 ****************
118 2024-08-28 20:09 34 ***************
119 2024-08-28 21:08 35 ****************
120 2024-08-28 22:07 38 *******************
121 2024-08-28 23:06 35 ****************
122 2024-08-29 00:05 35 ****************
123 2024-08-29 01:04 34 ***************
124 2024-08-29 02:03 34 ***************
125 2024-08-29 03:02 35 ****************
126 2024-08-29 04:01 35 ****************
127 2024-08-29 05:00 35 ****************
0 2024-08-29 05:59 34 ***************
1 2024-08-29 06:58 35 ****************
2 2024-08-29 07:57 34 ***************
3 2024-08-29 08:56 35 ****************
4 2024-08-29 09:55 35 ****************
5 2024-08-29 10:54 35 ****************
6 2024-08-29 11:53 36 *****************
7 2024-08-29 12:52 37 ******************
8 2024-08-29 13:51 37 ******************
9 2024-08-29 14:50 36 *****************
10 2024-08-29 15:49 35 ****************
11 2024-08-29 16:48 34 ***************
12 2024-08-29 17:47 34 ***************
13 2024-08-29 18:46 35 ****************
14 2024-08-29 19:45 34 ***************
15 2024-08-29 20:44 35 ****************
… …( 5 skipped). … ****************
21 2024-08-30 02:38 35 ****************
22 2024-08-30 03:37 34 ***************
… …( 3 skipped). … ***************
26 2024-08-30 07:33 34 ***************
27 2024-08-30 08:32 35 ****************
28 2024-08-30 09:31 34 ***************
29 2024-08-30 10:30 35 ****************
… …( 2 skipped). … ****************
32 2024-08-30 13:27 35 ****************
33 2024-08-30 14:26 34 ***************
34 2024-08-30 15:25 35 ****************
35 2024-08-30 16:24 34 ***************
36 2024-08-30 17:23 34 ***************
37 2024-08-30 18:22 35 ****************
… …( 3 skipped). … ****************
41 2024-08-30 22:18 35 ****************
42 2024-08-30 23:17 34 ***************
43 2024-08-31 00:16 34 ***************
44 2024-08-31 01:15 34 ***************
45 2024-08-31 02:14 35 ****************
46 2024-08-31 03:13 34 ***************
… …( 15 skipped). … ***************
62 2024-08-31 18:57 34 ***************
63 2024-08-31 19:56 35 ****************
64 2024-08-31 20:55 35 ****************
65 2024-08-31 21:54 34 ***************
66 2024-08-31 22:53 34 ***************
67 2024-08-31 23:52 34 ***************
68 2024-09-01 00:51 41 **********************
69 2024-09-01 01:50 42 ***********************
… …( 5 skipped). … ***********************
75 2024-09-01 07:44 42 ***********************
76 2024-09-01 08:43 36 *****************
77 2024-09-01 09:42 35 ****************
78 2024-09-01 10:41 35 ****************
79 2024-09-01 11:40 36 *****************
80 2024-09-01 12:39 34 ***************
81 2024-09-01 13:38 34 ***************
82 2024-09-01 14:37 34 ***************
83 2024-09-01 15:36 37 ******************
84 2024-09-01 16:35 35 ****************
85 2024-09-01 17:34 35 ****************
86 2024-09-01 18:33 34 ***************
87 2024-09-01 19:32 34 ***************
88 2024-09-01 20:31 35 ****************
89 2024-09-01 21:30 34 ***************
… …( 10 skipped). … ***************
100 2024-09-02 08:19 34 ***************
101 2024-09-02 09:18 35 ****************
102 2024-09-02 10:17 35 ****************
103 2024-09-02 11:16 35 ****************

SCT Error Recovery Control:
Read: 100 (10.0 seconds)
Write: 100 (10.0 seconds)

Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 20 — Lifetime Power-On Resets
0x01 0x010 4 192 — Power-on Hours
0x01 0x018 6 3394258148 — Logical Sectors Written
0x01 0x020 6 3321080 — Number of Write Commands
0x01 0x028 6 3252267450 — Logical Sectors Read
0x01 0x030 6 19378817 — Number of Read Commands
0x01 0x038 6 - — Date and Time TimeStamp
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 192 — Spindle Motor Power-on Hours
0x03 0x010 4 87 — Head Flying Hours
0x03 0x018 4 1345 — Head Load Events
0x03 0x020 4 0 — Number of Reallocated Logical Sectors
0x03 0x028 4 0 — Read Recovery Attempts
0x03 0x030 4 0 — Number of Mechanical Start Failures
0x03 0x038 4 0 — Number of Realloc. Candidate Logical Sectors
0x03 0x040 4 8 — Number of High Priority Unload Events
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 — Number of Reported Uncorrectable Errors
0x04 0x010 4 0 — Resets Between Cmd Acceptance and Completion
0x04 0x018 4 0 -D- Physical Element Status Changed
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 35 — Current Temperature
0x05 0x010 1 35 — Average Short Term Temperature
0x05 0x018 1 - — Average Long Term Temperature
0x05 0x020 1 42 — Highest Temperature
0x05 0x028 1 33 — Lowest Temperature
0x05 0x030 1 36 — Highest Average Short Term Temperature
0x05 0x038 1 34 — Lowest Average Short Term Temperature
0x05 0x040 1 - — Highest Average Long Term Temperature
0x05 0x048 1 - — Lowest Average Long Term Temperature
0x05 0x050 4 0 — Time in Over-Temperature
0x05 0x058 1 60 — Specified Maximum Operating Temperature
0x05 0x060 4 0 — Time in Under-Temperature
0x05 0x068 1 5 — Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 54 — Number of Hardware Resets
0x06 0x010 4 1 — Number of ASR Events
0x06 0x018 4 0 — Number of Interface CRC Errors
0xff ===== = = === == Vendor Specific Statistics (rev 1) ==
0xff 0x008 7 0 — Vendor Specific
0xff 0x010 7 0 — Vendor Specific
0xff 0x018 7 0 — Vendor Specific
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description

HemaN · September 2, 2024, 2:46pm

ada3-ada5

root@truenas[~]# smartctl -x /dev/ada3
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke,

=== START OF INFORMATION SECTION ===
Model Family: Seagate IronWolf
Device Model: ST6000VN001-2BB186
Serial Number: ZR12SR8E
LU WWN Device Id: 5 000c50 0e3811fbd
Firmware Version: SC60
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5425 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Sep 2 11:44:50 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 717) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 081 064 006 - 122903951
3 Spin_Up_Time PO---- 092 091 000 - 0
4 Start_Stop_Count -O–CK 100 100 020 - 84
5 Reallocated_Sector_Ct PO–CK 100 100 010 - 0
7 Seek_Error_Rate POSR-- 087 060 045 - 510862821
9 Power_On_Hours -O–CK 074 074 000 - 23452
10 Spin_Retry_Count PO–C- 100 100 097 - 0
12 Power_Cycle_Count -O–CK 100 100 020 - 84
183 Runtime_Bad_Block -O–CK 100 100 000 - 0
184 End-to-End_Error -O–CK 100 100 099 - 0
187 Reported_Uncorrect -O–CK 100 100 000 - 0
188 Command_Timeout -O–CK 100 100 000 - 0
189 High_Fly_Writes -O-RCK 100 100 000 - 0
190 Airflow_Temperature_Cel -O—K 065 056 040 - 35 (Min/Max 34/39)
191 G-Sense_Error_Rate -O–CK 100 100 000 - 1
192 Power-Off_Retract_Count -O–CK 100 100 000 - 979
193 Load_Cycle_Count -O–CK 100 100 000 - 1034
194 Temperature_Celsius -O—K 035 044 000 - 35 (0 20 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 081 064 000 - 122903951
197 Current_Pending_Sector -O–C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
240 Head_Flying_Hours ------ 100 253 000 - 23406h+16m+28.805s
241 Total_LBAs_Written ------ 100 253 000 - 41395022644
242 Total_LBAs_Read ------ 100 253 000 - 278507170808
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning

General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 5 Ext. Comprehensive SMART error log
0x04 GPL,SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x0c GPL R/O 2048 Pending Defects log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x24 GPL R/O 512 Current Device Internal Status Data log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa1 GPL,SL VS 24 Device vendor specific log
0xa2 GPL VS 8160 Device vendor specific log
0xa6 GPL VS 192 Device vendor specific log
0xa8-0xa9 GPL,SL VS 136 Device vendor specific log
0xab GPL VS 1 Device vendor specific log
0xb0 GPL VS 9048 Device vendor specific log
0xbd GPL VS 8 Device vendor specific log
0xbe-0xbf GPL VS 65535 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL,SL VS 16 Device vendor specific log
0xc3 GPL,SL VS 8 Device vendor specific log
0xc4 GPL,SL VS 24 Device vendor specific log
0xd1 GPL VS 264 Device vendor specific log
0xd3 GPL VS 1920 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 23441 -

2 Extended offline Completed without error 00% 23428 -

3 Short offline Completed without error 00% 23393 -

4 Short offline Completed without error 00% 23369 -

5 Short offline Completed without error 00% 23345 -

6 Short offline Completed without error 00% 23321 -

7 Short offline Completed without error 00% 23297 -

8 Short offline Completed without error 00% 23273 -

9 Extended offline Interrupted (host reset) 00% 23259 -

#10 Short offline Completed without error 00% 23226 -
#11 Short offline Completed without error 00% 23202 -
#12 Short offline Completed without error 00% 23178 -
#13 Short offline Completed without error 00% 23154 -
#14 Short offline Completed without error 00% 23130 -
#15 Short offline Completed without error 00% 23106 -
#16 Extended offline Completed without error 00% 23093 -
#17 Short offline Completed without error 00% 23058 -
#18 Short offline Completed without error 00% 23034 -
#19 Short offline Completed without error 00% 23010 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version: 3
SCT Version (vendor specific): 522 (0x020a)
Device State: Active (0)
Current Temperature: 35 Celsius
Power Cycle Min/Max Temperature: 35/39 Celsius
Lifetime Min/Max Temperature: 20/44 Celsius
Under/Over Temperature Limit Count: 0/0

SCT Temperature History Version: 2
Temperature Sampling Period: 3 minutes
Temperature Logging Interval: 94 minutes
Min/Max recommended Temperature: 1/61 Celsius
Min/Max Temperature Limit: 2/60 Celsius
Temperature History Size (Index): 128 (0)

Index Estimated Time Temperature Celsius
1 2024-08-25 03:14 33 **************
2 2024-08-25 04:48 ? -
3 2024-08-25 06:22 33 **************
4 2024-08-25 07:56 ? -
5 2024-08-25 09:30 34 ***************
6 2024-08-25 11:04 ? -
7 2024-08-25 12:38 34 ***************
8 2024-08-25 14:12 ? -
9 2024-08-25 15:46 35 ****************
10 2024-08-25 17:20 ? -
11 2024-08-25 18:54 34 ***************
12 2024-08-25 20:28 ? -
13 2024-08-25 22:02 35 ****************
14 2024-08-25 23:36 ? -
15 2024-08-26 01:10 35 ****************
16 2024-08-26 02:44 ? -
17 2024-08-26 04:18 35 ****************
18 2024-08-26 05:52 ? -
19 2024-08-26 07:26 35 ****************
20 2024-08-26 09:00 ? -
21 2024-08-26 10:34 34 ***************
22 2024-08-26 12:08 36 *****************
… …( 8 skipped). … *****************
31 2024-08-27 02:14 36 *****************
32 2024-08-27 03:48 35 ****************
33 2024-08-27 05:22 36 *****************
… …( 6 skipped). … *****************
40 2024-08-27 16:20 36 *****************
41 2024-08-27 17:54 35 ****************
… …( 6 skipped). … ****************
48 2024-08-28 04:52 35 ****************
49 2024-08-28 06:26 36 *****************
50 2024-08-28 08:00 36 *****************
51 2024-08-28 09:34 36 *****************
52 2024-08-28 11:08 ? -
53 2024-08-28 12:42 35 ****************
54 2024-08-28 14:16 ? -
55 2024-08-28 15:50 35 ****************
56 2024-08-28 17:24 36 *****************
57 2024-08-28 18:58 35 ****************
58 2024-08-28 20:32 36 *****************
59 2024-08-28 22:06 35 ****************
… …( 6 skipped). … ****************
66 2024-08-29 09:04 35 ****************
67 2024-08-29 10:38 36 *****************
68 2024-08-29 12:12 36 *****************
69 2024-08-29 13:46 35 ****************
… …( 36 skipped). … ****************
106 2024-08-31 23:44 35 ****************
107 2024-09-01 01:18 38 *******************
… …( 3 skipped). … *******************
111 2024-09-01 07:34 38 *******************
112 2024-09-01 09:08 37 ******************
113 2024-09-01 10:42 37 ******************
114 2024-09-01 12:16 35 ****************
115 2024-09-01 13:50 35 ****************
116 2024-09-01 15:24 36 *****************
117 2024-09-01 16:58 35 ****************
118 2024-09-01 18:32 35 ****************
119 2024-09-01 20:06 36 *****************
120 2024-09-01 21:40 35 ****************
… …( 7 skipped). … ****************
0 2024-09-02 10:12 35 ****************

SCT Error Recovery Control:
Read: Disabled
Write: Disabled

Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 84 — Lifetime Power-On Resets
0x01 0x010 4 23452 — Power-on Hours
0x01 0x018 6 41394160812 — Logical Sectors Written
0x01 0x020 6 91530978 — Number of Write Commands
0x01 0x028 6 277481184393 — Logical Sectors Read
0x01 0x030 6 243640098 — Number of Read Commands
0x01 0x038 6 - — Date and Time TimeStamp
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 23446 — Spindle Motor Power-on Hours
0x03 0x010 4 23441 — Head Flying Hours
0x03 0x018 4 1034 — Head Load Events
0x03 0x020 4 0 — Number of Reallocated Logical Sectors
0x03 0x028 4 0 — Read Recovery Attempts
0x03 0x030 4 0 — Number of Mechanical Start Failures
0x03 0x038 4 0 — Number of Realloc. Candidate Logical Sectors
0x03 0x040 4 979 — Number of High Priority Unload Events
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 — Number of Reported Uncorrectable Errors
0x04 0x010 4 0 — Resets Between Cmd Acceptance and Completion
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 35 — Current Temperature
0x05 0x010 1 35 — Average Short Term Temperature
0x05 0x018 1 34 — Average Long Term Temperature
0x05 0x020 1 44 — Highest Temperature
0x05 0x028 1 0 — Lowest Temperature
0x05 0x030 1 41 — Highest Average Short Term Temperature
0x05 0x038 1 29 — Lowest Average Short Term Temperature
0x05 0x040 1 38 — Highest Average Long Term Temperature
0x05 0x048 1 30 — Lowest Average Long Term Temperature
0x05 0x050 4 0 — Time in Over-Temperature
0x05 0x058 1 70 — Specified Maximum Operating Temperature
0x05 0x060 4 0 — Time in Under-Temperature
0x05 0x068 1 0 — Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 502 — Number of Hardware Resets
0x06 0x010 4 66 — Number of ASR Events
0x06 0x018 4 0 — Number of Interface CRC Errors
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

========================================================================================================================================

root@truenas[~]# smartctl -x /dev/ada4
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke,

=== START OF INFORMATION SECTION ===
Device Model: OOS6000G
Serial Number: 000EX9EJ
LU WWN Device Id: 5 000c50 0e6da71f2
Firmware Version: OOS1
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Sep 2 11:47:34 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Disabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Write SCT (Get) Feature Control Command failed: Input/output error
Wt Cache Reorder: Unknown (SCT Feature Control command failed)

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 559) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 528) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 075 064 044 - 28293368
3 Spin_Up_Time PO---- 093 092 000 - 0
4 Start_Stop_Count -O–CK 100 100 020 - 21
5 Reallocated_Sector_Ct PO–CK 100 100 010 - 0
7 Seek_Error_Rate POSR-- 068 060 045 - 6476959
9 Power_On_Hours -O–CK 100 100 000 - 192
10 Spin_Retry_Count PO–C- 100 100 097 - 0
12 Power_Cycle_Count -O–CK 100 100 020 - 20
18 Unknown_Attribute PO-R-- 100 100 050 - 0
187 Reported_Uncorrect -O–CK 100 100 000 - 0
188 Command_Timeout -O–CK 100 100 000 - 0
190 Airflow_Temperature_Cel -O—K 069 062 000 - 31 (Min/Max 31/38)
192 Power-Off_Retract_Count -O–CK 100 100 000 - 8
193 Load_Cycle_Count -O–CK 100 100 000 - 1331
194 Temperature_Celsius -O—K 031 040 000 - 31 (0 26 0 0 0)
197 Current_Pending_Sector -O–C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
200 Multi_Zone_Error_Rate PO—K 100 100 001 - 0
240 Head_Flying_Hours ------ 100 100 000 - 88 (29 83 0)
241 Total_LBAs_Written ------ 100 253 000 - 3390289196
242 Total_LBAs_Read ------ 100 253 000 - 3235326575
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning

General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 5 Ext. Comprehensive SMART error log
0x04 GPL R/O 256 Device Statistics log
0x04 SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x0a GPL R/W 8 Device Statistics Notification
0x0c GPL R/O 2048 Pending Defects log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x13 GPL R/O 1 SATA NCQ Send and Receive log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x24 GPL R/O 768 Current Device Internal Status Data log
0x2f GPL - 1 Set Sector Configuration
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa1 GPL,SL VS 160 Device vendor specific log
0xa2 GPL VS 16320 Device vendor specific log
0xa4 GPL,SL VS 160 Device vendor specific log
0xa6 GPL VS 192 Device vendor specific log
0xa8-0xa9 GPL,SL VS 136 Device vendor specific log
0xab GPL VS 1 Device vendor specific log
0xad GPL VS 16 Device vendor specific log
0xb1 GPL,SL VS 160 Device vendor specific log
0xb6 GPL VS 1920 Device vendor specific log
0xbe-0xbf GPL VS 65535 Device vendor specific log
0xc1 GPL,SL VS 8 Device vendor specific log
0xc3 GPL,SL VS 24 Device vendor specific log
0xc6 GPL VS 5184 Device vendor specific log
0xc7 GPL,SL VS 8 Device vendor specific log
0xc9 GPL,SL VS 8 Device vendor specific log
0xca GPL,SL VS 16 Device vendor specific log
0xcd GPL,SL VS 1 Device vendor specific log
0xce GPL VS 1 Device vendor specific log
0xcf GPL VS 512 Device vendor specific log
0xd1 GPL VS 656 Device vendor specific log
0xd2 GPL VS 10256 Device vendor specific log
0xd4 GPL VS 2048 Device vendor specific log
0xda GPL,SL VS 1 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 181 -

2 Extended offline Completed without error 00% 165 -

3 Short offline Completed without error 00% 133 -

4 Short offline Completed without error 00% 109 -

5 Short offline Completed without error 00% 85 -

6 Short offline Completed without error 00% 61 -

7 Short offline Completed without error 00% 37 -

8 Short offline Completed without error 00% 13 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version: 3
SCT Version (vendor specific): 522 (0x020a)
Device State: Active (0)
Current Temperature: 31 Celsius
Power Cycle Min/Max Temperature: 31/37 Celsius
Lifetime Min/Max Temperature: 26/38 Celsius
Under/Over Temperature Limit Count: 0/0
SMART Status: 0xc24f (PASSED)
Vendor specific:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version: 2
Temperature Sampling Period: 4 minutes
Temperature Logging Interval: 59 minutes
Min/Max recommended Temperature: 10/40 Celsius
Min/Max Temperature Limit: 5/60 Celsius
Temperature History Size (Index): 128 (102)

Index Estimated Time Temperature Celsius
103 2024-08-28 06:23 32 *************
104 2024-08-28 07:22 31 ************
105 2024-08-28 08:21 31 ************
106 2024-08-28 09:20 32 *************
107 2024-08-28 10:19 31 ************
108 2024-08-28 11:18 32 *************
… …( 3 skipped). … *************
112 2024-08-28 15:14 32 *************
113 2024-08-28 16:13 ? -
114 2024-08-28 17:12 32 *************
115 2024-08-28 18:11 ? -
116 2024-08-28 19:10 32 *************
117 2024-08-28 20:09 32 *************
118 2024-08-28 21:08 31 ************
119 2024-08-28 22:07 32 *************
120 2024-08-28 23:06 34 ***************
121 2024-08-29 00:05 32 *************
122 2024-08-29 01:04 32 *************
123 2024-08-29 02:03 31 ************
124 2024-08-29 03:02 31 ************
125 2024-08-29 04:01 32 *************
126 2024-08-29 05:00 32 *************
127 2024-08-29 05:59 32 *************
0 2024-08-29 06:58 31 ************
1 2024-08-29 07:57 32 *************
2 2024-08-29 08:56 31 ************
3 2024-08-29 09:55 32 *************
4 2024-08-29 10:54 32 *************
5 2024-08-29 11:53 32 *************
6 2024-08-29 12:52 33 **************
7 2024-08-29 13:51 34 ***************
8 2024-08-29 14:50 34 ***************
9 2024-08-29 15:49 32 *************
10 2024-08-29 16:48 32 *************
11 2024-08-29 17:47 31 ************
12 2024-08-29 18:46 31 ************
13 2024-08-29 19:45 32 *************
14 2024-08-29 20:44 31 ************
15 2024-08-29 21:43 32 *************
… …( 5 skipped). … *************
21 2024-08-30 03:37 32 *************
22 2024-08-30 04:36 31 ************
23 2024-08-30 05:35 32 *************
24 2024-08-30 06:34 31 ************
25 2024-08-30 07:33 31 ************
26 2024-08-30 08:32 31 ************
27 2024-08-30 09:31 32 *************
28 2024-08-30 10:30 31 ************
29 2024-08-30 11:29 32 *************
… …( 2 skipped). … *************
32 2024-08-30 14:26 32 *************
33 2024-08-30 15:25 31 ************
34 2024-08-30 16:24 32 *************
35 2024-08-30 17:23 31 ************
36 2024-08-30 18:22 31 ************
37 2024-08-30 19:21 32 *************
… …( 4 skipped). … *************
42 2024-08-31 00:16 32 *************
43 2024-08-31 01:15 31 ************
44 2024-08-31 02:14 31 ************
45 2024-08-31 03:13 32 *************
46 2024-08-31 04:12 31 ************
… …( 15 skipped). … ************
62 2024-08-31 19:56 31 ************
63 2024-08-31 20:55 32 *************
64 2024-08-31 21:54 32 *************
65 2024-08-31 22:53 32 *************
66 2024-08-31 23:52 31 ************
67 2024-09-01 00:51 31 ************
68 2024-09-01 01:50 37 ******************
… …( 6 skipped). … ******************
75 2024-09-01 08:43 37 ******************
76 2024-09-01 09:42 32 *************
77 2024-09-01 10:41 32 *************
78 2024-09-01 11:40 32 *************
79 2024-09-01 12:39 33 **************
80 2024-09-01 13:38 31 ************
81 2024-09-01 14:37 31 ************
82 2024-09-01 15:36 31 ************
83 2024-09-01 16:35 33 **************
84 2024-09-01 17:34 31 ************
85 2024-09-01 18:33 32 *************
86 2024-09-01 19:32 31 ************
87 2024-09-01 20:31 32 *************
88 2024-09-01 21:30 31 ************
… …( 11 skipped). … ************
100 2024-09-02 09:18 31 ************
101 2024-09-02 10:17 32 *************
102 2024-09-02 11:16 31 ************

SCT Error Recovery Control:
Read: 100 (10.0 seconds)
Write: 100 (10.0 seconds)

Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 20 — Lifetime Power-On Resets
0x01 0x010 4 192 — Power-on Hours
0x01 0x018 6 3390289196 — Logical Sectors Written
0x01 0x020 6 4637587 — Number of Write Commands
0x01 0x028 6 3235326563 — Logical Sectors Read
0x01 0x030 6 23652187 — Number of Read Commands
0x01 0x038 6 - — Date and Time TimeStamp
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 192 — Spindle Motor Power-on Hours
0x03 0x010 4 88 — Head Flying Hours
0x03 0x018 4 1331 — Head Load Events
0x03 0x020 4 0 — Number of Reallocated Logical Sectors
0x03 0x028 4 0 — Read Recovery Attempts
0x03 0x030 4 0 — Number of Mechanical Start Failures
0x03 0x038 4 0 — Number of Realloc. Candidate Logical Sectors
0x03 0x040 4 8 — Number of High Priority Unload Events
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 — Number of Reported Uncorrectable Errors
0x04 0x010 4 0 — Resets Between Cmd Acceptance and Completion
0x04 0x018 4 0 -D- Physical Element Status Changed
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 31 — Current Temperature
0x05 0x010 1 31 — Average Short Term Temperature
0x05 0x018 1 - — Average Long Term Temperature
0x05 0x020 1 37 — Highest Temperature
0x05 0x028 1 31 — Lowest Temperature
0x05 0x030 1 34 — Highest Average Short Term Temperature
0x05 0x038 1 31 — Lowest Average Short Term Temperature
0x05 0x040 1 - — Highest Average Long Term Temperature
0x05 0x048 1 - — Lowest Average Long Term Temperature
0x05 0x050 4 0 — Time in Over-Temperature
0x05 0x058 1 60 — Specified Maximum Operating Temperature
0x05 0x060 4 0 — Time in Under-Temperature
0x05 0x068 1 5 — Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 55 — Number of Hardware Resets
0x06 0x010 4 1 — Number of ASR Events
0x06 0x018 4 0 — Number of Interface CRC Errors
0xff ===== = = === == Vendor Specific Statistics (rev 1) ==
0xff 0x008 7 0 — Vendor Specific
0xff 0x010 7 0 — Vendor Specific
0xff 0x018 7 0 — Vendor Specific
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description

============================================================================================================================

root@truenas[~]# smartctl -x /dev/ada5
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke,

=== START OF INFORMATION SECTION ===
Model Family: Seagate IronWolf
Device Model: ST6000VN001-2BB186
Serial Number: ZR12SNEW
LU WWN Device Id: 5 000c50 0e38123fe
Firmware Version: SC60
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5425 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Sep 2 11:50:42 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 710) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 071 064 006 - 13270416
3 Spin_Up_Time PO---- 092 091 000 - 0
4 Start_Stop_Count -O–CK 100 100 020 - 84
5 Reallocated_Sector_Ct PO–CK 100 100 010 - 0
7 Seek_Error_Rate POSR-- 087 060 045 - 503774569
9 Power_On_Hours -O–CK 074 074 000 - 23452
10 Spin_Retry_Count PO–C- 100 100 097 - 0
12 Power_Cycle_Count -O–CK 100 100 020 - 84
183 Runtime_Bad_Block -O–CK 100 100 000 - 0
184 End-to-End_Error -O–CK 100 100 099 - 0
187 Reported_Uncorrect -O–CK 100 100 000 - 0
188 Command_Timeout -O–CK 100 100 000 - 0
189 High_Fly_Writes -O-RCK 100 100 000 - 0
190 Airflow_Temperature_Cel -O—K 064 057 040 - 36 (Min/Max 36/41)
191 G-Sense_Error_Rate -O–CK 100 100 000 - 0
192 Power-Off_Retract_Count -O–CK 100 100 000 - 988
193 Load_Cycle_Count -O–CK 100 100 000 - 1041
194 Temperature_Celsius -O—K 036 043 000 - 36 (0 20 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 071 064 000 - 13270416
197 Current_Pending_Sector -O–C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
240 Head_Flying_Hours ------ 100 253 000 - 23406h+16m+27.134s
241 Total_LBAs_Written ------ 100 253 000 - 41492928220
242 Total_LBAs_Read ------ 100 253 000 - 275401013512
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning

General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 5 Ext. Comprehensive SMART error log
0x04 GPL,SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x0c GPL R/O 2048 Pending Defects log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x24 GPL R/O 512 Current Device Internal Status Data log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa1 GPL,SL VS 24 Device vendor specific log
0xa2 GPL VS 8160 Device vendor specific log
0xa6 GPL VS 192 Device vendor specific log
0xa8-0xa9 GPL,SL VS 136 Device vendor specific log
0xab GPL VS 1 Device vendor specific log
0xb0 GPL VS 9048 Device vendor specific log
0xbd GPL VS 8 Device vendor specific log
0xbe-0xbf GPL VS 65535 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL,SL VS 16 Device vendor specific log
0xc3 GPL,SL VS 8 Device vendor specific log
0xc4 GPL,SL VS 24 Device vendor specific log
0xd1 GPL VS 264 Device vendor specific log
0xd3 GPL VS 1920 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 23440 -

2 Extended offline Completed without error 00% 23427 -

3 Short offline Completed without error 00% 23392 -

4 Short offline Completed without error 00% 23368 -

5 Short offline Completed without error 00% 23344 -

6 Short offline Completed without error 00% 23321 -

7 Short offline Completed without error 00% 23297 -

8 Short offline Completed without error 00% 23273 -

9 Extended offline Interrupted (host reset) 00% 23259 -

#10 Short offline Completed without error 00% 23226 -
#11 Short offline Completed without error 00% 23202 -
#12 Short offline Completed without error 00% 23178 -
#13 Short offline Completed without error 00% 23154 -
#14 Short offline Completed without error 00% 23129 -
#15 Short offline Completed without error 00% 23105 -
#16 Extended offline Completed without error 00% 23093 -
#17 Short offline Completed without error 00% 23058 -
#18 Short offline Completed without error 00% 23034 -
#19 Short offline Completed without error 00% 23010 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version: 3
SCT Version (vendor specific): 522 (0x020a)
Device State: Active (0)
Current Temperature: 36 Celsius
Power Cycle Min/Max Temperature: 36/41 Celsius
Lifetime Min/Max Temperature: 20/43 Celsius
Under/Over Temperature Limit Count: 0/0

SCT Temperature History Version: 2
Temperature Sampling Period: 3 minutes
Temperature Logging Interval: 94 minutes
Min/Max recommended Temperature: 1/61 Celsius
Min/Max Temperature Limit: 2/60 Celsius
Temperature History Size (Index): 128 (0)

Index Estimated Time Temperature Celsius
1 2024-08-25 04:48 34 ***************
2 2024-08-25 06:22 ? -
3 2024-08-25 07:56 34 ***************
4 2024-08-25 09:30 ? -
5 2024-08-25 11:04 35 ****************
6 2024-08-25 12:38 ? -
7 2024-08-25 14:12 35 ****************
8 2024-08-25 15:46 ? -
9 2024-08-25 17:20 36 *****************
10 2024-08-25 18:54 ? -
11 2024-08-25 20:28 35 ****************
12 2024-08-25 22:02 ? -
13 2024-08-25 23:36 36 *****************
14 2024-08-26 01:10 ? -
15 2024-08-26 02:44 36 *****************
16 2024-08-26 04:18 ? -
17 2024-08-26 05:52 36 *****************
18 2024-08-26 07:26 ? -
19 2024-08-26 09:00 36 *****************
20 2024-08-26 10:34 ? -
21 2024-08-26 12:08 35 ****************
22 2024-08-26 13:42 37 ******************
… …( 2 skipped). … ******************
25 2024-08-26 18:24 37 ******************
26 2024-08-26 19:58 38 *******************
27 2024-08-26 21:32 38 *******************
28 2024-08-26 23:06 37 ******************
29 2024-08-27 00:40 38 *******************
30 2024-08-27 02:14 38 *******************
31 2024-08-27 03:48 37 ******************
… …( 9 skipped). … ******************
41 2024-08-27 19:28 37 ******************
42 2024-08-27 21:02 36 *****************
… …( 4 skipped). … *****************
47 2024-08-28 04:52 36 *****************
48 2024-08-28 06:26 37 ******************
… …( 2 skipped). … ******************
51 2024-08-28 11:08 37 ******************
52 2024-08-28 12:42 ? -
53 2024-08-28 14:16 37 ******************
54 2024-08-28 15:50 ? -
55 2024-08-28 17:24 36 *****************
56 2024-08-28 18:58 37 ******************
… …( 2 skipped). … ******************
59 2024-08-28 23:40 37 ******************
60 2024-08-29 01:14 36 *****************
61 2024-08-29 02:48 36 *****************
62 2024-08-29 04:22 37 ******************
63 2024-08-29 05:56 36 *****************
64 2024-08-29 07:30 36 *****************
65 2024-08-29 09:04 37 ******************
66 2024-08-29 10:38 37 ******************
67 2024-08-29 12:12 37 ******************
68 2024-08-29 13:46 38 *******************
69 2024-08-29 15:20 37 ******************
70 2024-08-29 16:54 36 *****************
… …( 3 skipped). … *****************
74 2024-08-29 23:10 36 *****************
75 2024-08-30 00:44 37 ******************
76 2024-08-30 02:18 36 *****************
… …( 4 skipped). … *****************
81 2024-08-30 10:08 36 *****************
82 2024-08-30 11:42 37 ******************
83 2024-08-30 13:16 37 ******************
84 2024-08-30 14:50 36 *****************
85 2024-08-30 16:24 36 *****************
86 2024-08-30 17:58 36 *****************
87 2024-08-30 19:32 37 ******************
88 2024-08-30 21:06 37 ******************
89 2024-08-30 22:40 36 *****************
… …( 16 skipped). … *****************
106 2024-09-01 01:18 36 *****************
107 2024-09-01 02:52 41 **********************
108 2024-09-01 04:26 40 *********************
… …( 2 skipped). … *********************
111 2024-09-01 09:08 40 *********************
112 2024-09-01 10:42 39 ********************
113 2024-09-01 12:16 39 ********************
114 2024-09-01 13:50 36 *****************
115 2024-09-01 15:24 36 *****************
116 2024-09-01 16:58 37 ******************
117 2024-09-01 18:32 36 *****************
118 2024-09-01 20:06 36 *****************
119 2024-09-01 21:40 37 ******************
120 2024-09-01 23:14 36 *****************
… …( 7 skipped). … *****************
0 2024-09-02 11:46 36 *****************

SCT Error Recovery Control:
Read: Disabled
Write: Disabled

Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 84 — Lifetime Power-On Resets
0x01 0x010 4 23452 — Power-on Hours
0x01 0x018 6 41492065948 — Logical Sectors Written
0x01 0x020 6 82105404 — Number of Write Commands
0x01 0x028 6 274373908609 — Logical Sectors Read
0x01 0x030 6 291331911 — Number of Read Commands
0x01 0x038 6 - — Date and Time TimeStamp
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 23445 — Spindle Motor Power-on Hours
0x03 0x010 4 23441 — Head Flying Hours
0x03 0x018 4 1041 — Head Load Events
0x03 0x020 4 0 — Number of Reallocated Logical Sectors
0x03 0x028 4 0 — Read Recovery Attempts
0x03 0x030 4 0 — Number of Mechanical Start Failures
0x03 0x038 4 0 — Number of Realloc. Candidate Logical Sectors
0x03 0x040 4 988 — Number of High Priority Unload Events
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 — Number of Reported Uncorrectable Errors
0x04 0x010 4 0 — Resets Between Cmd Acceptance and Completion
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 36 — Current Temperature
0x05 0x010 1 36 — Average Short Term Temperature
0x05 0x018 1 35 — Average Long Term Temperature
0x05 0x020 1 43 — Highest Temperature
0x05 0x028 1 0 — Lowest Temperature
0x05 0x030 1 41 — Highest Average Short Term Temperature
0x05 0x038 1 29 — Lowest Average Short Term Temperature
0x05 0x040 1 37 — Highest Average Long Term Temperature
0x05 0x048 1 30 — Lowest Average Long Term Temperature
0x05 0x050 4 0 — Time in Over-Temperature
0x05 0x058 1 70 — Specified Maximum Operating Temperature
0x05 0x060 4 0 — Time in Under-Temperature
0x05 0x068 1 0 — Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 502 — Number of Hardware Resets
0x06 0x010 4 66 — Number of ASR Events
0x06 0x018 4 0 — Number of Interface CRC Errors
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description

essinghigh · September 2, 2024, 2:56pm

Quick things that are sticking out to me, which means I’m guessing these are the drives that failed?:

/dev/ada2 - High Raw Read Error Rate, High Seek Error Rate
/dev/ada3 - High Raw Read Error Rate, High Seek Error Rate
/dev/ada4 - High Raw Read Error Rate, High Seek Error Rate
/dev/ada5 - High Raw Read Error Rate, High Seek Error Rate

HemaN · September 2, 2024, 4:15pm

No, these are the drives that are now working. Ada0 and ada1 are the boot drives, ada2 to ada5 are second pool I upgraded from 2 disk mirror to 4 disk RAID-Z2. I can’t find a way to view historic data for drives that are disconnected (4 disks I sent for data recovery).

So what can cause these read and seek errors? Drives ada0 to ada5 are connected directly to motherboard.

Jorsher · September 2, 2024, 4:44pm

Did you try those drives directly connected or in another system before sending back?

The chance of four drives failing simultaneously is next to nothing. If they did, it was likely due to a hardware issue up the chain.

HemaN · September 3, 2024, 3:06pm

My first though was to connect all 4 drives to my own Truenas server, but when I heard strange sounds like faulty heads in disks I resigned from that idea, becouse I was affraiding that futher torment of affected drives can make things worst and will be harder to recover data. Only thing I checked was to connect affected drives directly to motherboard instead of HBA, but it didn’t change anything.

HemaN · October 26, 2024, 10:12am

Update:

I received all new drives as a replacement for faulty ones and all data was recovered, so thats a good news. But I’m still struggling with server trying to figure out what is wrong with it. While faulty disks was sent for recovery I plugged old drives to HBA just to try what can be wrong. And this rised more question than answers. I knew, that these old drives can be already faulty, but strange thing is that no matter which drive I connected, always disk recognized as da0 was the one that was failing. And it was independent from which SFF-8087 cable sata connectors it was connected, same with second end of this cable, switching to second HBA output didn’t change enything. I’m also tried to switch power cables between drives, but it made no difference. And what is funny, drives that was passing SMART test, LONG and SHORT, for the last 3 weeks, as soon as they where recognized as da0 becouse I disconected faulty drive perviously recognized as da0, suddenly started to report errors after next SMART test. Today I disconnected third and fourth drive, from which one drive was recognized as da0, becouse it was faulty and left only two drives connected to HBA. Guess what happened? As soon as another drive get da0 label it stopped passing SMART test.

And in the meantime server experienced one of the boot drive death, at the same time second one that was in the mirror lost bootloader file which prevent server from boot. After instaling fresh install on that survived drive everything started to work again. Boot drives was two (now one left) Goodram CL100 120 GB, so this can be just cheap and poor quality drive that die becouse of load, but again strange thing is that second one lost bootloader file at the same time, although with no problems passing all SMART test even today. And those boot drives was connected directly to motherboard SATA ports. I ordered Intel 3610 200 GB as a replacement and until I get it I connected WD blue 240GB as a mirror boot drive.

Second pool consisting four 6TB drives all the time is up and running with no problems. I’m affraiding to connect 4 new drives to HBA to not damage it. I’m planning to replace PSU and check whether this will change anything. Last thing to check is HBA card, but still can’t beleive that HBA can damage drives.

If someone have an idea what can be wrong please let me know .

HemaN · November 16, 2024, 6:15pm

Update 2:

Problem finally solved! I never ever had such a strange issue as this. One of the Silverstone PSU cable have only 3 molex connectors on it and I was using 2 of them to power drives (HBA ones) through a two pieces of molex-> 2 x SATA adapters. I used third molex connector to connect old Zalman 120 mm fan to cool HBA which only have a molex connector. It turns out that this Zalman fan and Ironwolf drives do not like each other and when they are connected to this practicular Silverstone PSU via one cable fan is causing some kind of interference manifested by random logical errors in ZFS file system which is leading to disk randomly showing up as faulty or unavailable. As soon as I disconnected fan all problems just gone and everyting is working like a charm. What is even stranger – when I was searching for reason of problems I disconnected all Ironwolfs from HBA and connected my spare WD Red 1TB drives to infected system and it did work without problems.

I’m suprised that Silverstone PSU didn’t filtered interferences. One thing is sure, this Zalman fan is going to junk, I should buy new fan instead of doing recycling. Lesson learned.