Pool resilverer after a disk died question :)

jeffv03 · March 22, 2025, 7:04am

Hello folks ^^)
one of my Truenas scale disk died (click click sound, no spin) and got ‘ejected’ from the RaidZ2 pool by the system. so I popped it out and placed a new disk at its open slot and ticked ‘replace’ and the resilver debuted.

But, to my surprise, although the system said it had enough stuff to recreate the disk contents entirely, the resilver craaaaaaaaaaaaawl and this is what I get today :

What the heck ! 39 YEARS to resilver ONE disk ??? what is happening there ?

Help please…

Jeff

Johnny_Fartpants · March 22, 2025, 7:26am

Hi Jeff,

Can you share the output of zpool status and also smartctl -a /dev/sdX for some of your drives. Perhaps an old one and the new one you’ve added.

etorix · March 22, 2025, 7:39am

Time estimates at the beginning of the process are WAY off…

Sanity check: Are there any SMR drives involved?

jeffv03 · March 22, 2025, 9:16am

here it are :

zpool status

elNas[~]$ sudo zpool status
pool: elZ2
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Mar 15 12:02:30 2025
964G / 118T scanned at 15.6M/s, 5.38G / 118T issued at 89.0K/s
285M resilvered, 0.00% done, no estimated completion time
expand: expansion of raidz2-0 in progress since Thu Feb 27 15:19:37 2025
5.28T / 118T copied at 2.81M/s, 4.48% done, paused for resilver or clear
config:

    NAME                                        STATE     READ WRITE CKSUM
    elZ2                                          DEGRADED     0     0     0
      raidz2-0                                  DEGRADED 3.75K     9     0
        aab036a6-7e2b-4632-b16d-cd7866845e5e    ONLINE       0     0     0
        7a8f8da1-836b-4a4c-9eaa-2aa29da2beb7    ONLINE       0     0     0
        a10d9ac9-ef2c-4ede-9e6b-eba01e901361    ONLINE       0     0     0
        9f2dce88-cde7-4923-b0bb-bf4de890ea52    ONLINE       0     0     0
        replacing-4                             DEGRADED 8.83K    14     0
          b74d4b85-0436-4814-8515-b405d7e40e24  REMOVED      0     0     0
          181fbb56-496c-4540-a107-d35504e0d61b  ONLINE   8.85K    16     0  (resilvering)
        96d445ec-e8b2-42ce-8f1a-63ca963eead9    FAULTED  1.90K 10.2K     3  too many errors
        b4dbc538-a334-417c-9b6a-7265693bc710    REMOVED      0     0     0
        fd7f4ef7-237a-48c0-8a72-bebaf1cd3fb5    ONLINE       0     0     0
        83e62634-a0a5-4c2f-abb8-10b763833042    ONLINE       0     0     0
        ea3b911a-d478-4341-b4cd-4549385fcdde    ONLINE       0     0     0
        52c0cc19-1576-4186-b41f-8ee005e7ebaf    ONLINE       0     0     0
        ee9542ae-6266-4615-bcaa-6a0f06dafb87    ONLINE       0     0     0
        0b764a40-fd5d-4631-922d-625178717347    ONLINE       0     0     0
        e0df314c-cc1e-460e-9591-842264306d5b    ONLINE       0     0     0
        af112e1e-211c-4061-a0bf-4e7f943903ca    ONLINE       0     0     0
        c2c7b568-7eff-4f85-998b-eb4e1ac51897    ONLINE       0     0     0
        ebd9e063-ebb9-4bdb-9ac6-3302a748390a    ONLINE       0     0     0
    cache
      b7523e48-e07b-4fbb-94c5-c6de97a7d33c      ONLINE       0     0     0

errors: 289356 data errors, use ‘-v’ for a list

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:01:57 with 0 errors on Tue Mar 18 03:46:58 2025
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      sds3      ONLINE       0     0     0

errors: No known data errors

Smart stuff for /dev/sds that was logged as 'not smart capable/responding' by a warning...

elNas[~]$ sudo smartctl -ax /dev/sds
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-1ER166
Serial Number: Z500Q8NE
LU WWN Device Id: 5 000c50 079c7c9d4
Firmware Version: CC25
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5528
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is: Sat Mar 22 10:14:02 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Disabled
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Unavailable

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 89) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 338) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x1085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 117 099 006 - 153135864
3 Spin_Up_Time PO---- 094 093 000 - 0
4 Start_Stop_Count -O–CK 098 098 020 - 2912
5 Reallocated_Sector_Ct PO–CK 100 100 010 - 0
7 Seek_Error_Rate POSR-- 057 052 030 - 1159761830886
9 Power_On_Hours -O–CK 049 049 000 - 45249
10 Spin_Retry_Count PO–C- 100 100 097 - 0
12 Power_Cycle_Count -O–CK 098 098 020 - 2778
183 Runtime_Bad_Block -O–CK 099 099 000 - 1
184 End-to-End_Error -O–CK 100 100 099 - 0
187 Reported_Uncorrect -O–CK 100 100 000 - 0
188 Command_Timeout -O–CK 099 099 000 - 1 1 3
189 High_Fly_Writes -O-RCK 096 096 000 - 4
190 Airflow_Temperature_Cel -O—K 062 047 045 - 38 (Min/Max 20/39)
191 G-Sense_Error_Rate -O–CK 100 100 000 - 0
192 Power-Off_Retract_Count -O–CK 100 100 000 - 1705
193 Load_Cycle_Count -O–CK 024 024 000 - 153691
194 Temperature_Celsius -O—K 038 053 000 - 38 (0 13 0 0 0)
197 Current_Pending_Sector -O–C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
240 Head_Flying_Hours ------ 100 253 000 - 21647h+29m+38.396s
241 Total_LBAs_Written ------ 100 253 000 - 36482369806
242 Total_LBAs_Read ------ 100 253 000 - 82677010674
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning

General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 5 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa1 GPL,SL VS 20 Device vendor specific log
0xa2 GPL VS 4496 Device vendor specific log
0xa8 GPL,SL VS 129 Device vendor specific log
0xa9 GPL,SL VS 1 Device vendor specific log
0xab GPL VS 1 Device vendor specific log
0xb0 GPL VS 5176 Device vendor specific log
0xbe-0xbf GPL VS 65535 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL,SL VS 10 Device vendor specific log
0xc3 GPL,SL VS 8 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Error Log Version: 1
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Extended offline Aborted by host 90% 27717 -

2 Extended offline Interrupted (host reset) 80% 27551 -

3 Extended offline Completed without error 00% 27387 -

4 Extended offline Completed without error 00% 27218 -

5 Extended offline Completed without error 00% 27051 -

6 Extended offline Completed without error 00% 26884 -

7 Extended offline Completed without error 00% 26716 -

8 Extended offline Completed without error 00% 26548 -

9 Extended offline Completed without error 00% 26379 -

#10 Extended offline Completed without error 00% 26211 -
#11 Extended offline Completed without error 00% 26051 -
#12 Extended offline Completed without error 00% 25884 -
#13 Extended offline Completed without error 00% 25716 -
#14 Extended offline Completed without error 00% 25548 -
#15 Extended offline Completed without error 00% 25380 -
#16 Extended offline Completed without error 00% 25212 -
#17 Extended offline Completed without error 00% 25044 -
#18 Extended offline Completed without error 00% 24876 -
#19 Extended offline Completed without error 00% 24708 -

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Extended offline Aborted by host 90% 27717 -

2 Extended offline Interrupted (host reset) 80% 27551 -

3 Extended offline Completed without error 00% 27387 -

4 Extended offline Completed without error 00% 27218 -

5 Extended offline Completed without error 00% 27051 -

6 Extended offline Completed without error 00% 26884 -

7 Extended offline Completed without error 00% 26716 -

8 Extended offline Completed without error 00% 26548 -

9 Extended offline Completed without error 00% 26379 -

#10 Extended offline Completed without error 00% 26211 -
#11 Extended offline Completed without error 00% 26051 -
#12 Extended offline Completed without error 00% 25884 -
#13 Extended offline Completed without error 00% 25716 -
#14 Extended offline Completed without error 00% 25548 -
#15 Extended offline Completed without error 00% 25380 -
#16 Extended offline Completed without error 00% 25212 -
#17 Extended offline Completed without error 00% 25044 -
#18 Extended offline Completed without error 00% 24876 -
#19 Extended offline Completed without error 00% 24708 -
#20 Extended offline Completed without error 00% 24540 -
#21 Extended offline Completed without error 00% 24372 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version: 3
SCT Version (vendor specific): 522 (0x020a)
Device State: Active (0)
Current Temperature: 38 Celsius
Power Cycle Min/Max Temperature: 20/39 Celsius
Lifetime Min/Max Temperature: 13/53 Celsius
Under/Over Temperature Limit Count: 0/0

SCT Data Table command not supported

SCT Error Recovery Control command not supported

Device Statistics (GP/SMART Log 0x04) not supported

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x000a 2 22 Device-to-host register FISes sent due to a COMRESET
0x0001 2 0 Command failed due to ICRC error
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS

jeffv03 · March 22, 2025, 9:24am

and the same for /dev/sdt that’s ticked as ‘faulted’ :

Summary

elNas[~]$ sudo smartctl -ax /dev/sdt
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: ST10000NM0046
Serial Number: ZA23L3YS
LU WWN Device Id: 5 000c50 0a57409c6
Firmware Version: G00A
User Capacity: 10,009,952,870,400 bytes [10.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database 7.3/5528
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 22 10:20:53 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1], Master PW ID: 0xfffd
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 575) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 934) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x50bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 076 064 044 - 37596860
3 Spin_Up_Time PO---- 097 094 000 - 0
4 Start_Stop_Count -O–CK 079 079 020 - 21523
5 Reallocated_Sector_Ct PO–CK 100 100 010 - 0
7 Seek_Error_Rate POSR-- 083 060 045 - 186697981
9 Power_On_Hours -O–CK 100 100 000 - 820
10 Spin_Retry_Count PO–C- 100 100 097 - 0
12 Power_Cycle_Count -O–CK 079 079 020 - 21530
184 End-to-End_Error -O–CK 100 100 099 - 0
187 Reported_Uncorrect -O–CK 100 100 000 - 0
188 Command_Timeout -O–CK 100 100 000 - 0
189 High_Fly_Writes -O-RCK 060 060 000 - 40
190 Airflow_Temperature_Cel -O—K 069 065 040 - 31 (Min/Max 31/32)
191 G-Sense_Error_Rate -O–CK 099 099 000 - 2590
192 Power-Off_Retract_Count -O–CK 090 090 000 - 21538
193 Load_Cycle_Count -O–CK 090 090 000 - 21990
194 Temperature_Celsius -O—K 031 040 000 - 31 (0 21 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 008 002 000 - 37596860
197 Current_Pending_Sector -O–C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
200 Multi_Zone_Error_Rate PO—K 100 100 001 - 0
240 Head_Flying_Hours ------ 100 253 000 - 738 (48 226 0)
241 Total_LBAs_Written ------ 100 253 000 - 98475864760
242 Total_LBAs_Read ------ 100 253 000 - 37346883016
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning

General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 5 Ext. Comprehensive SMART error log
0x04 GPL R/O 256 Device Statistics log
0x04 SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x0c GPL R/O 2048 Pending Defects log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x13 GPL R/O 1 SATA NCQ Send and Receive log
0x15 GPL R/W 1 Rebuild Assist log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x24 GPL R/O 768 Current Device Internal Status Data log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa1 GPL,SL VS 24 Device vendor specific log
0xa2 GPL VS 16320 Device vendor specific log
0xa6 GPL VS 192 Device vendor specific log
0xa8-0xa9 GPL,SL VS 136 Device vendor specific log
0xab GPL VS 1 Device vendor specific log
0xad GPL VS 16 Device vendor specific log
0xaf GPL,SL VS 1 Device vendor specific log
0xb0 GPL VS 17208 Device vendor specific log
0xbe-0xbf GPL VS 65535 Device vendor specific log
0xc1 GPL,SL VS 16 Device vendor specific log
0xc3 GPL,SL VS 8 Device vendor specific log
0xc9 GPL,SL VS 8 Device vendor specific log
0xca GPL,SL VS 16 Device vendor specific log
0xd1 GPL VS 304 Device vendor specific log
0xd2 GPL VS 10000 Device vendor specific log
0xd4 GPL VS 2048 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Error Log Version: 1
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version: 3
SCT Version (vendor specific): 522 (0x020a)
Device State: Active (0)
Current Temperature: 31 Celsius
Power Cycle Min/Max Temperature: 31/32 Celsius
Lifetime Min/Max Temperature: 21/35 Celsius
Under/Over Temperature Limit Count: 0/19
Vendor specific:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 3f 01 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version: 2
Temperature Sampling Period: 3 minutes
Temperature Logging Interval: 59 minutes
Min/Max recommended Temperature: 0/ 0 Celsius
Min/Max Temperature Limit: 0/ 0 Celsius
Temperature History Size (Index): 128 (76)

Index Estimated Time Temperature Celsius
77 2025-03-17 04:38 29 **********
78 2025-03-17 05:37 29 **********
79 2025-03-17 06:36 29 **********
80 2025-03-17 07:35 28 *********
81 2025-03-17 08:34 29 **********
… …( 3 skipped). … **********
85 2025-03-17 12:30 29 **********
86 2025-03-17 13:29 30 ***********
87 2025-03-17 14:28 29 **********
… …( 15 skipped). … **********
103 2025-03-18 06:12 29 **********
104 2025-03-18 07:11 28 *********
105 2025-03-18 08:10 29 **********
106 2025-03-18 09:09 29 **********
107 2025-03-18 10:08 29 **********
108 2025-03-18 11:07 30 ***********
109 2025-03-18 12:06 30 ***********
110 2025-03-18 13:05 ? -
111 2025-03-18 14:04 30 ***********
112 2025-03-18 15:03 ? -
113 2025-03-18 16:02 30 ***********
… …( 22 skipped). … ***********
8 2025-03-19 14:39 30 ***********
9 2025-03-19 15:38 29 **********
10 2025-03-19 16:37 30 ***********
… …( 20 skipped). … ***********
31 2025-03-20 13:16 30 ***********
32 2025-03-20 14:15 31 ************
… …( 23 skipped). … ************
56 2025-03-21 13:51 31 ************
57 2025-03-21 14:50 ? -
58 2025-03-21 15:49 31 ************
… …( 8 skipped). … ************
67 2025-03-22 00:40 31 ************
68 2025-03-22 01:39 32 *************
69 2025-03-22 02:38 31 ************
… …( 6 skipped). … ************
76 2025-03-22 09:31 31 ************

SCT Error Recovery Control:
Read: 40 (4.0 seconds)
Write: 40 (4.0 seconds)

Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 21530 — Lifetime Power-On Resets
0x01 0x010 4 820 — Power-on Hours
0x01 0x018 6 82299394600 — Logical Sectors Written
0x01 0x020 6 801227294 — Number of Write Commands
0x01 0x028 6 35772935892 — Logical Sectors Read
0x01 0x030 6 721663423 — Number of Read Commands
0x01 0x038 6 - — Date and Time TimeStamp
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 687 — Spindle Motor Power-on Hours
0x03 0x010 4 311 — Head Flying Hours
0x03 0x018 4 21990 — Head Load Events
0x03 0x020 4 0 — Number of Reallocated Logical Sectors
0x03 0x028 4 0 — Read Recovery Attempts
0x03 0x030 4 0 — Number of Mechanical Start Failures
0x03 0x038 4 0 — Number of Realloc. Candidate Logical Sectors
0x03 0x040 4 21538 — Number of High Priority Unload Events
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 — Number of Reported Uncorrectable Errors
0x04 0x010 4 0 — Resets Between Cmd Acceptance and Completion
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 31 — Current Temperature
0x05 0x010 1 31 — Average Short Term Temperature
0x05 0x018 1 - — Average Long Term Temperature
0x05 0x020 1 35 — Highest Temperature
0x05 0x028 1 25 — Lowest Temperature
0x05 0x030 1 34 — Highest Average Short Term Temperature
0x05 0x038 1 28 — Lowest Average Short Term Temperature
0x05 0x040 1 - — Highest Average Long Term Temperature
0x05 0x048 1 - — Lowest Average Long Term Temperature
0x05 0x050 4 0 — Time in Over-Temperature
0x05 0x058 1 60 — Specified Maximum Operating Temperature
0x05 0x060 4 0 — Time in Under-Temperature
0x05 0x068 1 5 — Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 88 — Number of Hardware Resets
0x06 0x010 4 0 — Number of ASR Events
0x06 0x018 4 0 — Number of Interface CRC Errors
0xff ===== = = === == Vendor Specific Statistics (rev 1) ==
0xff 0x008 7 0 — Vendor Specific
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x000a 2 2 Device-to-host register FISes sent due to a COMRESET
0x0001 2 0 Command failed due to ICRC error
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS

Seagate FARM log (GP Log 0xa6) supported [try: -l farm]

jeffv03 · March 22, 2025, 9:30am

oups, last note : the change of disk and resilvering is running since one week now, not just today, as some jerks are toying regularly with the EDF here making me want to storm their office with military grade killing equipment…
my UPSes makes screams so often that I’m hurt for it.
I really don’t like that kind of play from the electricity provider… should I call my lawyers ?

prez02 · March 22, 2025, 9:31am

Is that a unfinished raidz expansion?

Barracudas are listed as being SMR, but maybe this is an older model that predates the introduction of SMR?

jeffv03 · March 22, 2025, 9:32am

yes, it had just begun when the poor disk died and was put on hold until the dead ones been replaced…

jeffv03 · March 22, 2025, 9:34am

The barracudda disk has nothing to do with the Raidz2 pool has being the system disk and yes it is old but still no deffects on it

jeffv03 · March 22, 2025, 9:36am

all the RaidZ2 disks are 10 TB, most of it enterprise grade / helium NAS/datacenter ones, I replace the consumer grade ones as I can.

prez02 · March 22, 2025, 9:45am

So I there might be SMR drives in that pool?!

Arwen · March 22, 2025, 9:50am

One comment. Very wide, (larger than say 12), RAID-Zx are not recommended. You have 17 disks, (probably including the new one for expansion).

Sometimes they work well, and other times extreme slowness is the result. Their is no “fix” if experiencing that problem, other than backup and rebuild.

Now there is some debate on what is too wide for a single ZFS RAID-Zx vDev. I put it at 12 and others say more or less. But, in my opinion, 17 is noticeably over the threshold.

Remember, in a full width stripe of a 17 disk RAID-Z2, replacement needs to read 15 disks to re-create the data for the failed disk. Of course, ZFS will vary the stripe width if a file does not fit the full width at the time it was written.

etorix · March 22, 2025, 9:54am

Probably not in 10 TB capacity, but the single vdev is too wide and there are too many issues.
8k read errors on the replacement drive, another drive faulted with 10k write errors and yet another drive removed. 289k data errors
Do you have a backup?

prez02 · March 22, 2025, 9:58am

Phew, you are right, Seagate at least lists all 10TB drives as CMR…

jeffv03 · March 22, 2025, 10:08am

if you see 17 disks too wide, how to see 60 raid disks racks in datacenters ?

etorix · March 22, 2025, 10:10am

Multiple vdevs. Or dRAID.

jeffv03 · March 22, 2025, 10:11am

dRaid ? don’t mind found it

is dRAID possible with TrueNAS Scale ?

jeffv03 · March 22, 2025, 10:16am

ho, the dead disk being replaced was an Helium NAS grade disk almost new…

jeffv03 · March 22, 2025, 4:16pm

hum, if I look deeper it says that the pool io is suspended, the apps aren’t available as ix-apps can’t be mounted, and I can’t list the errored files .

btw, the 200000 errors comes all from the dead disk that is no more in the machine.

and, lastly, before the disk’s death the RaidZ2 pool was working well with NO errors after it’s been resilvered 'cause some disks was giving ‘errors’ (read and write) that, after the offending disks was replaced and tested in another machine was exempt of any errors and came both with OK smart tests stating no defects at all…

with RaidZ2 I’m censed being ok to have TWO disks dieing at once without loosing one byte of data and a single problem to rebuild the Raid no ?

I use the same disks brand/models in my other machines (not in Raid ! and in ext4 ) and haven’t had a single problem with since years of use. only my nas throw me errors and failures

is Truenas scale ZFS really stable and fine working at all ???

Johnny_Fartpants · March 22, 2025, 4:35pm

So lets try and unpick this.

NAME                                        STATE     READ WRITE CKSUM
    elZ2                                          DEGRADED     0     0     0
      raidz2-0                                  DEGRADED 3.75K     9     0
        aab036a6-7e2b-4632-b16d-cd7866845e5e    ONLINE       0     0     0
        7a8f8da1-836b-4a4c-9eaa-2aa29da2beb7    ONLINE       0     0     0
        a10d9ac9-ef2c-4ede-9e6b-eba01e901361    ONLINE       0     0     0
        9f2dce88-cde7-4923-b0bb-bf4de890ea52    ONLINE       0     0     0
        replacing-4                             DEGRADED 8.83K    14     0
          b74d4b85-0436-4814-8515-b405d7e40e24  REMOVED      0     0     0
          181fbb56-496c-4540-a107-d35504e0d61b  ONLINE   8.85K    16     0  (resilvering)
        96d445ec-e8b2-42ce-8f1a-63ca963eead9    FAULTED  1.90K 10.2K     3  too many errors
        b4dbc538-a334-417c-9b6a-7265693bc710    REMOVED      0     0     0
        fd7f4ef7-237a-48c0-8a72-bebaf1cd3fb5    ONLINE       0     0     0
        83e62634-a0a5-4c2f-abb8-10b763833042    ONLINE       0     0     0
        ea3b911a-d478-4341-b4cd-4549385fcdde    ONLINE       0     0     0
        52c0cc19-1576-4186-b41f-8ee005e7ebaf    ONLINE       0     0     0
        ee9542ae-6266-4615-bcaa-6a0f06dafb87    ONLINE       0     0     0
        0b764a40-fd5d-4631-922d-625178717347    ONLINE       0     0     0
        e0df314c-cc1e-460e-9591-842264306d5b    ONLINE       0     0     0
        af112e1e-211c-4061-a0bf-4e7f943903ca    ONLINE       0     0     0
        c2c7b568-7eff-4f85-998b-eb4e1ac51897    ONLINE       0     0     0
        ebd9e063-ebb9-4bdb-9ac6-3302a748390a    ONLINE       0     0     0
    cache
      b7523e48-e07b-4fbb-94c5-c6de97a7d33c      ONLINE       0     0     0

You have a 17 disk Z2 which means you can withstand up to two drive failures before losing the pool.

b4dbc538-a334-417c-9b6a-7265693bc710    REMOVED      0     0     0

Has been removed (1)

96d445ec-e8b2-42ce-8f1a-63ca963eead9    FAULTED  1.90K 10.2K     3  too many errors

Has faulted with too many errors. (2)

b74d4b85-0436-4814-8515-b405d7e40e24  REMOVED      0     0     0

Has also been removed and is currently being replaced. (3)

Unless I am reading this output wrong (which I may well be doing as its hurting my head) then you have currently lost 3 disks in the RAID-Z2 which in my book is a loss of the pool.

Im not very familiar with RAIDZ expansion so this process could well be confusing the output so please correct me if Im wrong.