Help with degraded pool and potentially replacing drives

Burb · May 3, 2024, 6:15pm

Hi, in need of some tech support.

Running SCALE 24.04

It seems my zfs pool status is showing up as DEGRADED in the dashboard and is showing two disks of my data VDEVs as Faulted.

sda is showing 38 read errors and 1 checksum error, no write errors.
sdf is showing 33 read errors, no checksum or write errors.

Pool consists of 6x Toshiba N300 (10 TB) 3.5" drives and should still be under warranty, only had them for a year and a half.
Running RAIDZ2.

First of all, I need to know how bad it is right now? Haven’t had any issues accessing or writing to the pool and still using it daily to serve media through Emby with no issues.

Is it urgent that I replace the drives now to avoid losing the entire pool? Are they confirmed “bad” by just “a few” read errors? Any further manual disk checks I should run?

Also I need to know the process of replacing bad disks in SCALE. I assume it’s not as easy as shut down server, pull out bad, insert good, power back on. Require some further manual intervention through gui or terminal?

Also can I swap them out for larger disks as well over time? Say I start buying 18 TB drives one by one.
And after I replace all six I will get the full bump up in capacity?

Thanks in advance and let me know if I missed some important information!

kris · May 3, 2024, 7:30pm

First of all, welcome to the community!

In this case you are running a RAIDZ2, which means you can lose two drives without loosing data. However you now have 2 drives now showing errors. There is a chance that these are some issue related to the controller or cables, but assuming you rule that out you will want to replace them as soon as possible. You are at risk of loosing the pool during the re-silver operation (which takes time and is rough on the remaining disks). If you have a third start to flake out during that you will be in serious trouble. Make sure your backups are up to date!

Replacing a disk is pretty straightforward. You do it physically and through the UI. Good details on how to do this in our docs here:

To answer your question about larger drives, after you swap out all 6 with the 18TB’s, yes you will be able to bump up and expose the larger capacity.

Good luck!

joeschmuck · May 3, 2024, 8:00pm

@Burb I recommend you run a Scrub, you can do it via the GUI or CLI zpool scrub poolname. “poolname” is the name of the pool. If you are unsure, type zpool status and all your pool data should be listed. Then when the scrub has completed, examine the results using zpool -v poolname to see if there are any files that are listed as corrupt.

If no files are listed as corrupt, and you only have a few (1 to ~200, 10,000 is a lot!) check sum or write errors, “NO specific file errors”, then run the command zpool clear poolname

The next step is to run a SMART Long test on the two drives in question (10TB drives will take a long time, don’t rush it and the system must remain powered on, and after those are complete, check to ensure there are no problems. Feel free to post the output of smartctl -x /dev/sda for example and someone will tell you if it is a problem or not. Once that is done, run a SMART Long test on the rest of your drives over the next few days, just to give you that warm feeling.

I recommend a Daily SMART Short test and a weekly SMART Long test.

And yes, you can replace your drives with larger drives, you just can’t replace using a smaller drive. You will not see an increase in capacity until ALL the drives have been replaced.

Burb · May 3, 2024, 9:54pm

Is this the correct command? I ran a scrub earlier during the week so I tried running this right away without another scrub first - but I just get the zpool help text, seems the syntax is wrong?
The -v flag looks like it’s used for the upgrade command, so should I run zpool upgrade -v poolname ?

Running another scrub now, I’ll let it run over night. Then I’ll run the SMART test during the weekend.

Here’s the output of zpool status:

  pool: Storage01
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
	repaired.
  scan: scrub repaired 1M in 19:27:52 with 0 errors on Mon Apr 29 07:58:20 2024
config:

	NAME                                      STATE     READ WRITE CKSUM
	Storage01                                 DEGRADED     0     0     0
	  raidz2-0                                DEGRADED     0     0     0
	    3f6c9d7f-a4ce-488e-87d7-c26237820c8c  FAULTED     38     0     1  too many errors
	    1b63c2e1-d8f6-477a-a431-c3cca3400d4f  ONLINE       0     0     0
	    94f1a669-a748-4003-b31d-b911bdee68ca  ONLINE       0     0     0
	    9ea6bbb2-7d1b-4526-9ef7-c9afbf28651e  ONLINE       0     0     0
	    eb3ecd61-ac1d-48ad-bbcd-699a7e488f85  ONLINE       0     0     0
	    b8778a53-6172-4b4b-ab1a-f0b0c02fa6f2  FAULTED     33     0     0  too many errors

errors: No known data errors

  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
	The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:01:04 with 0 errors on Mon Apr 29 03:46:06 2024
config:

	NAME        STATE     READ WRITE CKSUM
	boot-pool   ONLINE       0     0     0
	  sdc3      ONLINE       0     0     0

errors: No known data errors

joeschmuck · May 3, 2024, 11:05pm

Sorry about the syntax, I left out the ‘status’ as you ran apparently.

Now you want to enter:
zpool scrub Storage01

This may take a while, the more data you have, the longer a scrub takes. You can check the status using zpool status -v Storage01 The fact that the previous scrub repaired the data is promising. When the line that starts with “scan:” says the scrub completed and IF it did not have to repair more data, then you are good to run the next command zpool clear Storage01 and you can check the status again, all should be good again.

joeschmuck · May 3, 2024, 11:12pm

Not sure if it look correct on someone else’s screen but the first 'zpool scrub Storage01 is a number zero and number 1. Very odd.

And if you get this far, run those SMART Long tests, make sure they say self-test Complete, not 90% complete or aborted.

If after doing all this and the problem returns again, note which drives were affected, see if it’s the same drives. The data corrupt you had may have been from a crash (if you had one) or something else like the upgrade to SCALE 24.04.0 but hopefully this will not occur again. If the same drives are affected, consider replacing them, but I would not do that just yet.

Stux · May 3, 2024, 11:45pm

It’s also possible to get these types of errors through physical factors, power glitches, bumping the server! Etc.

You’re trying to determine if you had a temporary glitch, or if one or both of your drives is failing.

Either way. ZFS is doing what it’s supposed to do, so scrub, clear and test the drives and if problems persist then replace the cause

Burb · May 6, 2024, 10:48am

Long test still at 10%, not sure if its even still going (started it through the gui almost 48h ago) but, judging by this already and seeing “sector error” I should probably be looking at replacing the drive asap.

Stux · May 6, 2024, 12:44pm

Yes.

etorix · May 6, 2024, 1:17pm

“Good”, in the sense that you’re finding the issue.
Look at the output of smartctl -a /dev/sda (or -x). You may post it here if you want help to read it. And send it to Toshiba alonside your application for RMA.

And check also smartctl output for the other drives.

Burb · May 6, 2024, 2:56pm

Yeah I think the plan is to run the long smart on both the faulted drives, I’ll run it manually through tmux so I can keep an eye on it.
Print it out on paper and send the reports in with the RMA claim.

Now, just to double check and make it absolutely clear and to avoid data loss…
I’m running RAIDZ2. Two drives are bad, but haven’t completely failed yet. With RAIDZ2 then If two drives fail then I will lose the entire pool, right?

So, If I offline the drives and send both drives in for RMA at the same time, then I will lose all data, correct? Since then two drives are missing.

If that’s the case, I’m thinking of doing this scenario:
Purchase one new drive (preferrably larger capacity like 14 TB for future upgrade path)
Offline one of the HEALTHY drives, replace with brand new 14 TB disk.
Offline one of the FAULTED drives, replace with 10TB healthy disk (wiped) that was previously already in the pool.

At this point can I offline the last faulty drive and send in both for RMA (and leave the server offline until I get replacement drives?)

edit: not sure if the long smart test finished as it was stuck on 10% in the truenas gui despite running for almost 48 hours but here’s the output of smartctl -x

root@truenas[~]# smartctl -x /dev/sda
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.20-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba N300/MN NAS HDD
Device Model:     TOSHIBA HDWG11A
Serial Number:    52H0A0A9FATG
LU WWN Device Id: 5 000039 ba8d127a5
Firmware Version: 0603
User Capacity:    10,000,831,348,736 bytes [10.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon May  6 08:21:41 2024 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Disabled
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (  73) The previous self-test completed having
                                        a test element that failed and the test
                                        element that failed is not known.
Total time to complete Offline 
data collection:                (  120) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1013) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     PO-R--   100   001   050    Past 0
  2 Throughput_Performance  P-S---   100   100   050    -    0
  3 Spin_Up_Time            POS--K   100   100   001    -    5106
  4 Start_Stop_Count        -O--CK   100   100   000    -    33
  5 Reallocated_Sector_Ct   PO--CK   100   100   050    -    0
  7 Seek_Error_Rate         PO-R--   100   097   050    -    0
  8 Seek_Time_Performance   P-S---   100   100   050    -    0
  9 Power_On_Hours          -O--CK   078   078   000    -    9171
 10 Spin_Retry_Count        PO--CK   100   100   030    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    33
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    2
192 Power-Off_Retract_Count -O--CK   100   100   000    -    14
193 Load_Cycle_Count        -O--CK   100   100   000    -    34
194 Temperature_Celsius     -O---K   100   100   000    -    65 (Min/Max 18/71)
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O--CK   100   100   000    -    8
198 Offline_Uncorrectable   ----CK   100   100   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
220 Disk_Shift              -O----   100   100   000    -    1310744
222 Loaded_Hours            -O--CK   078   078   000    -    9171
223 Load_Retry_Count        -O--CK   100   100   000    -    0
224 Load_Friction           -O---K   100   100   000    -    0
226 Load-in_Time            -OS--K   100   100   000    -    534
240 Head_Flying_Hours       P-----   100   100   001    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O     51  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0c       GPL     R/O    513  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x24       GPL     R/O  49152  Current Device Internal Status Data log
0x25       GPL     R/O  49152  Saved Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
Device Error Count: 1
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 [0] occurred at disk power-on lifetime: 8989 hours (374 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 43 00 b0 00 00 04 a7 1b a0 40 00  Error: WP at LBA = 0x04a71ba0 = 78060448

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 80 00 58 00 00 dd c5 e5 f0 40 00 17d+10:48:08.513  WRITE FPDMA QUEUED
  61 02 68 00 78 00 00 dd c5 e3 88 40 00 17d+10:48:08.512  WRITE FPDMA QUEUED
  61 05 98 00 70 00 00 dd c5 dd f0 40 00 17d+10:48:08.510  WRITE FPDMA QUEUED
  61 00 40 00 38 00 00 e0 2f af b0 40 00 17d+10:48:08.510  WRITE FPDMA QUEUED
  61 00 80 00 30 00 00 e0 2f af 30 40 00 17d+10:48:08.510  WRITE FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: unknown failure    90%      9171         0
# 2  Extended offline    Completed: unknown failure    90%      9171         0
# 3  Extended offline    Completed: unknown failure    90%      9171         0
# 4  Extended offline    Completed: unknown failure    90%      9168         0
# 5  Extended offline    Aborted by host               90%      9168         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       1 (0x0001)
Device State:                        Active (0)
Current Temperature:                    65 Celsius
Power Cycle Min/Max Temperature:     61/69 Celsius
Lifetime    Min/Max Temperature:     18/71 Celsius
Specified Max Operating Temperature:    55 Celsius
Under/Over Temperature Limit Count:   0/36052

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      5/55 Celsius
Min/Max Temperature Limit:           -40/70 Celsius
Temperature History Size (Index):    478 (272)

Index    Estimated Time   Temperature Celsius
 273    2024-05-06 00:24    66  ***************************************+
 274    2024-05-06 00:25    65  ***************************************+
 275    2024-05-06 00:26    66  ***************************************+
 276    2024-05-06 00:27    65  ***************************************+
 277    2024-05-06 00:28    66  ***************************************+
 278    2024-05-06 00:29    65  ***************************************+
 279    2024-05-06 00:30    66  ***************************************+
 280    2024-05-06 00:31    66  ***************************************+
 281    2024-05-06 00:32    66  ***************************************+
 282    2024-05-06 00:33    65  ***************************************+
 283    2024-05-06 00:34    66  ***************************************+
 284    2024-05-06 00:35    65  ***************************************+
 285    2024-05-06 00:36    66  ***************************************+
 286    2024-05-06 00:37    65  ***************************************+
 287    2024-05-06 00:38    65  ***************************************+
 288    2024-05-06 00:39    65  ***************************************+
 289    2024-05-06 00:40    66  ***************************************+
 290    2024-05-06 00:41    65  ***************************************+
 291    2024-05-06 00:42    65  ***************************************+
 292    2024-05-06 00:43    65  ***************************************+
 293    2024-05-06 00:44    66  ***************************************+
 294    2024-05-06 00:45    66  ***************************************+
 295    2024-05-06 00:46    66  ***************************************+
 296    2024-05-06 00:47    65  ***************************************+
 297    2024-05-06 00:48    65  ***************************************+
 298    2024-05-06 00:49    66  ***************************************+
 299    2024-05-06 00:50    65  ***************************************+
 300    2024-05-06 00:51    65  ***************************************+
 301    2024-05-06 00:52    66  ***************************************+
 302    2024-05-06 00:53    65  ***************************************+
 303    2024-05-06 00:54    65  ***************************************+
 304    2024-05-06 00:55    66  ***************************************+
 305    2024-05-06 00:56    66  ***************************************+
 306    2024-05-06 00:57    65  ***************************************+
 307    2024-05-06 00:58    66  ***************************************+
 308    2024-05-06 00:59    66  ***************************************+
 309    2024-05-06 01:00    65  ***************************************+
 ...    ..(  3 skipped).    ..  ***************************************+
 313    2024-05-06 01:04    65  ***************************************+
 314    2024-05-06 01:05    66  ***************************************+
 315    2024-05-06 01:06    65  ***************************************+
 ...    ..(  5 skipped).    ..  ***************************************+
 321    2024-05-06 01:12    65  ***************************************+
 322    2024-05-06 01:13    66  ***************************************+
 323    2024-05-06 01:14    65  ***************************************+
 324    2024-05-06 01:15    66  ***************************************+
 325    2024-05-06 01:16    66  ***************************************+
 326    2024-05-06 01:17    66  ***************************************+
 327    2024-05-06 01:18    65  ***************************************+
 328    2024-05-06 01:19    66  ***************************************+
 329    2024-05-06 01:20    65  ***************************************+
 ...    ..(  3 skipped).    ..  ***************************************+
 333    2024-05-06 01:24    65  ***************************************+
 334    2024-05-06 01:25    66  ***************************************+
 335    2024-05-06 01:26    66  ***************************************+
 336    2024-05-06 01:27    65  ***************************************+
 337    2024-05-06 01:28    66  ***************************************+
 338    2024-05-06 01:29    65  ***************************************+
 339    2024-05-06 01:30    66  ***************************************+
 340    2024-05-06 01:31    66  ***************************************+
 341    2024-05-06 01:32    65  ***************************************+
 342    2024-05-06 01:33    65  ***************************************+
 343    2024-05-06 01:34    65  ***************************************+
 344    2024-05-06 01:35    66  ***************************************+
 345    2024-05-06 01:36    66  ***************************************+
 346    2024-05-06 01:37    65  ***************************************+
 347    2024-05-06 01:38    65  ***************************************+
 348    2024-05-06 01:39    66  ***************************************+
 349    2024-05-06 01:40    66  ***************************************+
 350    2024-05-06 01:41    66  ***************************************+
 351    2024-05-06 01:42    65  ***************************************+
 352    2024-05-06 01:43    66  ***************************************+
 353    2024-05-06 01:44    65  ***************************************+
 354    2024-05-06 01:45    65  ***************************************+
 355    2024-05-06 01:46    65  ***************************************+
 356    2024-05-06 01:47    66  ***************************************+
 ...    ..(  2 skipped).    ..  ***************************************+
 359    2024-05-06 01:50    66  ***************************************+
 360    2024-05-06 01:51    65  ***************************************+
 361    2024-05-06 01:52    65  ***************************************+
 362    2024-05-06 01:53    66  ***************************************+
 363    2024-05-06 01:54    65  ***************************************+
 364    2024-05-06 01:55    65  ***************************************+
 365    2024-05-06 01:56    65  ***************************************+
 366    2024-05-06 01:57    66  ***************************************+
 367    2024-05-06 01:58    65  ***************************************+
 ...    ..( 14 skipped).    ..  ***************************************+
 382    2024-05-06 02:13    65  ***************************************+
 383    2024-05-06 02:14    66  ***************************************+
 384    2024-05-06 02:15    65  ***************************************+
 385    2024-05-06 02:16    65  ***************************************+
 386    2024-05-06 02:17    65  ***************************************+
 387    2024-05-06 02:18    66  ***************************************+
 388    2024-05-06 02:19    65  ***************************************+
 ...    ..(  4 skipped).    ..  ***************************************+
 393    2024-05-06 02:24    65  ***************************************+
 394    2024-05-06 02:25    66  ***************************************+
 395    2024-05-06 02:26    65  ***************************************+
 ...    ..(  2 skipped).    ..  ***************************************+
 398    2024-05-06 02:29    65  ***************************************+
 399    2024-05-06 02:30    66  ***************************************+
 400    2024-05-06 02:31    65  ***************************************+
 401    2024-05-06 02:32    66  ***************************************+
 402    2024-05-06 02:33    65  ***************************************+
 403    2024-05-06 02:34    66  ***************************************+
 404    2024-05-06 02:35    65  ***************************************+
 405    2024-05-06 02:36    65  ***************************************+
 406    2024-05-06 02:37    66  ***************************************+
 407    2024-05-06 02:38    65  ***************************************+
 408    2024-05-06 02:39    65  ***************************************+
 409    2024-05-06 02:40    66  ***************************************+
 410    2024-05-06 02:41    65  ***************************************+
 ...    ..(  4 skipped).    ..  ***************************************+
 415    2024-05-06 02:46    65  ***************************************+
 416    2024-05-06 02:47    66  ***************************************+
 417    2024-05-06 02:48    65  ***************************************+
 418    2024-05-06 02:49    65  ***************************************+
 419    2024-05-06 02:50    66  ***************************************+
 420    2024-05-06 02:51    65  ***************************************+
 ...    ..(  6 skipped).    ..  ***************************************+
 427    2024-05-06 02:58    65  ***************************************+
 428    2024-05-06 02:59    66  ***************************************+
 429    2024-05-06 03:00    66  ***************************************+
 430    2024-05-06 03:01    66  ***************************************+
 431    2024-05-06 03:02    65  ***************************************+
 432    2024-05-06 03:03    65  ***************************************+
 433    2024-05-06 03:04    66  ***************************************+
 434    2024-05-06 03:05    65  ***************************************+
 435    2024-05-06 03:06    66  ***************************************+
 436    2024-05-06 03:07    65  ***************************************+
 437    2024-05-06 03:08    65  ***************************************+
 438    2024-05-06 03:09    65  ***************************************+
 439    2024-05-06 03:10    66  ***************************************+
 440    2024-05-06 03:11    65  ***************************************+
 441    2024-05-06 03:12    65  ***************************************+
 442    2024-05-06 03:13    65  ***************************************+
 443    2024-05-06 03:14    66  ***************************************+
 444    2024-05-06 03:15    65  ***************************************+
 445    2024-05-06 03:16    65  ***************************************+
 446    2024-05-06 03:17    66  ***************************************+
 447    2024-05-06 03:18    66  ***************************************+
 448    2024-05-06 03:19    65  ***************************************+
 ...    ..(  5 skipped).    ..  ***************************************+
 454    2024-05-06 03:25    65  ***************************************+
 455    2024-05-06 03:26    66  ***************************************+
 456    2024-05-06 03:27    65  ***************************************+
 457    2024-05-06 03:28    66  ***************************************+
 458    2024-05-06 03:29    65  ***************************************+
 ...    ..(  3 skipped).    ..  ***************************************+
 462    2024-05-06 03:33    65  ***************************************+
 463    2024-05-06 03:34    66  ***************************************+
 464    2024-05-06 03:35    65  ***************************************+
 465    2024-05-06 03:36    65  ***************************************+
 466    2024-05-06 03:37    65  ***************************************+
 467    2024-05-06 03:38    66  ***************************************+
 468    2024-05-06 03:39    65  ***************************************+
 469    2024-05-06 03:40    65  ***************************************+
 470    2024-05-06 03:41    66  ***************************************+
 471    2024-05-06 03:42    65  ***************************************+
 472    2024-05-06 03:43    65  ***************************************+
 473    2024-05-06 03:44    65  ***************************************+
 474    2024-05-06 03:45    66  ***************************************+
 475    2024-05-06 03:46    65  ***************************************+
 ...    ..(  3 skipped).    ..  ***************************************+
   1    2024-05-06 03:50    65  ***************************************+
   2    2024-05-06 03:51    66  ***************************************+
   3    2024-05-06 03:52    65  ***************************************+
 ...    ..( 11 skipped).    ..  ***************************************+
  15    2024-05-06 04:04    65  ***************************************+
  16    2024-05-06 04:05    66  ***************************************+
  17    2024-05-06 04:06    65  ***************************************+
 ...    ..( 14 skipped).    ..  ***************************************+
  32    2024-05-06 04:21    65  ***************************************+
  33    2024-05-06 04:22    66  ***************************************+
  34    2024-05-06 04:23    66  ***************************************+
  35    2024-05-06 04:24    65  ***************************************+
 ...    ..(  3 skipped).    ..  ***************************************+
  39    2024-05-06 04:28    65  ***************************************+
  40    2024-05-06 04:29    66  ***************************************+
  41    2024-05-06 04:30    65  ***************************************+
 ...    ..(  6 skipped).    ..  ***************************************+
  48    2024-05-06 04:37    65  ***************************************+
  49    2024-05-06 04:38    66  ***************************************+
  50    2024-05-06 04:39    65  ***************************************+
  51    2024-05-06 04:40    65  ***************************************+
  52    2024-05-06 04:41    65  ***************************************+
  53    2024-05-06 04:42    66  ***************************************+
  54    2024-05-06 04:43    65  ***************************************+
 ...    ..(  6 skipped).    ..  ***************************************+
  61    2024-05-06 04:50    65  ***************************************+
  62    2024-05-06 04:51    66  ***************************************+
  63    2024-05-06 04:52    65  ***************************************+
  64    2024-05-06 04:53    65  ***************************************+
  65    2024-05-06 04:54    65  ***************************************+
  66    2024-05-06 04:55    66  ***************************************+
  67    2024-05-06 04:56    65  ***************************************+
  68    2024-05-06 04:57    65  ***************************************+
  69    2024-05-06 04:58    65  ***************************************+
  70    2024-05-06 04:59    66  ***************************************+
  71    2024-05-06 05:00    65  ***************************************+
  72    2024-05-06 05:01    65  ***************************************+
  73    2024-05-06 05:02    65  ***************************************+
  74    2024-05-06 05:03    66  ***************************************+
  75    2024-05-06 05:04    66  ***************************************+
  76    2024-05-06 05:05    65  ***************************************+
  77    2024-05-06 05:06    66  ***************************************+
  78    2024-05-06 05:07    66  ***************************************+
  79    2024-05-06 05:08    65  ***************************************+
  80    2024-05-06 05:09    65  ***************************************+
  81    2024-05-06 05:10    65  ***************************************+
  82    2024-05-06 05:11    66  ***************************************+
  83    2024-05-06 05:12    66  ***************************************+
  84    2024-05-06 05:13    65  ***************************************+
  85    2024-05-06 05:14    66  ***************************************+
  86    2024-05-06 05:15    65  ***************************************+
 ...    ..( 13 skipped).    ..  ***************************************+
 100    2024-05-06 05:29    65  ***************************************+
 101    2024-05-06 05:30    66  ***************************************+
 102    2024-05-06 05:31    65  ***************************************+
 ...    ..(  8 skipped).    ..  ***************************************+
 111    2024-05-06 05:40    65  ***************************************+
 112    2024-05-06 05:41    66  ***************************************+
 113    2024-05-06 05:42    65  ***************************************+
 ...    ..( 14 skipped).    ..  ***************************************+
 128    2024-05-06 05:57    65  ***************************************+
 129    2024-05-06 05:58    66  ***************************************+
 130    2024-05-06 05:59    65  ***************************************+
 ...    ..( 11 skipped).    ..  ***************************************+
 142    2024-05-06 06:11    65  ***************************************+
 143    2024-05-06 06:12    66  ***************************************+
 144    2024-05-06 06:13    65  ***************************************+
 145    2024-05-06 06:14    65  ***************************************+
 146    2024-05-06 06:15    66  ***************************************+
 147    2024-05-06 06:16    65  ***************************************+
 148    2024-05-06 06:17    65  ***************************************+
 149    2024-05-06 06:18    66  ***************************************+
 150    2024-05-06 06:19    65  ***************************************+
 ...    ..(  4 skipped).    ..  ***************************************+
 155    2024-05-06 06:24    65  ***************************************+
 156    2024-05-06 06:25    66  ***************************************+
 157    2024-05-06 06:26    65  ***************************************+
 ...    ..(  7 skipped).    ..  ***************************************+
 165    2024-05-06 06:34    65  ***************************************+
 166    2024-05-06 06:35    66  ***************************************+
 167    2024-05-06 06:36    65  ***************************************+
 ...    ..(  4 skipped).    ..  ***************************************+
 172    2024-05-06 06:41    65  ***************************************+
 173    2024-05-06 06:42    66  ***************************************+
 174    2024-05-06 06:43    66  ***************************************+
 175    2024-05-06 06:44    65  ***************************************+
 176    2024-05-06 06:45    65  ***************************************+
 177    2024-05-06 06:46    65  ***************************************+
 178    2024-05-06 06:47    66  ***************************************+
 179    2024-05-06 06:48    65  ***************************************+
 180    2024-05-06 06:49    65  ***************************************+
 181    2024-05-06 06:50    65  ***************************************+
 182    2024-05-06 06:51    66  ***************************************+
 183    2024-05-06 06:52    65  ***************************************+
 184    2024-05-06 06:53    65  ***************************************+
 185    2024-05-06 06:54    65  ***************************************+
 186    2024-05-06 06:55    66  ***************************************+
 187    2024-05-06 06:56    65  ***************************************+
 188    2024-05-06 06:57    66  ***************************************+
 189    2024-05-06 06:58    65  ***************************************+
 190    2024-05-06 06:59    66  ***************************************+
 191    2024-05-06 07:00    65  ***************************************+
 192    2024-05-06 07:01    65  ***************************************+
 193    2024-05-06 07:02    65  ***************************************+
 194    2024-05-06 07:03    66  ***************************************+
 195    2024-05-06 07:04    65  ***************************************+
 ...    ..(  6 skipped).    ..  ***************************************+
 202    2024-05-06 07:11    65  ***************************************+
 203    2024-05-06 07:12    66  ***************************************+
 204    2024-05-06 07:13    65  ***************************************+
 205    2024-05-06 07:14    65  ***************************************+
 206    2024-05-06 07:15    65  ***************************************+
 207    2024-05-06 07:16    66  ***************************************+
 208    2024-05-06 07:17    65  ***************************************+
 ...    ..(  8 skipped).    ..  ***************************************+
 217    2024-05-06 07:26    65  ***************************************+
 218    2024-05-06 07:27    66  ***************************************+
 219    2024-05-06 07:28    65  ***************************************+
 220    2024-05-06 07:29    65  ***************************************+
 221    2024-05-06 07:30    65  ***************************************+
 222    2024-05-06 07:31    66  ***************************************+
 223    2024-05-06 07:32    65  ***************************************+
 ...    ..(  7 skipped).    ..  ***************************************+
 231    2024-05-06 07:40    65  ***************************************+
 232    2024-05-06 07:41    66  ***************************************+
 233    2024-05-06 07:42    65  ***************************************+
 234    2024-05-06 07:43    65  ***************************************+
 235    2024-05-06 07:44    66  ***************************************+
 236    2024-05-06 07:45    65  ***************************************+
 ...    ..( 35 skipped).    ..  ***************************************+
 272    2024-05-06 08:21    65  ***************************************+

SCT Error Recovery Control:
           Read: Disabled
          Write: Disabled

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 3) ==
0x01  0x008  4              33  ---  Lifetime Power-On Resets
0x01  0x010  4            9171  ---  Power-on Hours
0x01  0x018  6     28605574235  ---  Logical Sectors Written
0x01  0x020  6       420876977  ---  Number of Write Commands
0x01  0x028  6     62821831226  ---  Logical Sectors Read
0x01  0x030  6       156909979  ---  Number of Read Commands
0x01  0x038  6     33015600000  ---  Date and Time TimeStamp
0x02  =====  =               =  ===  == Free-Fall Statistics (rev 1) ==
0x02  0x010  4               2  ---  Overlimit Shock Events
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4             820  ---  Spindle Motor Power-on Hours
0x03  0x010  4             819  ---  Head Flying Hours
0x03  0x018  4              34  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4          273945  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               8  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4              14  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               1  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4              27  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              65  ---  Current Temperature
0x05  0x010  1              65  N--  Average Short Term Temperature
0x05  0x018  1              62  N--  Average Long Term Temperature
0x05  0x020  1              71  ---  Highest Temperature
0x05  0x028  1              18  ---  Lowest Temperature
0x05  0x030  1              67  N--  Highest Average Short Term Temperature
0x05  0x038  1              51  N--  Lowest Average Short Term Temperature
0x05  0x040  1              64  N--  Highest Average Long Term Temperature
0x05  0x048  1              55  N--  Lowest Average Long Term Temperature
0x05  0x050  4          516066  ---  Time in Over-Temperature
0x05  0x058  1              55  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               5  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4             129  ---  Number of Hardware Resets
0x06  0x010  4              15  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0x07  =====  =               =  ===  == Solid State Device Statistics (rev 1) ==
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
Index                LBA    Hours
    0           78057184     9154
    1           78057185     9154
    2           78057186     9154
    3           78057187     9154
    4           78057188     9154
    5           78057189     9154
    6           78057190     9154
    7           78057191     9154

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  4            0  Command failed due to ICRC error
0x0002  4            0  R_ERR response for data FIS
0x0003  4            0  R_ERR response for device-to-host data FIS
0x0004  4            0  R_ERR response for host-to-device data FIS
0x0005  4            0  R_ERR response for non-data FIS
0x0006  4            0  R_ERR response for device-to-host non-data FIS
0x0007  4            0  R_ERR response for host-to-device non-data FIS
0x0008  4            0  Device-to-host non-data FIS retries
0x0009  4           28  Transition from drive PhyRdy to drive PhyNRdy
0x000a  4           23  Device-to-host register FISes sent due to a COMRESET
0x000b  4            0  CRC errors within host-to-device FIS
0x000d  4            0  Non-CRC errors within host-to-device FIS
0x000f  4            0  R_ERR response for host-to-device data FIS, CRC
0x0010  4            0  R_ERR response for host-to-device data FIS, non-CRC
0x0012  4            0  R_ERR response for host-to-device non-data FIS, CRC
0x0013  4            0  R_ERR response for host-to-device non-data FIS, non-CRC

root@truenas[~]#

etorix · May 6, 2024, 4:32pm

There’s no need for tmux. SMART test are run by the drives themselves: Start them (smartctl -t long /dev/sdX), note when they will finish and wait to ask for the result (smartctl -a). Test all drives.

A raidz2 pool can loose two drives… but then any further incident will result in data loss.
As long as the drives are not totally failed, they can still contribute some resiliency.

If you have a free SATA port, plug in the new drive, go to Storage>Pool>Status and replace a drive. When it’s done, physically remove the old drive. If there’s no free port, then offline the drive, physically remove the drive and put the replacement in its place and replace the missing drive from the GUI.
Repeat as needed.

Ideally, Toshiba would accept to send replacement drives in advance. If not then, proceed to RMA and replacement one drive at a time and/or buy an extra drive to initiate the replacement(s)—you’ll end up with an extra drive, which can be kept as cold spare.

Burb · May 6, 2024, 5:08pm

Unfortunately, just checked on their (EU) site and it claims:

2.If you bought the Toshiba Hard Disk Drive from a shop, then guarantee claims can only be made with the shop which sold you the drive. The contents of the guarantee comply with the legal regulations in the country, in which the product was purchased.

Toshiba does NOT provide a direct repair service for anyone other than our OEM customers. It is not possible to make claims directly to Toshiba in cases of possible defects. Therefore you should NEVER send a defective drive to Toshiba representatives, because they cannot process this type of transaction!

So most likely I will have to pull the drive(s) out, return it to the store where I got it from (physical store here in town, did not buy online) they make the claim with toshiba, I wait 3-4 weeks to get a replacement.

Inconvenient, but I guess no way around it.

Alright, well I know which way to go from here. Thanks for all the help!

Stux · May 6, 2024, 10:08pm

If a third drive fails. This is a simplification. It depends on if it’s a complete failure vs collocated block failures, which just lead to missing files.

In this scenario, I normally buy a replacement AND RMA the failure. When I get the replacement drive from manufacturer (may be refurb) then I burn it in and make it a spare, or replace the new one and make that a spare.

Stux · May 6, 2024, 10:11pm

You risk replacing a healthy drive with a new faulty drive here.

neofusion · May 6, 2024, 10:38pm

You are cooking your drives. If anyone takes more than a passing look at your SMART output the RMA will be likely be denied.

Current Temperature:                    65 Celsius
Power Cycle Min/Max Temperature:     61/69 Celsius
Lifetime    Min/Max Temperature:     18/71 Celsius
Specified Max Operating Temperature:    55 Celsius
Under/Over Temperature Limit Count:   0/36052

Burb · May 7, 2024, 3:08pm

Fair enough, I should definitely look into the cooling situation going forward.
Though, the specified max operating temperature here seems to be incorrect compared to what toshiba’s datasheet says.

Temperature (operating/ non-operating):
0 to 65 °C
-40 to 70 °C