HELP- Boot pool degraded - disk in removed state

TrueNAS General
1

Hi

My boot pool ran a scrub last night and one out of the two became degraded.
It is now sitting in a “removed” state.
The disks are a pair of - Silcon Power SSD 128GB which i only purchased last year August.

Here is a snippet of the logs:

Jul 17 00:14:44 S-STORE01 ahcich1: Timeout on slot 23 port 0
Jul 17 00:14:44 S-STORE01 ahcich1: is 00000000 cs 00000000 ss 00800000 rs 00800000 tfd 40 serr 00080000 cmd 0004d717
Jul 17 00:14:44 S-STORE01 (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 04 34 8a 7f 40 0c 00 00 00 00 00
Jul 17 00:14:44 S-STORE01 (ada1:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:14:44 S-STORE01 (ada1:ahcich1:0:0:0): Retrying command, 3 more tries remain
Jul 17 00:15:47 S-STORE01 ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Jul 17 00:15:47 S-STORE01 ahcich1: Timeout on slot 24 port 0
Jul 17 00:15:47 S-STORE01 ahcich1: is 00000000 cs 01000000 ss 00000000 rs 01000000 tfd 80 serr 00000000 cmd 0004d817
Jul 17 00:15:47 S-STORE01 (aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jul 17 00:15:47 S-STORE01 (aprobe0:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:15:47 S-STORE01 (aprobe0:ahcich1:0:0:0): Error 5, Retry was blocked
Jul 17 00:16:50 S-STORE01 ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Jul 17 00:16:50 S-STORE01 ahcich1: Timeout on slot 25 port 0
Jul 17 00:16:50 S-STORE01 ahcich1: is 00000000 cs 02000000 ss 00000000 rs 02000000 tfd 80 serr 00000000 cmd 0004d917
Jul 17 00:16:50 S-STORE01 (aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jul 17 00:16:50 S-STORE01 (aprobe0:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:16:50 S-STORE01 (aprobe0:ahcich1:0:0:0): Retrying command, 0 more tries remain
Jul 17 00:17:53 S-STORE01 ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Jul 17 00:17:53 S-STORE01 ahcich1: Timeout on slot 26 port 0
Jul 17 00:17:53 S-STORE01 ahcich1: is 00000000 cs 04000000 ss 00000000 rs 04000000 tfd 80 serr 00000000 cmd 0004da17
Jul 17 00:17:53 S-STORE01 (aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jul 17 00:17:53 S-STORE01 (aprobe0:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:17:53 S-STORE01 (aprobe0:ahcich1:0:0:0): Error 5, Retries exhausted
Jul 17 00:17:53 S-STORE01 ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
Jul 17 00:17:53 S-STORE01 ada1: <SPCC Solid State Disk V0823A0> s/n AA230202S3128003036 detached
Jul 17 00:18:56 S-STORE01 ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Jul 17 00:18:56 S-STORE01 ahcich1: Timeout on slot 27 port 0
Jul 17 00:18:56 S-STORE01 ahcich1: is 00000000 cs 08000000 ss 00000000 rs 08000000 tfd 80 serr 00000000 cmd 0004db17
Jul 17 00:18:56 S-STORE01 (aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jul 17 00:18:56 S-STORE01 (aprobe0:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:18:56 S-STORE01 (aprobe0:ahcich1:0:0:0): Error 5, Retry was blocked
Jul 17 00:19:44 S-STORE01 ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Jul 17 00:19:44 S-STORE01 ahcich1: Poll timeout on slot 29 port 0
Jul 17 00:19:44 S-STORE01 ahcich1: is 00000000 cs 20000000 ss 00000000 rs 20000000 tfd 80 serr 00000000 cmd 0004dd17
Jul 17 00:19:44 S-STORE01 (aprobe0:ahcich1:0:0:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Jul 17 00:19:44 S-STORE01 (aprobe0:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:19:44 S-STORE01 (aprobe0:ahcich1:0:0:0): Error 5, Retries exhausted
Jul 17 00:20:44 S-STORE01 ahcich1: Timeout on slot 30 port 0
Jul 17 00:20:44 S-STORE01 ahcich1: is 00000000 cs 40000000 ss 00000000 rs 40000000 tfd 80 serr 00000000 cmd 0004de17
Jul 17 00:20:44 S-STORE01 (ada1:ahcich1:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00
Jul 17 00:20:44 S-STORE01 (ada1:ahcich1:0:0:0): CAM status: Unconditionally Re-queue Request
Jul 17 00:20:44 S-STORE01 (ada1:ahcich1:0:0:0): Error 5, Periph was invalidated
Jul 17 00:21:32 S-STORE01 ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Jul 17 00:21:32 S-STORE01 ahcich1: Poll timeout on slot 0 port 0
Jul 17 00:21:32 S-STORE01 ahcich1: is 00000000 cs 00000001 ss 00000000 rs 00000001 tfd 80 serr 00000000 cmd 0004c017
Jul 17 00:21:32 S-STORE01 (aprobe0:ahcich1:0:0:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Jul 17 00:21:32 S-STORE01 (aprobe0:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:21:32 S-STORE01 (aprobe0:ahcich1:0:0:0): Error 5, Retries exhausted
Jul 17 00:22:02 S-STORE01 ahcich1: Timeout on slot 1 port 0
Jul 17 00:22:02 S-STORE01 ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 80 serr 00000000 cmd 0004c117
Jul 17 00:22:02 S-STORE01 (ada1:ahcich1:0:0:0): SETFEATURES ENABLE WCACHE. ACB: ef 02 00 00 00 40 00 00 00 00 00 00
Jul 17 00:22:02 S-STORE01 (ada1:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:22:02 S-STORE01 (ada1:ahcich1:0:0:0): Error 5, Periph was invalidated
Jul 17 00:22:50 S-STORE01 ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Jul 17 00:22:50 S-STORE01 ahcich1: Poll timeout on slot 3 port 0
Jul 17 00:22:50 S-STORE01 ahcich1: is 00000000 cs 00000008 ss 00000000 rs 00000008 tfd 80 serr 00000000 cmd 0004c317
Jul 17 00:22:50 S-STORE01 (aprobe0:ahcich1:0:0:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Jul 17 00:22:50 S-STORE01 (aprobe0:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:22:50 S-STORE01 (aprobe0:ahcich1:0:0:0): Error 5, Retries exhausted
Jul 17 00:23:20 S-STORE01 ahcich1: Timeout on slot 4 port 0
Jul 17 00:23:20 S-STORE01 ahcich1: is 00000000 cs 00000010 ss 00000010 rs 00000010 tfd 80 serr 00000000 cmd 0004c417
Jul 17 00:23:20 S-STORE01 (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 04 34 8a 7f 40 0c 00 00 00 00 00
Jul 17 00:23:20 S-STORE01 (ada1:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:23:20 S-STORE01 (ada1:ahcich1:0:0:0): Error 5, Periph was invalidated
Jul 17 00:23:20 S-STORE01 (ada1:ahcich1:0:0:0): Periph destroyed
Jul 17 00:24:08 S-STORE01 ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Jul 17 00:24:08 S-STORE01 ahcich1: Poll timeout on slot 6 port 0
Jul 17 00:24:08 S-STORE01 ahcich1: is 00000000 cs 00000040 ss 00000000 rs 00000040 tfd 80 serr 00000000 cmd 0004c617
Jul 17 00:24:08 S-STORE01 (aprobe0:ahcich1:0:0:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Jul 17 00:24:08 S-STORE01 (aprobe0:ahcich1:0:0:0): CAM status: Command timeout
Jul 17 00:24:08 S-STORE01 (aprobe0:ahcich1:0:0:0): Error 5, Retries exhausted

I want to try changing the cable first to see if that is the cause.

The board supports hot swapping etc.

Can anyone advise what is the best method to do this.

  1. can it be done while it is on - I have two options when I click on the eclipses to either detach or replace.

or

  1. Power it off and swap the cable… Upon powering on the unit, will the system try to recognise the drive again?

If the drive is faulty, I was thinking to replace it with an Intel® SSD DC S3500 Series, 2.5" SATA 6Gb/s SSDSC2BB120G4 drive. I have seen these mentioned quite a lot here.

Any ideas what the best approach for this.

Thanks

2

So i changed the cables for both of the drives.

The errors cleared. But when I attempt to run a smart test on the drive it keeps failing. Any ideas?

Below is an output of the drives smart test

Here is an output of the drive a

=== START OF INFORMATION SECTION ===
Device Model:     SPCC Solid State Disk
Serial Number:    
Firmware Version: V0823A0
User Capacity:    128,035,676,160 bytes [128 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      M.2
TRIM Command:     Available
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Jul 17 16:42:48 2024 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x03) Offline data collection activity
                                        is in progress.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (  41) The self-test routine was interrupted
                                        by the host with a hard or soft reset.
Total time to complete Offline
data collection:                (  120) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0002) Does not save SMART data before
                                        entering power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  10) minutes.
SCT capabilities:              (0x0001) SCT Status supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   100   100   050    Old_age   Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   050    Old_age   Always       -       7853
 12 Power_Cycle_Count       0x0032   100   100   050    Old_age   Always       -       71
160 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       0
161 Unknown_Attribute       0x0033   100   100   050    Pre-fail  Always       -       100
163 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       6
164 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       13486
165 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       76
166 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       11
167 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       26
168 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       5050
169 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       100
175 Program_Fail_Count_Chip 0x0032   100   100   050    Old_age   Always       -       0
176 Erase_Fail_Count_Chip   0x0032   100   100   050    Old_age   Always       -       0
177 Wear_Leveling_Count     0x0032   100   100   050    Old_age   Always       -       0
178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   050    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   050    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       30
194 Temperature_Celsius     0x0022   100   100   050    Old_age   Always       -       40
195 Hardware_ECC_Recovered  0x0032   100   100   050    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   050    Old_age   Always       -       0
232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       100
241 Total_LBAs_Written      0x0030   100   100   050    Old_age   Offline      -       20785
242 Total_LBAs_Read         0x0030   100   100   050    Old_age   Offline      -       7055
245 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       74068

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Interrupted (host reset)      90%      7849         -
# 2  Extended offline    Interrupted (host reset)      80%      7849         -
# 3  Short offline       Completed without error       00%      7129         -
# 4  Extended offline    Completed without error       00%      6463         -
# 5  Short offline       Completed without error       00%      6223         -
# 6  Short offline       Completed without error       00%      5503         -
# 7  Extended offline    Completed without error       00%      5272         -
# 8  Short offline       Completed without error       00%      5271         -
# 9  Extended offline    Interrupted (host reset)      80%      5271         -
#10  Extended offline    Interrupted (host reset)      90%      4999         -
#11  Short offline       Completed without error       00%      4759         -
#12  Short offline       Completed without error       00%      4064         -
#13  Extended offline    Completed without error       00%      3560         -
#14  Short offline       Completed without error       00%      3320         -
#15  Short offline       Completed without error       00%      2576         -
#16  Extended offline    Completed without error       00%      2096         -
#17  Short offline       Completed without error       00%      1856         -
#18  Short offline       Completed without error       00%      1393         -
#19  Short offline       Completed without error       00%      1225         -
#20  Short offline       Completed without error       00%      1084         -
#21  Short offline       Completed without error       00%       917         -
3

Replace the drive with the same size or larger. SSD can just fail.

If you don’t trust that brand, get two replacements, go through process of replacing the first bad SSD and let resilver. Go through process once again with second SSD replacement.

Another option is to back up config. Insert both new SSD drives and reinstall OS and config file.

4

I just realised these two SSD currently in my boot pool were 128GB and the Intel drives are 120GB.

I guess this is going to be a problem, right?

5

Yes, but if you get a current backup of your configuration, you can still do a complete reinstall with the 120Gb and 128Gb. You might have to play with the menu for mirror selecting the 120 then the 128. You might have to install on 120 then add second 128 as mirror later. Temporary single boot then making it a mirror pair