Hey guys, first post on the new forum since it moved from the old one
Just noticed my Truenas core server has been spitting errors since last night. I think its a failing drive but the other errors in the syslog are not reassuring…
da6 is a WD Red 3TB that has been powered on for a bit over 4years… Not that old, so not really an infantile mortality, and not excessively old either… However SMART seems to show 1 “Current_Pending_Sector”…
I’m gonna replace it by a new WD 4TB (burnt in) just to be on the safe side… What do you guys think?
zpool status
root@freenas:~/scripts # zpool status zpool
pool: zpool
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 820K in 05:54:11 with 0 errors on Tue Dec 10 07:54:14 2024
config:
NAME STATE READ WRITE CKSUM
zpool ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
gptid/4a751424-5a4a-11e5-82f2-0030487f11ba ONLINE 0 0 0
gptid/7231ce76-0fb8-11e4-9267-0030487f11ba ONLINE 0 0 0
gptid/74010031-0fb8-11e4-9267-0030487f11ba ONLINE 0 0 0
gptid/3010e8b6-1d80-11e7-ac2f-0025907ad3a1 ONLINE 0 0 0
gptid/e3595fbd-e2fb-11ea-a043-0025907ad3a1 ONLINE 35 0 0
gptid/7799b692-0fb8-11e4-9267-0030487f11ba ONLINE 0 0 0
gptid/f91c6b45-c377-11ec-88b9-0025907ad3a1 ONLINE 0 0 0
gptid/7ba4673f-0fb8-11e4-9267-0030487f11ba ONLINE 0 0 0
syslog
Dec 10 05:45:18 freenas mps0: Controller reported scsi ioc terminated tgt 9 SMID 1164 loginfo 31080000
Dec 10 05:45:18 freenas mps0: Controller reported scsi ioc terminated tgt 9 SMID 1705 loginfo 31080000
Dec 10 05:45:18 freenas mps0: Controller reported scsi ioc terminated tgt 9 SMID 1968 loginfo 31080000
Dec 10 05:45:18 freenas mps0: Controller reported scsi ioc terminated tgt 9 SMID 1686 loginfo 31080000
Dec 10 05:45:18 freenas mps0: Controller reported scsi ioc terminated tgt 9 SMID 1965 loginfo 31080000
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): READ(10). CDB: 28 00 10 bb ee 40 00 00 38 00
Dec 10 05:45:18 freenas mps0: Controller reported scsi ioc terminated tgt 9 SMID 1104 loginfo 31080000
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): CAM status: CCB request completed with an error
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): Retrying command, 3 more tries remain
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): READ(10). CDB: 28 00 10 bb e8 98 00 01 00 00
Dec 10 05:45:18 freenas mps0: Controller reported scsi ioc terminated tgt 9 SMID 1807 loginfo 31080000
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): CAM status: CCB request completed with an error
Dec 10 05:45:18 freenas mps0: Controller reported scsi ioc terminated tgt 9 SMID 959 loginfo 31080000
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): Retrying command, 3 more tries remain
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): READ(10). CDB: 28 00 10 bb e9 98 00 01 00 00
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): CAM status: CCB request completed with an error
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): Retrying command, 3 more tries remain
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): READ(10). CDB: 28 00 10 bb ea 98 00 01 00 00
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): CAM status: CCB request completed with an error
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): Retrying command, 3 more tries remain
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): READ(10). CDB: 28 00 10 bb eb 98 00 01 00 00
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): CAM status: CCB request completed with an error
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): Retrying command, 3 more tries remain
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): READ(10). CDB: 28 00 10 bb ec 98 00 01 00 00
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): CAM status: CCB request completed with an error
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): Retrying command, 3 more tries remain
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): READ(10). CDB: 28 00 10 bb ed 98 00 00 a8 00
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): CAM status: CCB request completed with an error
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): Retrying command, 3 more tries remain
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): READ(10). CDB: 28 00 10 bb ee 78 00 00 38 00
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): CAM status: CCB request completed with an error
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): Retrying command, 3 more tries remain
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): READ(10). CDB: 28 00 10 bb e7 98 00 01 00 00
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): CAM status: SCSI Status Error
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): SCSI status: Check Condition
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): Info: 0x10bbe800
Dec 10 05:45:18 freenas (da6:mps0:0:9:0): Error 5, Unretryable error
Dec 10 06:13:05 freenas 1 2024-12-10T06:13:05.059178-05:00 freenas.tuxdomain smartd 1388 - - Device: /dev/da6 [SAT], 1 Currently unreadable (pending) sectors
Dec 10 06:13:05 freenas 1 2024-12-10T06:13:05.399585-05:00 freenas.tuxdomain smartd 1388 - - Device: /dev/da6 [SAT], 1 Currently unreadable (pending) sectors
Dec 10 06:43:04 freenas 1 2024-12-10T06:43:04.135283-05:00 freenas.tuxdomain smartd 1388 - - Device: /dev/da6 [SAT], 1 Currently unreadable (pending) sectors
Dec 10 06:43:04 freenas 1 2024-12-10T06:43:04.261487-05:00 freenas.tuxdomain smartd 1388 - - Device: /dev/da6 [SAT], 1 Currently unreadable (pending) sectors
smartctl for da6
root@freenas:~/scripts # smartctl -a /dev/da6
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD30EFRX-68N32N0
Serial Number: WD-WCC7K3KL0CV0
LU WWN Device Id: 5 0014ee 2ba472f91
Firmware Version: 82.00A82
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Dec 10 19:20:10 2024 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (34080) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 362) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 161 158 021 Pre-fail Always - 6941
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 37
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
9 Power_On_Hours 0x0032 049 049 000 Old_age Always - 37247
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 37
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 33
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 26
194 Temperature_Celsius 0x0022 125 104 000 Old_age Always - 25
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 37239 -
# 2 Short offline Completed without error 00% 37215 -
# 3 Short offline Completed without error 00% 37191 -
# 4 Short offline Completed without error 00% 37167 -
# 5 Short offline Completed without error 00% 37143 -
# 6 Short offline Completed without error 00% 37119 -
# 7 Short offline Completed without error 00% 37095 -
# 8 Short offline Completed without error 00% 37071 -
# 9 Short offline Completed without error 00% 37047 -
#10 Short offline Completed without error 00% 37023 -
#11 Extended offline Completed without error 00% 37020 -
#12 Short offline Completed without error 00% 36999 -
#13 Short offline Completed without error 00% 36975 -
#14 Short offline Completed without error 00% 36951 -
#15 Short offline Completed without error 00% 36927 -
#16 Short offline Completed without error 00% 36903 -
#17 Short offline Completed without error 00% 36879 -
#18 Short offline Completed without error 00% 36855 -
#19 Short offline Completed without error 00% 36831 -
#20 Short offline Completed without error 00% 36807 -
#21 Short offline Completed without error 00% 36783 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.