Hello everyone
My first post here, so i hope it looks good enough and all the needed info is here.
Its my first time trying TrueNAS, and therefore dont have all the knowledge i probably should have either
I have built a system, been lucky to get most very cheap or free, thats why the âhugeâ system.
I have just bought 6 brand new Seagate-drives, and im having issues with at least 3 of them consistently.
The problem im having at the moment, is with what seems to be defective drives, because i keep getting ZFS read/write-errors on the drives during scrub-task, and this puts the drives in a âFAULTEDâ state.
- All drives have PASSED short and LONG SMART-test, with no errors.
- All drives is âGoodâ according to Seagateâs own âSeaToolsâ.
- I have moved drives to different slots in cabinet, utilizing a different HBA (see system info). Error follows drives to new slot when running new scrub-task.
- I have tried to upgrade firmware on drives to newest SN04.
- I formatted drives in Windows, to force a resilver of the drive after firmware upgrade, still getting the ZFS read/write errors.
There is no issues copying data to the pool, i have only seen the error during scrub-task.
Question 1:
Is there any commands or so to âforceâ reset of a drive in regards to the read/write errors? Like, since you can swap drives around because its the ID of the drive being the identifier, does TrueNAS somehow ârememberâ that this drive was giving errors?
Question 2
I have noticed that it seems my drives get a new /dev/sd* letter on host reboot, is this normal, or is it a issue?
All 6 drives basically have the same test-results as shown below, so only posting two atm.
System info
OS: TrueNAS Scale 24.10.1
System
SuperMicro X10DRI-T4+
2x Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
256GB ECC RAM
HBA IT-mode
1x LSI 9300-16i
1x HP220/LSI 9205-8i
Cabinet:
InterTech 4U-4424
6 rows of 4-wide drive-slots
Each row has its own SAS/SATA backplane, one SAS-cable to each row from the HBAâs.
Pool info
Pools:
Boot
2x 480GB HPE MK000480GWUGF SSD-drives in MIRROR
Storage:
âSSD_Poolâ
6x 960GB HPE-branded MK000960GWSSD
âBygdePoolâ
18x 14TBdrives, 3 VDEVâs of RAIDZ2
12 drives HPE-branded MB014000GWUDA
6 drives Seagate ST14000NM001G
SMART info
SMART-info /dev/sda
root@truenas[/home/truenas_admin]# smartctl -a /dev/sda
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Exos X16
Device Model: ST14000NM001G-2KJ103
Serial Number: ZL2B769T
LU WWN Device Id: 5 000c50 0c81ccd58
Firmware Version: SN04
User Capacity: 14,000,519,643,136 bytes [14.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5625
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Jan 14 16:55:30 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 567) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: (1242) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 081 065 044 Pre-fail Always - 134535817
3 Spin_Up_Time 0x0003 093 093 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 14
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 072 060 045 Pre-fail Always - 16764616
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 291
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 13
18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 073 068 040 Old_age Always - 27 (Min/Max 27/29)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 6
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 560
194 Temperature_Celsius 0x0022 027 040 000 Old_age Always - 27 (0 21 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 63h+52m+22.824s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 4654908096
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 4767249956
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 44 -
# 2 Extended offline Interrupted (host reset) 00% 2 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
The above only provides legacy SMART information - try 'smartctl -x' for more
SMART-info /dev/sde
root@truenas[/home/truenas_admin]# smartctl -a /dev/sde
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Exos X16
Device Model: ST14000NM001G-2KJ103
Serial Number: ZLW1GZ28
LU WWN Device Id: 5 000c50 0c64f034f
Firmware Version: SN04
User Capacity: 14,000,519,643,136 bytes [14.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5625
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Jan 14 16:57:19 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 567) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: (1285) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 081 064 044 Pre-fail Always - 133653505
3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 20
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 074 060 045 Pre-fail Always - 27844817
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 292
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 17
18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 070 064 040 Old_age Always - 30 (Min/Max 29/32)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 9
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 510
194 Temperature_Celsius 0x0022 030 040 000 Old_age Always - 30 (0 22 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 84h+45m+15.183s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 13151632442
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 411820984
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 254 -
# 2 Extended offline Completed without error 00% 21 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
The above only provides legacy SMART information - try 'smartctl -x' for more
Hopefully all needed info is here, will fill in the blanks as needed.
I have the feeling that the drives are the problem, but i also have a feeling that im not gonna get it on warranty since SeaTools says its all good, but will see.
Thanks for any replies