Hi,
I’m looking for advice on the safest recovery path for my pool.
System
TrueNAS SCALE 25.10.3.1
AMD Ryzen 5 2600X
ASRock B450 Pro4 Rev. 1
4 × 16 GB ECC UDIMM (SMBIOS reports 64 GB installed, Linux currently reports only 32 GB usable)
LSI/Broadcom 9300-16i (IT mode)
Intel X520-DA2
4 × 12 TB Seagate IronWolf recertified drives in a RAIDZ1 pool (“main”)
Second RAIDZ1 pool (“Random”) is also connected to the same 9300-16i
What happened
I believe my LSI 9300-16i overheated due to inadequate cooling.
The card was mounted in a tight area between another PCIe card and initially only had a custom 60 mm fan shroud. Looking back, I don’t think the HBA was receiving enough airflow.
During normal operation one of the drives began reporting transport/I/O errors and was eventually marked FAULTED. TrueNAS automatically started a resilver because of this.
While the controller was still unstable, additional errors occurred, another drive was eventually marked REMOVED, and the pool entered an I/O suspended state.
I have since:
Installed a much more powerful Noctua fan running at 100%
Reseated both SFF-8643 SAS cables on the HBA
Rebooted the system
Since improving cooling, the controller appears stable and all drives are detected again.
Current state
Originally:
zpool commands were hanging
TrueNAS middleware became unresponsive
Pool was I/O suspended
One disk was FAULTED, later REMOVED
After improving cooling and running:
zpool clear main
The previously REMOVED disk came back ONLINE.
Current zpool status -v main:
pool: main
state: DEGRADED
scan: resilver in progress since Wed Jul 8 20:19:22 2026
247G / 3.08T scanned at 316M/s
0B / 3.08T issued
0B resilvered
0.00% done
config:
main
raidz1-0
ffc3fb7d... ONLINE
5e6e9220... DEGRADED (too many errors)
9884a530... ONLINE
4b385e6b... ONLINE
errors:
List of errors unavailable: pool I/O is currently suspended
The pool is no longer completely wedged, but the resilver never progresses beyond scanning. It scans data but issues 0 bytes and resilvers 0 bytes.
Hardware verification
All four drives are present and detected by Linux.
All four ZFS labels are readable:
zdb -l /dev/sdX1
The previously REMOVED disk (GUID 8548864585217646564) still contains valid ZFS labels and matches the pool metadata.
Both the previously REMOVED disk and the currently DEGRADED disk can be read normally:
dd if=/dev/sdl1 of=/dev/null bs=1M count=100
≈163 MB/s
dd if=/dev/sda1 of=/dev/null bs=1M count=100
≈157 MB/s
SMART does not indicate failing media (no reallocated sectors, no pending sectors, no offline uncorrectables).
PCIe reports the HBA running normally (x8 @ 8 GT/s) without AER errors.
Other observations
Both pools connected to the same 9300-16i experienced issues during the overheating event.
The second pool (“Random”) also accumulated checksum errors on one disk during the same period.
That is one reason I suspect the HBA overheating rather than simultaneous drive failures.
What I’ve already tried
Improved HBA cooling with a Noctua fan
Reseated both SAS cables
Rebooted
zpool clear main
After zpool clear, the previously REMOVED device returned ONLINE, but the pool still reports I/O suspended and the resilver remains stalled.
Question
Given that:
All four drives are present
All four drives are readable
All four ZFS labels are intact
The previously REMOVED device has returned ONLINE
The resilver is stalled (0 B issued)
The pool still reports I/O suspended
What would be the recommended recovery procedure?
Would the next step typically be:
zpool reopen
zpool replace using the same physical disk
export/import with rewind (-F)
another recovery method
I’d like to avoid making the situation worse and would appreciate advice on the safest recovery path.
Here are some “diagnostics“:
truenas_admin@truenas[~]$ zpool status -v main
pool: main
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 8 20:19:22 2026
247G / 3.08T scanned at 178M/s, 0B / 3.08T issued
0B resilvered, 0.00% done, no estimated completion time
config:
NAME STATE READ WRITE CKSUM
main DEGRADED 0 0 0
raidz1-0 DEGRADED 1 0 0
ffc3fb7d-bf05-454b-b033-03e29ff317be ONLINE 1 0 1
5e6e9220-e621-4ac6-abfc-10e0c3da91bd DEGRADED 0 82 0 too many errors
9884a530-9f73-4a72-aa74-cae9707fc325 ONLINE 0 0 0
4b385e6b-865e-459f-9d50-fa55263d860e ONLINE 0 0 0
errors: List of errors unavailable: pool I/O is currently suspended
truenas_admin@truenas[~]$
truenas_admin@truenas[~]$ sudo zpool events -v | head -200
[sudo] password for truenas_admin:
TIME CLASS
Jul 8 2026 19:38:36.780728337 ereport.fs.zfs.data
class = “ereport.fs.zfs.data”
ena = 0x162aab0b2802001
detector = (embedded nvlist)
version = 0x0
scheme = “zfs”
pool = 0xec0767b6aa9b3b97
(end detector)
pool = “main”
pool_guid = 0xec0767b6aa9b3b97
pool_state = 0x0
pool_context = 0x0
pool_failmode = “continue”
zio_err = 0x6
zio_flags = 0x10080a1 [DONT_AGGREGATE SCAN_THREAD CANFAIL IO_RETRY RAW_COMPRESS]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100000 [READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_objset = 0x40b
zio_object = 0x1875a
zio_level = 0x1
zio_blkid = 0x3
time = 0x6a4e8b1c 0x2e88f811
eid = 0x30d6
Jul 8 2026 19:38:36.780728337 ereport.fs.zfs.io
class = “ereport.fs.zfs.io”
ena = 0x162aaa748802c01
detector = (embedded nvlist)
version = 0x0
scheme = “zfs”
pool = 0xec0767b6aa9b3b97
vdev = 0xf0f776c77eace8c
(end detector)
pool = “main”
pool_guid = 0xec0767b6aa9b3b97
pool_state = 0x0
pool_context = 0x0
pool_failmode = “continue”
vdev_guid = 0xf0f776c77eace8c
vdev_type = “raidz”
vdev_ashift = 0xc
vdev_complete_ts = 0x0
vdev_delta_ts = 0x0
vdev_read_errors = 0x28a7
vdev_write_errors = 0x811
vdev_cksum_errors = 0x0
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0xec0767b6aa9b3b97
parent_type = “root”
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x6
zio_flags = 0x2080a1 [DONT_AGGREGATE SCAN_THREAD CANFAIL IO_RETRY DONT_PROPAGATE]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100000 [READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_offset = 0x5b00b61c000
zio_size = 0x3000
zio_objset = 0x40b
zio_object = 0x1875a
zio_level = 0x1
zio_blkid = 0x4
time = 0x6a4e8b1c 0x2e88f811
eid = 0x30d7
Jul 8 2026 19:38:36.780728337 ereport.fs.zfs.io
class = “ereport.fs.zfs.io”
ena = 0x162aaa05e801401
detector = (embedded nvlist)
version = 0x0
scheme = “zfs”
pool = 0xec0767b6aa9b3b97
vdev = 0xf0f776c77eace8c
(end detector)
pool = “main”
pool_guid = 0xec0767b6aa9b3b97
pool_state = 0x0
pool_context = 0x0
pool_failmode = “continue”
vdev_guid = 0xf0f776c77eace8c
vdev_type = “raidz”
vdev_ashift = 0xc
vdev_complete_ts = 0x0
vdev_delta_ts = 0x0
vdev_read_errors = 0x28a7
vdev_write_errors = 0x811
vdev_cksum_errors = 0x0
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0xec0767b6aa9b3b97
parent_type = “root”
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x6
zio_flags = 0x2080a1 [DONT_AGGREGATE SCAN_THREAD CANFAIL IO_RETRY DONT_PROPAGATE]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100000 [READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_offset = 0x5b00b620000
zio_size = 0x3000
zio_objset = 0x40b
zio_object = 0x1875a
zio_level = 0x1
zio_blkid = 0x5
time = 0x6a4e8b1c 0x2e88f811
eid = 0x30d8
Jul 8 2026 19:38:36.780728337 ereport.fs.zfs.io
class = “ereport.fs.zfs.io”
ena = 0x162aa9adbe00001
detector = (embedded nvlist)
version = 0x0
scheme = “zfs”
pool = 0xec0767b6aa9b3b97
vdev = 0xf0f776c77eace8c
(end detector)
pool = “main”
pool_guid = 0xec0767b6aa9b3b97
pool_state = 0x0
pool_context = 0x0
pool_failmode = “continue”
vdev_guid = 0xf0f776c77eace8c
vdev_type = “raidz”
vdev_ashift = 0xc
vdev_complete_ts = 0x0
vdev_delta_ts = 0x0
vdev_read_errors = 0x28a8
vdev_write_errors = 0x811
vdev_cksum_errors = 0x0
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0xec0767b6aa9b3b97
parent_type = “root”
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x6
zio_flags = 0x2080a1 [DONT_AGGREGATE SCAN_THREAD CANFAIL IO_RETRY DONT_PROPAGATE]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100000 [READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_offset = 0x5b00b624000
zio_size = 0x3000
zio_objset = 0x40b
zio_object = 0x1875a
zio_level = 0x1
zio_blkid = 0x6
time = 0x6a4e8b1c 0x2e88f811
eid = 0x30d9
Jul 8 2026 19:38:36.780728337 ereport.fs.zfs.io
class = “ereport.fs.zfs.io”
ena = 0x162aa94f9002001
detector = (embedded nvlist)
version = 0x0
scheme = “zfs”
pool = 0xec0767b6aa9b3b97
vdev = 0xf0f776c77eace8c
(end detector)
pool = “main”
pool_guid = 0xec0767b6aa9b3b97
pool_state = 0x0
pool_context = 0x0
pool_failmode = “continue”
vdev_guid = 0xf0f776c77eace8c
vdev_type = “raidz”
vdev_ashift = 0xc
vdev_complete_ts = 0x0
vdev_delta_ts = 0x0
vdev_read_errors = 0x28aa
vdev_write_errors = 0x811
vdev_cksum_errors = 0x0
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0xec0767b6aa9b3b97
parent_type = “root”
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x6
zio_flags = 0x2080a1 [DONT_AGGREGATE SCAN_THREAD CANFAIL IO_RETRY DONT_PROPAGATE]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100000 [READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
truenas_admin@truenas[~]$
truenas_admin@truenas[~]$ sudo dd if=/dev/sda1 of=/dev/null bs=1M count=100
sudo dd if=/dev/sdc1 of=/dev/null bs=1M count=100
sudo dd if=/dev/sdd1 of=/dev/null bs=1M count=100
sudo dd if=/dev/sdl1 of=/dev/null bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.665959 s, 157 MB/s
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.545953 s, 192 MB/s
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.533498 s, 197 MB/s
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.517768 s, 203 MB/s
truenas_admin@truenas[~]$
truenas_admin@truenas[~]$ sudo smartctl -a /dev/sda
sudo smartctl -a /dev/sdc
sudo smartctl -a /dev/sdd
sudo smartctl -a /dev/sdl
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate IronWolf
Device Model: ST12000VN0007-2GS116
Serial Number: ZJV5FL4J
LU WWN Device Id: 5 000c50 0b6aaf3de
Firmware Version: SC60
User Capacity: 12,000,138,625,024 bytes [12.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5528
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Jul 8 20:51:15 2026 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 567) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: (1091) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x50bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 074 064 044 Pre-fail Always - 25270248
3 Spin_Up_Time 0x0003 092 087 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1835
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 066 060 045 Pre-fail Always - 3904590
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1541
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 248
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 061 055 040 Old_age Always - 39 (Min/Max 39/39)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 259
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1903
194 Temperature_Celsius 0x0022 039 045 000 Old_age Always - 39 (0 15 0 0 0)
195 Hardware_ECC_Recovered 0x001a 008 004 000 Old_age Always - 25270248
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 1293h+54m+14.866s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2197399905
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1735780204
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
1 Short offline Completed without error 00% 407 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
The above only provides legacy SMART information - try ‘smartctl -x’ for more
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate IronWolf
Device Model: ST12000VN0007-2GS116
Serial Number: ZJV64VZR
LU WWN Device Id: 5 000c50 0c3575fe2
Firmware Version: SC60
User Capacity: 12,000,138,625,024 bytes [12.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5528
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Jul 8 20:51:15 2026 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 592) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: (1102) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x50bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 076 064 044 Pre-fail Always - 42985704
3 Spin_Up_Time 0x0003 089 088 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1858
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 066 060 045 Pre-fail Always - 4027886
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1639
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 265
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 099 000 Old_age Always - 4295032833
190 Airflow_Temperature_Cel 0x0022 061 047 040 Old_age Always - 39 (Min/Max 35/39)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 276
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1925
194 Temperature_Celsius 0x0022 039 053 000 Old_age Always - 39 (0 15 0 0 0)
195 Hardware_ECC_Recovered 0x001a 008 007 000 Old_age Always - 42985704
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 1353h+47m+36.508s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2198445273
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1758024167
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
1 Short offline Completed without error 00% 505 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
The above only provides legacy SMART information - try ‘smartctl -x’ for more
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate IronWolf
Device Model: ST12000VN0007-2GS116
Serial Number: ZJV4G1ZF
LU WWN Device Id: 5 000c50 0b58ac71f
Firmware Version: SC60
User Capacity: 12,000,138,625,024 bytes [12.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5528
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Jul 8 20:51:15 2026 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 567) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: (1100) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x50bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 075 066 044 Pre-fail Always - 32557904
3 Spin_Up_Time 0x0003 088 087 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1830
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 066 060 045 Pre-fail Always - 3967384
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1541
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 237
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 062 050 040 Old_age Always - 38 (Min/Max 35/38)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 246
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1899
194 Temperature_Celsius 0x0022 038 050 000 Old_age Always - 38 (0 14 0 0 0)
195 Hardware_ECC_Recovered 0x001a 008 006 000 Old_age Always - 32557904
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 1292h+43m+29.265s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2197296321
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1743497358
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
1 Short offline Completed without error 00% 407 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
The above only provides legacy SMART information - try ‘smartctl -x’ for more
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate IronWolf
Device Model: ST12000VN0007-2GS116
Serial Number: ZJV28SG4
LU WWN Device Id: 5 000c50 0b2ce4ac6
Firmware Version: SC60
User Capacity: 12,000,138,625,024 bytes [12.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5528
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Jul 8 20:51:15 2026 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 567) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: (1065) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x50bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 064 044 Pre-fail Always - 229320
3 Spin_Up_Time 0x0003 093 087 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1834
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 065 060 045 Pre-fail Always - 3271209
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1639
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 295
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 062 049 040 Old_age Always - 38 (Min/Max 38/38)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 308
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1899
194 Temperature_Celsius 0x0022 038 051 000 Old_age Always - 38 (0 15 0 0 0)
195 Hardware_ECC_Recovered 0x001a 100 007 000 Old_age Always - 229320
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 1327h+09m+32.393s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 1500311649
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 943633014
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
1 Short offline Completed without error 00% 505 -
2 Short offline Completed without error 00% 505 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
The above only provides legacy SMART information - try ‘smartctl -x’ for more
truenas_admin@truenas[~]$
truenas_admin@truenas[~]$ lsblk -o NAME,SIZE,MODEL,SERIAL
NAME SIZE MODEL SERIAL
sda 10.9T ST12000VN0007-2GS116 ZJV5FL4J
└─sda1 10.9T
sdc 10.9T ST12000VN0007-2GS116 ZJV64VZR
└─sdc1 10.9T
sdd 10.9T ST12000VN0007-2GS116 ZJV4G1ZF
└─sdd1 10.9T
sde 223.6G INTENSO SSD 1832501006002521
└─sde1 221.6G
sdf 931.5G ST1000DM003-1SB10C Z9A1XKFB
└─sdf1 931.5G
sdg 931.5G ST1000DM003-1SB10C Z9A13YMF
└─sdg1 931.5G
sdh 931.5G ST1000DM010-2EP102 Z9ASMH73
└─sdh1 931.5G
sdi 223.6G INTENSO SSD 1832501006002527
└─sdi1 221.6G
sdj 119.2G INTENSO SSD 1642312010002910
├─sdj1 1M
├─sdj2 512M
└─sdj3 118.7G
sdk 119.2G INTENSO SSD 1642411017005680
├─sdk1 1M
├─sdk2 512M
└─sdk3 118.7G
sdl 10.9T ST12000VN0007-2GS116 ZJV28SG4
└─sdl1 10.9T
truenas_admin@truenas[~]$ ls -l /dev/disk/by-partuuid
total 0
lrwxrwxrwx 1 root root 10 Jul 8 19:37 0f6467f6-75e2-4a14-b86b-cef09a2ec4d1 → ../../sdj3
lrwxrwxrwx 1 root root 10 Jul 8 19:37 4383b79f-676d-441e-b227-3ef1ac0ac7c7 → ../../sdj1
lrwxrwxrwx 1 root root 10 Jul 8 19:37 46f49cae-c0db-4d76-9bbb-e5e614bdc625 → ../../sdh1
lrwxrwxrwx 1 root root 10 Jul 8 19:37 4b385e6b-865e-459f-9d50-fa55263d860e → ../../sdd1
lrwxrwxrwx 1 root root 10 Jul 8 20:19 5e6e9220-e621-4ac6-abfc-10e0c3da91bd → ../../sda1
lrwxrwxrwx 1 root root 10 Jul 8 19:37 73eb4ce7-fb0e-42f7-98ce-6268ba7b8b1d → ../../sdf1
lrwxrwxrwx 1 root root 10 Jul 8 19:37 90e2a293-5ec6-4cdb-b06e-8ac072b602bb → ../../sdk3
lrwxrwxrwx 1 root root 10 Jul 8 19:37 9884a530-9f73-4a72-aa74-cae9707fc325 → ../../sdc1
lrwxrwxrwx 1 root root 10 Jul 8 19:37 9df24b74-2a15-4055-9618-a38b6f0d5e07 → ../../sdg1
lrwxrwxrwx 1 root root 10 Jul 8 19:37 a31e17a3-8230-4612-b21c-249270e28764 → ../../sdk1
lrwxrwxrwx 1 root root 10 Jul 8 19:37 bb251bde-f7fb-4699-ac46-c09806361a8d → ../../sdi1
lrwxrwxrwx 1 root root 10 Jul 8 19:37 c00f8e3c-770d-425a-991a-2a78ea0c66a1 → ../../sde1
lrwxrwxrwx 1 root root 10 Jul 8 19:37 c92f52b5-f406-425f-9fe4-34115d01db06 → ../../sdj2
lrwxrwxrwx 1 root root 10 Jul 8 19:37 d1521c2a-3369-4951-a360-fe41e2004a8f → ../../sdk2
lrwxrwxrwx 1 root root 10 Jul 8 19:38 ffc3fb7d-bf05-454b-b033-03e29ff317be → ../../sdl1
truenas_admin@truenas[~]$ blkid
/dev/sdf1: LABEL=“Random” UUID=“12582594206820849199” UUID_SUB=“2607645664706507753” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTLABEL=“data” PARTUUID=“73eb4ce7-fb0e-42f7-98ce-6268ba7b8b1d”
/dev/sdd1: LABEL=“main” UUID=“17007676552031976343” UUID_SUB=“16372654193624753384” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTLABEL=“data” PARTUUID=“4b385e6b-865e-459f-9d50-fa55263d860e”
/dev/sdk3: LABEL=“boot-pool” UUID=“2951156790391227957” UUID_SUB=“1496345295812187660” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTUUID=“90e2a293-5ec6-4cdb-b06e-8ac072b602bb”
/dev/sdk2: LABEL_FATBOOT=“EFI” LABEL=“EFI” UUID=“57F1-E67D” BLOCK_SIZE=“512” TYPE=“vfat” PARTUUID=“d1521c2a-3369-4951-a360-fe41e2004a8f”
/dev/sdi1: LABEL=“container” UUID=“16125537511583003480” UUID_SUB=“6442284821801367228” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTUUID=“bb251bde-f7fb-4699-ac46-c09806361a8d”
/dev/sdg1: LABEL=“Random” UUID=“12582594206820849199” UUID_SUB=“1545178342034039521” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTLABEL=“data” PARTUUID=“9df24b74-2a15-4055-9618-a38b6f0d5e07”
/dev/sde1: LABEL=“container” UUID=“16125537511583003480” UUID_SUB=“17308548853587993958” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTUUID=“c00f8e3c-770d-425a-991a-2a78ea0c66a1”
/dev/sdc1: LABEL=“main” UUID=“17007676552031976343” UUID_SUB=“2656408007331260726” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTLABEL=“data” PARTUUID=“9884a530-9f73-4a72-aa74-cae9707fc325”
/dev/sdj3: LABEL=“boot-pool” UUID=“2951156790391227957” UUID_SUB=“2071784999058496617” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTUUID=“0f6467f6-75e2-4a14-b86b-cef09a2ec4d1”
/dev/sdj2: LABEL_FATBOOT=“EFI” LABEL=“EFI” UUID=“5804-4E12” BLOCK_SIZE=“512” TYPE=“vfat” PARTUUID=“c92f52b5-f406-425f-9fe4-34115d01db06”
/dev/sdh1: LABEL=“Random” UUID=“12582594206820849199” UUID_SUB=“8573114513596911256” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTLABEL=“data” PARTUUID=“46f49cae-c0db-4d76-9bbb-e5e614bdc625”
truenas_admin@truenas[~]$
truenas_admin@truenas[~]$ lsscsi
[0:0:2:0] disk ATA ST12000VN0007-2G SC60 /dev/sdc
[0:0:3:0] disk ATA ST12000VN0007-2G SC60 /dev/sdd
[0:0:4:0] disk ATA ST12000VN0007-2G SC60 /dev/sdl
[0:0:6:0] disk ATA ST12000VN0007-2G SC60 /dev/sda
[1:0:0:0] disk ATA INTENSO SSD 9A0 /dev/sde
[5:0:0:0] disk ATA INTENSO SSD 9A0 /dev/sdk
[6:0:0:0] disk ATA INTENSO SSD 4A0 /dev/sdj
[9:0:0:0] disk ATA INTENSO SSD 9A0 /dev/sdi
[12:0:0:0] disk ATA ST1000DM003-1SB1 CC43 /dev/sdf
[12:0:1:0] disk ATA ST1000DM003-1SB1 CC43 /dev/sdg
[12:0:2:0] disk ATA ST1000DM010-2EP1 CC43 /dev/sdh
truenas_admin@truenas[~]$ lspci -nn | grep -i sas
0c:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
0e:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
truenas_admin@truenas[~]$
truenas_admin@truenas[~]$ zpool status -g main
pool: main
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 8 20:19:22 2026
247G / 3.08T scanned at 122M/s, 0B / 3.08T issued
0B resilvered, 0.00% done, no estimated completion time
config:
NAME STATE READ WRITE CKSUM
main DEGRADED 0 0 0
1085217342971629196 DEGRADED 1 0 0
8548864585217646564 ONLINE 1 0 1
1042007926756595463 DEGRADED 0 82 0 too many errors
2656408007331260726 ONLINE 0 0 0
16372654193624753384 ONLINE 0 0 0
errors: 2230 data errors, use ‘-v’ for a list
truenas_admin@truenas[~]$
