Can we replace disk prior pool import?

TrueNAS General
,
1

Can i still save this?

root@truenas[/home/admin]# zpool import           
  pool: ThePool
    id: 2403027406607053466
 state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:

        ThePool                                   DEGRADED
          indirect-0                              ONLINE
          indirect-1                              ONLINE
          indirect-2                              ONLINE
          mirror-3                                ONLINE
            2a2eedf3-947d-44f1-be7a-218837ad9d1f  ONLINE
            ad597db7-b4c0-4032-a811-ba5c8b336248  ONLINE
          mirror-4                                DEGRADED
            ae13e53b-c330-4a3f-877f-3209735dc59d  ONLINE
            d486c78a-f360-4873-9132-9d2898f6c5e3  UNAVAIL
          mirror-5                                ONLINE
            850d4eb8-f5be-497c-93f9-873f84d32ad7  ONLINE
            f42cd54a-c9e7-4484-bd3c-996be41c6292  ONLINE
          mirror-6                                ONLINE
            34beb21d-18bf-4f19-b44f-688f4dab38d8  ONLINE
            9f55ee92-25d9-4069-bd39-b2b9a48a4568  ONLINE
          mirror-7                                ONLINE
            9c13fea5-8486-44d9-9fc5-0f7fec409119  ONLINE
            0bd5a09a-0a9c-402f-b5af-086c9bf08c73  ONLINE
          mirror-8                                DEGRADED
            93afe618-bf4a-49bd-96f8-6bf008a68312  ONLINE
            ef26faca-d964-468b-9a87-712fa02ceee0  UNAVAIL
          mirror-9                                ONLINE
            7868ef19-47e1-4647-af83-39f848b96ada  ONLINE
            3b8401eb-0714-4976-97be-ed7ccf8b3eae  ONLINE
          mirror-10                               ONLINE
            b16aa769-6f78-4757-80cb-b073e56f87ae  ONLINE
            4d255512-b2d5-4da4-b520-1287263d4d44  ONLINE
root@truenas[/home/admin]#

mirror-10 was added with larger capacity to replace mirror-4 which already showing error. i should just replace disk instead i add more vdev. nevertheless, while removing mirror-4 vdev, my system reboots and stuck.

being ignorant, once able to login in to the GUI i exported the pool, trying to import again and the OS keep restarting.

i have pin point several disk that showing error and now my spare disk has arrived.

my query here, i have pullout the disk, can i replace those and import again my pool? or i just have lost everything?

2

mirror-4 and mirror-8 are degraded - so they are still working and your pool is still ok. If you lose the other disk in either mirror before you resolve this, you will lose your entire pool.

If mirror-10 was to replace mirror-4 then (rather than resilvering) the simplest solution to this mirror being degraded is to remove the vDev from the pool. TrueNAS / ZFS will move the data off that vDev and once that has completed that issue will be resolved.

mirror-8 will need to have the UNAVAIL drive replaced and resilvered.

We don’t know what size your drives are, but you could potentially replace the mirror-8 drive with the freed up mirror-4 drive.

You should probably do a smartctl -x /dev/sdX for the UNAVAIL drives to see what their hardware status is. You might want to run the following command in order to get the mappings of device names to partuuids:

  • lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID

P.S. In future an alternative (and perhaps better) way of replacing mirror-4 would be to add both new drives as additional mirrors to the mirror-4 vDev and then remove the two old drives.

2 Likes
3

What are these ? I hope not a stripe attached to the mirrors…

4

Pointers to previous mirror vDevs that were removed in the past.

3 Likes
5

Latest update. system wont boot at all when i slotted in either of the mirror-10 disks. and now i think im fck-ed

root@truenas[/home/admin]# zpool import
  pool: ThePool
    id: 2403027406607053466
 state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E
config:

        ThePool                                   UNAVAIL  insufficient replicas
          indirect-0                              ONLINE
          indirect-1                              ONLINE
          indirect-2                              ONLINE
          mirror-3                                ONLINE
            2a2eedf3-947d-44f1-be7a-218837ad9d1f  ONLINE
            ad597db7-b4c0-4032-a811-ba5c8b336248  ONLINE
          mirror-4                                ONLINE
            ae13e53b-c330-4a3f-877f-3209735dc59d  ONLINE
            d486c78a-f360-4873-9132-9d2898f6c5e3  ONLINE
          mirror-5                                ONLINE
            850d4eb8-f5be-497c-93f9-873f84d32ad7  ONLINE
            f42cd54a-c9e7-4484-bd3c-996be41c6292  ONLINE
          mirror-6                                ONLINE
            34beb21d-18bf-4f19-b44f-688f4dab38d8  ONLINE
            9f55ee92-25d9-4069-bd39-b2b9a48a4568  ONLINE
          mirror-7                                ONLINE
            9c13fea5-8486-44d9-9fc5-0f7fec409119  ONLINE
            0bd5a09a-0a9c-402f-b5af-086c9bf08c73  ONLINE
          mirror-8                                DEGRADED
            93afe618-bf4a-49bd-96f8-6bf008a68312  ONLINE
            16447861092670188178                  UNAVAIL
          mirror-9                                ONLINE
            7868ef19-47e1-4647-af83-39f848b96ada  ONLINE
            3b8401eb-0714-4976-97be-ed7ccf8b3eae  ONLINE
          mirror-10                               UNAVAIL  insufficient replicas
            b16aa769-6f78-4757-80cb-b073e56f87ae  UNAVAIL
            16279357827871785108                  UNAVAIL
root@truenas[/home/admin]# 
root@truenas[/home/admin]# lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID
NAME   MODEL                ROTA PTTYPE TYPE   START          SIZE PARTTYPENAME             PARTUUID
sda    HUS726040ALA610         1 gpt    disk         4000787030016                          
└─sda1                         1 gpt    part    2048 4000785964544 Solaris /usr & Apple ZFS f42cd54a-c9e7-4484-bd3c-996be41c6292
sdb    ST4000NM0033-9ZM170     1 gpt    disk         4000787030016                          
└─sdb1                         1 gpt    part    2048 4000785964544 Solaris /usr & Apple ZFS ae13e53b-c330-4a3f-877f-3209735dc59d
sdc    ST6000NM0115-1YZ110     1 gpt    disk         6001175126016                          
└─sdc1                         1 gpt    part    2048 6001174060544 Solaris /usr & Apple ZFS 0bd5a09a-0a9c-402f-b5af-086c9bf08c73
sdd    HGST HUS724040ALA640    1 gpt    disk         4000787030016                          
└─sdd1                         1 gpt    part    2048 4000785964544 Solaris /usr & Apple ZFS 9f55ee92-25d9-4069-bd39-b2b9a48a4568
sde    ST6000NM0024-1HT17Z     1 gpt    disk         6001175126016                          
└─sde1                         1 gpt    part    2048 6001174060544 Solaris /usr & Apple ZFS 3b8401eb-0714-4976-97be-ed7ccf8b3eae
sdf    ST6000NM0115-1YZ110     1 gpt    disk         6001175126016                          
└─sdf1                         1 gpt    part    2048 6001174060544 Solaris /usr & Apple ZFS 9c13fea5-8486-44d9-9fc5-0f7fec409119
sdg    ST6000NM0024-1HT17Z     1 gpt    disk         6001175126016                          
└─sdg1                         1 gpt    part    2048 6001174060544 Solaris /usr & Apple ZFS 7868ef19-47e1-4647-af83-39f848b96ada
sdh    WDC WD60EFRX-68L0BN1    1 gpt    disk         6001175126016                          
└─sdh1                         1 gpt    part    2048 6001174060544 Solaris /usr & Apple ZFS 93afe618-bf4a-49bd-96f8-6bf008a68312
sdi    WDC WD10EZEX-00BN5A0    1 gpt    disk         1000204886016                          
└─sdi1                         1 gpt    part    2048 1000203820544 Solaris /usr & Apple ZFS ad597db7-b4c0-4032-a811-ba5c8b336248
sdj    TOSHIBA HDWE140         1 gpt    disk         4000787030016                          
└─sdj1                         1 gpt    part    2048 4000785964544 Solaris /usr & Apple ZFS 34beb21d-18bf-4f19-b44f-688f4dab38d8
sdk    ST1000DM010-2EP102      1 gpt    disk         1000204886016                          
├─sdk1                         1 gpt    part    4096       1048576 BIOS boot                d8c2595c-f565-456b-ba06-5e2fd870d03c
├─sdk2                         1 gpt    part    6144     536870912 EFI System               9f9ef43f-76b8-4c76-8791-7a7535a4fefc
└─sdk3                         1 gpt    part 1054720  999664852480 Solaris /usr & Apple ZFS 3917b03d-d3f8-4442-8a10-0f4b208f5609
sdl    HGST HUS724040ALA640    1 gpt    disk         4000787030016                          
└─sdl1                         1 gpt    part    2048 4000785964544 Solaris /usr & Apple ZFS 850d4eb8-f5be-497c-93f9-873f84d32ad7
sdm    WDC WD10EZEX-00BN5A0    1 gpt    disk         1000204886016                          
└─sdm1                         1 gpt    part    2048 1000203820544 Solaris /usr & Apple ZFS 2a2eedf3-947d-44f1-be7a-218837ad9d1f
sdn    ST4000NM0033-9ZM170     1 gpt    disk         4000787030016                          
└─sdn1                         1 gpt    part    2048 4000785964544 Solaris /usr & Apple ZFS d486c78a-f360-4873-9132-9d2898f6c5e3
root@truenas[/home/admin]# 
root@truenas[/home/admin]#
6

If you can bring one or both of the mirror-10 drives back online your pool should be importable.

What exactly did you do to make them go offline?

And how / why did mirror-4 become non-degraded?

7

What i can see, with all disk, during boot startup initializing, it freezes and stuck. like this.

image
image1156Ă—868 154 KB

if pulling out one of the mirror-10 disk, it stuck here…

image
image1156Ă—868 161 KB

Well after pulling off couple of problematic disk which i remember having the checksum error. then it starting gracefully. Earlier i had to reseat the disk from mirror-4. But now seems booting fine.
But if i inserted the disks back and run zpool import, systems restarts and stuck again

here are the smartctl for mirror-10 disks. after boot, i insert it back again…

root@truenas[/home/admin]# smartctl -x /dev/sdo   
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     HGST Ultrastar DC HC520 (He12)
Device Model:     HGST HUH721212ALE600
Serial Number:    D5HRT9LF
LU WWN Device Id: 5 000cca 294d872ec
Add. Product Id:  DELL(tm)
Firmware Version: LEDENT10
User Capacity:    12,000,138,625,024 bytes [12.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5671
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Apr 25 18:40:00 2025 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM level is:     254 (maximum performance)
Rd look-ahead is: Enabled
Write cache is:   Disabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (   90) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1339) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     PO-R--   100   100   016    -    0
  2 Throughput_Performance  --S---   130   130   000    -    100
  3 Spin_Up_Time            POS---   162   162   024    -    394 (Average 418)
  4 Start_Stop_Count        -O--C-   100   100   000    -    45
  5 Reallocated_Sector_Ct   PO--CK   100   100   005    -    0
  7 Seek_Error_Rate         -O-R--   100   100   000    -    0
  8 Seek_Time_Performance   --S---   128   128   000    -    18
  9 Power_On_Hours          -O--C-   100   100   000    -    2067
 10 Spin_Retry_Count        -O--C-   100   100   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    44
 22 Helium_Level            PO---K   100   100   025    -    6553700
192 Power-Off_Retract_Count -O--CK   100   100   000    -    129
193 Load_Cycle_Count        -O--C-   100   100   000    -    129
194 Temperature_Celsius     -O----   193   193   000    -    31 (Min/Max 18/50)
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O---K   100   100   000    -    0
198 Offline_Uncorrectable   ---R--   100   100   000    -    0
199 UDMA_CRC_Error_Count    -O-R--   200   200   000    -    0
223 Load_Retry_Count        -O-R--   100   100   000    -    0
241 Total_LBAs_Written      -O--C-   100   100   000    -    355001130
242 Total_LBAs_Read         -O--C-   100   100   000    -    23437586461
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      1  Comprehensive SMART error log
0x03       GPL     R/O      1  Ext. Comprehensive SMART error log
0x04       GPL     R/O    256  Device Statistics log
0x04       SL      R/O    255  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0c       GPL     R/O   5501  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x12       GPL     R/O      1  SATA NCQ Non-Data log
0x13       GPL     R/O      1  SATA NCQ Send and Receive log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    256  Current Device Internal Status Data log
0x25       GPL     R/O    256  Saved Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xdf       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Aborted by host               20%      2066         -
# 2  Short offline       Completed without error       00%      2049         -
# 3  Vendor (0xdf)       Completed without error       00%         4         -
# 4  Short offline       Completed without error       00%         2         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       256 (0x0100)
Device State:                        Active (0)
Current Temperature:                    31 Celsius
Power Cycle Min/Max Temperature:     31/31 Celsius
Lifetime    Min/Max Temperature:     18/50 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -40/70 Celsius
Temperature History Size (Index):    128 (10)

Index    Estimated Time   Temperature Celsius
  11    2025-04-25 16:33    48  *****************************
  12    2025-04-25 16:34    49  ******************************
  13    2025-04-25 16:35    48  *****************************
 ...    ..( 20 skipped).    ..  *****************************
  34    2025-04-25 16:56    48  *****************************
  35    2025-04-25 16:57    49  ******************************
  36    2025-04-25 16:58    49  ******************************
  37    2025-04-25 16:59    48  *****************************
 ...    ..(  3 skipped).    ..  *****************************
  41    2025-04-25 17:03    48  *****************************
  42    2025-04-25 17:04    49  ******************************
  43    2025-04-25 17:05    48  *****************************
  44    2025-04-25 17:06    48  *****************************
  45    2025-04-25 17:07    49  ******************************
  46    2025-04-25 17:08    48  *****************************
  47    2025-04-25 17:09    48  *****************************
  48    2025-04-25 17:10    48  *****************************
  49    2025-04-25 17:11    49  ******************************
  50    2025-04-25 17:12    48  *****************************
  51    2025-04-25 17:13    49  ******************************
  52    2025-04-25 17:14    49  ******************************
  53    2025-04-25 17:15    48  *****************************
  54    2025-04-25 17:16    48  *****************************
  55    2025-04-25 17:17    49  ******************************
  56    2025-04-25 17:18    48  *****************************
  57    2025-04-25 17:19    49  ******************************
  58    2025-04-25 17:20    49  ******************************
  59    2025-04-25 17:21    48  *****************************
  60    2025-04-25 17:22    49  ******************************
 ...    ..( 45 skipped).    ..  ******************************
 106    2025-04-25 18:08    49  ******************************
 107    2025-04-25 18:09    48  *****************************
 108    2025-04-25 18:10    48  *****************************
 109    2025-04-25 18:11    47  ****************************
 110    2025-04-25 18:12    47  ****************************
 111    2025-04-25 18:13    47  ****************************
 112    2025-04-25 18:14    45  **************************
 ...    ..(  7 skipped).    ..  **************************
 120    2025-04-25 18:22    45  **************************
 121    2025-04-25 18:23    38  *******************
 122    2025-04-25 18:24    39  ********************
 123    2025-04-25 18:25    39  ********************
 124    2025-04-25 18:26    36  *****************
 125    2025-04-25 18:27    36  *****************
 126    2025-04-25 18:28    37  ******************
 127    2025-04-25 18:29    37  ******************
   0    2025-04-25 18:30    37  ******************
   1    2025-04-25 18:31    36  *****************
   2    2025-04-25 18:32    36  *****************
   3    2025-04-25 18:33    37  ******************
   4    2025-04-25 18:34    37  ******************
   5    2025-04-25 18:35    28  *********
   6    2025-04-25 18:36    29  **********
   7    2025-04-25 18:37    29  **********
   8    2025-04-25 18:38    31  ************
   9    2025-04-25 18:39    31  ************
  10    2025-04-25 18:40    48  *****************************

SCT Error Recovery Control:
           Read:     80 (8.0 seconds)
          Write:     80 (8.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              44  ---  Lifetime Power-On Resets
0x01  0x010  4            2067  ---  Power-on Hours
0x01  0x018  6       355001130  ---  Logical Sectors Written
0x01  0x020  6          218823  ---  Number of Write Commands
0x01  0x028  6     23437586461  ---  Logical Sectors Read
0x01  0x030  6        45785472  ---  Number of Read Commands
0x01  0x038  6      7443162000  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4            2065  ---  Spindle Motor Power-on Hours
0x03  0x010  4            2065  ---  Head Flying Hours
0x03  0x018  4             129  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               2  ---  Number of Mechanical Start Failures
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              31  ---  Current Temperature
0x05  0x010  1              46  N--  Average Short Term Temperature
0x05  0x018  1              34  N--  Average Long Term Temperature
0x05  0x020  1              50  ---  Highest Temperature
0x05  0x028  1              18  ---  Lowest Temperature
0x05  0x030  1              47  N--  Highest Average Short Term Temperature
0x05  0x038  1              24  N--  Lowest Average Short Term Temperature
0x05  0x040  1              34  N--  Highest Average Long Term Temperature
0x05  0x048  1              24  N--  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              60  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4             137  ---  Number of Hardware Resets
0x06  0x010  4              43  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0xff  =====  =               =  ===  == Vendor Specific Statistics (rev 1) ==
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            1  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2            1  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS

root@truenas[/home/admin]# smartctl -x /dev/sdp
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     HGST Ultrastar DC HC520 (He12)
Device Model:     HGST HUH721212ALE600
Serial Number:    D5HMG0LF
LU WWN Device Id: 5 000cca 294d6ef2e
Add. Product Id:  DELL(tm)
Firmware Version: LEDENT10
User Capacity:    12,000,138,625,024 bytes [12.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5671
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Apr 25 18:40:03 2025 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM level is:     254 (maximum performance)
Rd look-ahead is: Enabled
Write cache is:   Disabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (   90) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1164) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     PO-R--   100   100   016    -    0
  2 Throughput_Performance  --S---   133   133   000    -    92
  3 Spin_Up_Time            POS---   164   164   024    -    399 (Average 401)
  4 Start_Stop_Count        -O--C-   100   100   000    -    51
  5 Reallocated_Sector_Ct   PO--CK   100   100   005    -    0
  7 Seek_Error_Rate         -O-R--   100   100   000    -    0
  8 Seek_Time_Performance   --S---   128   128   000    -    18
  9 Power_On_Hours          -O--C-   100   100   000    -    2265
 10 Spin_Retry_Count        -O--C-   100   100   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    50
 22 Helium_Level            PO---K   100   100   025    -    6553700
192 Power-Off_Retract_Count -O--CK   100   100   000    -    142
193 Load_Cycle_Count        -O--C-   100   100   000    -    142
194 Temperature_Celsius     -O----   193   193   000    -    31 (Min/Max 17/48)
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O---K   100   100   000    -    0
198 Offline_Uncorrectable   ---R--   100   100   000    -    0
199 UDMA_CRC_Error_Count    -O-R--   200   200   000    -    0
223 Load_Retry_Count        -O-R--   100   100   000    -    0
241 Total_LBAs_Written      -O--C-   100   100   000    -    355873260
242 Total_LBAs_Read         -O--C-   100   100   000    -    23437513862
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      1  Comprehensive SMART error log
0x03       GPL     R/O      1  Ext. Comprehensive SMART error log
0x04       GPL     R/O    256  Device Statistics log
0x04       SL      R/O    255  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0c       GPL     R/O   5501  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x12       GPL     R/O      1  SATA NCQ Non-Data log
0x13       GPL     R/O      1  SATA NCQ Send and Receive log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    256  Current Device Internal Status Data log
0x25       GPL     R/O    256  Saved Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xdf       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Aborted by host               10%      2264         -
# 2  Short offline       Completed without error       00%      2246         -
# 3  Vendor (0xdf)       Completed without error       00%       202         -
# 4  Short offline       Completed without error       00%       200         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       256 (0x0100)
Device State:                        Active (0)
Current Temperature:                    31 Celsius
Power Cycle Min/Max Temperature:     31/31 Celsius
Lifetime    Min/Max Temperature:     17/48 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -40/70 Celsius
Temperature History Size (Index):    128 (31)

Index    Estimated Time   Temperature Celsius
  32    2025-04-25 16:33    48  *****************************
 ...    ..( 84 skipped).    ..  *****************************
 117    2025-04-25 17:58    48  *****************************
 118    2025-04-25 17:59    47  ****************************
 ...    ..(  3 skipped).    ..  ****************************
 122    2025-04-25 18:03    47  ****************************
 123    2025-04-25 18:04    45  **************************
 ...    ..(  3 skipped).    ..  **************************
 127    2025-04-25 18:08    45  **************************
   0    2025-04-25 18:09    39  ********************
 ...    ..(  4 skipped).    ..  ********************
   5    2025-04-25 18:14    39  ********************
   6    2025-04-25 18:15    40  *********************
 ...    ..(  2 skipped).    ..  *********************
   9    2025-04-25 18:18    40  *********************
  10    2025-04-25 18:19    36  *****************
 ...    ..(  3 skipped).    ..  *****************
  14    2025-04-25 18:23    36  *****************
  15    2025-04-25 18:24    35  ****************
  16    2025-04-25 18:25    28  *********
  17    2025-04-25 18:26    28  *********
  18    2025-04-25 18:27    29  **********
  19    2025-04-25 18:28    29  **********
  20    2025-04-25 18:29    30  ***********
  21    2025-04-25 18:30    30  ***********
  22    2025-04-25 18:31    31  ************
 ...    ..(  2 skipped).    ..  ************
  25    2025-04-25 18:34    31  ************
  26    2025-04-25 18:35    32  *************
  27    2025-04-25 18:36    32  *************
  28    2025-04-25 18:37    32  *************
  29    2025-04-25 18:38    31  ************
  30    2025-04-25 18:39    31  ************
  31    2025-04-25 18:40    48  *****************************

SCT Error Recovery Control:
           Read:     80 (8.0 seconds)
          Write:     80 (8.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              50  ---  Lifetime Power-On Resets
0x01  0x010  4            2265  ---  Power-on Hours
0x01  0x018  6       355873260  ---  Logical Sectors Written
0x01  0x020  6          218854  ---  Number of Write Commands
0x01  0x028  6     23437513862  ---  Logical Sectors Read
0x01  0x030  6        45785080  ---  Number of Read Commands
0x01  0x038  6      8154788250  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4            2255  ---  Spindle Motor Power-on Hours
0x03  0x010  4            2255  ---  Head Flying Hours
0x03  0x018  4             142  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               1  ---  Number of Mechanical Start Failures
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               3  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              31  ---  Current Temperature
0x05  0x010  1              45  N--  Average Short Term Temperature
0x05  0x018  1              32  N--  Average Long Term Temperature
0x05  0x020  1              48  ---  Highest Temperature
0x05  0x028  1              17  ---  Lowest Temperature
0x05  0x030  1              46  N--  Highest Average Short Term Temperature
0x05  0x038  1              24  N--  Lowest Average Short Term Temperature
0x05  0x040  1              37  N--  Highest Average Long Term Temperature
0x05  0x048  1              24  N--  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              60  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4             269  ---  Number of Hardware Resets
0x06  0x010  4              86  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0xff  =====  =               =  ===  == Vendor Specific Statistics (rev 1) ==
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            1  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2            2  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS

root@truenas[/home/admin]#

Moving forward, i had to accept the fact that i’ve lost this pool.
what do you think? is this still salvageable?

8

here are for mirror-4 disks

root@truenas[/home/admin]# smartctl -x /dev/sde                                                  
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Constellation ES.3
Device Model:     ST4000NM0033-9ZM170
Serial Number:    Z1Z44WNV
LU WWN Device Id: 5 000c50 066f1eb65
Add. Product Id:  DELL(tm)
Firmware Version: GA0A
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5671
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Apr 25 18:48:35 2025 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (   90) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 491) minutes.
Conveyance self-test routine
recommended polling time:        (   3) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--+  078   063   044    -    68315687
  3 Spin_Up_Time            PO----+  091   089   000    -    0
  4 Start_Stop_Count        -O--CK   099   099   020    -    2040
  5 Reallocated_Sector_Ct   PO--CK+  100   100   010    -    63
  7 Seek_Error_Rate         POSR--   093   060   030    -    2029644592
  9 Power_On_Hours          -O--CK   012   012   000    -    77174
 10 Spin_Retry_Count        PO--C-   100   100   097    -    0
 12 Power_Cycle_Count       -O--CK   100   100   020    -    977
184 End-to-End_Error        -O--CK   100   100   099    -    0
187 Reported_Uncorrect      -O--CK   097   097   000    -    3
188 Command_Timeout         -O--CK   100   099   000    -    1
189 High_Fly_Writes         -O-RCK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   060   046   045    -    40 (Min/Max 34/40)
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    0
192 Power-Off_Retract_Count -O--CK   100   100   000    -    255
193 Load_Cycle_Count        -O--CK   096   096   000    -    8232
194 Temperature_Celsius     -O---K   040   054   000    -    40 (0 23 0 0 0)
195 Hardware_ECC_Recovered  -O-RC-   018   006   000    -    68315687
196 Reallocated_Event_Count -O--CK   000   000   000    -    63
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    5
240 Head_Flying_Hours       ------   100   253   000    -    75221 (207 118 0)
241 Total_LBAs_Written      ------   100   253   000    -    642161850457
242 Total_LBAs_Read         ------   100   253   000    -    7805600921027
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL,SL  R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    512  Current Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS      20  Device vendor specific log
0xa2       GPL     VS    4496  Device vendor specific log
0xa8       GPL,SL  VS     129  Device vendor specific log
0xa9       GPL,SL  VS       1  Device vendor specific log
0xb0       GPL     VS    5176  Device vendor specific log
0xbd       GPL     VS     512  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc1       GPL,SL  VS      10  Device vendor specific log
0xc2       GPL,SL  VS      50  Device vendor specific log
0xc3       GPL,SL  VS       8  Device vendor specific log
0xc4       GPL,SL  VS       5  Device vendor specific log
0xd1       GPL,SL  VS       8  Device vendor specific log
0xdf       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
Device Error Count: 3
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 3 [2] occurred at disk power-on lifetime: 10711 hours (446 days + 7 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 53 00 00 00 01 12 b6 cf 9a 00 00  Error: UNC at LBA = 0x112b6cf9a = 4608937882

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 00 08 00 00 01 12 b6 db 70 40 00  2d+03:41:27.702  READ FPDMA QUEUED
  60 00 00 08 00 00 01 12 b6 d3 70 40 00  2d+03:41:27.656  READ FPDMA QUEUED
  60 00 00 08 00 00 01 12 b6 cb 70 40 00  2d+03:41:27.656  READ FPDMA QUEUED
  60 00 00 08 00 00 01 12 b6 c3 70 40 00  2d+03:41:27.616  READ FPDMA QUEUED
  60 00 00 08 00 00 01 12 b6 bb 70 40 00  2d+03:41:27.615  READ FPDMA QUEUED

Error 2 [1] occurred at disk power-on lifetime: 45457 hours (1894 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 01 c8 87 0f 46 00 00  Error: UNC at LBA = 0x1c8870f46 = 7659261766

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  42 00 00 10 00 00 01 c8 87 0f 46 40 00     14:09:21.788  READ VERIFY SECTOR(S) EXT
  42 00 00 00 01 00 01 d1 c0 7c 57 40 00     14:09:21.773  READ VERIFY SECTOR(S) EXT
  61 00 00 00 01 00 01 d1 c0 7c 57 40 00     14:09:21.768  WRITE FPDMA QUEUED
  42 00 00 00 03 00 01 d1 c0 7c 99 40 00     14:09:21.753  READ VERIFY SECTOR(S) EXT
  61 00 00 00 03 00 01 d1 c0 7c 99 40 00     14:09:21.752  WRITE FPDMA QUEUED

Error 1 [0] occurred at disk power-on lifetime: 45457 hours (1894 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 01 c8 87 0f 45 00 00  Error: UNC at LBA = 0x1c8870f45 = 7659261765

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  42 00 00 10 00 00 01 c8 86 70 00 40 00     14:09:18.047  READ VERIFY SECTOR(S) EXT
  42 00 00 10 00 00 01 c8 86 60 00 40 00     14:09:18.023  READ VERIFY SECTOR(S) EXT
  42 00 00 10 00 00 01 c8 86 50 00 40 00     14:09:18.000  READ VERIFY SECTOR(S) EXT
  42 00 00 10 00 00 01 c8 86 40 00 40 00     14:09:17.976  READ VERIFY SECTOR(S) EXT
  42 00 00 10 00 00 01 c8 86 30 00 40 00     14:09:17.935  READ VERIFY SECTOR(S) EXT

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     11477         -
# 2  Short offline       Completed without error       00%     11309         -
# 3  Short offline       Completed without error       00%     11141         -
# 4  Extended offline    Completed without error       00%     11079         -
# 5  Short offline       Completed without error       00%     10989         -
# 6  Short offline       Completed without error       00%     10823         -
# 7  Short offline       Completed without error       00%     10660         -
# 8  Short offline       Completed without error       00%     10657         -
# 9  Short offline       Completed without error       00%     10489         -
#10  Short offline       Completed without error       00%     10321         -
#11  Short offline       Completed without error       00%     10154         -
#12  Extended offline    Completed without error       00%     10036         -
#13  Short offline       Completed without error       00%      9986         -
#14  Short offline       Completed without error       00%      9818         -
#15  Short offline       Completed without error       00%      9650         -
#16  Short offline       Completed without error       00%      9482         -
#17  Short offline       Completed without error       00%      9314         -
#18  Extended offline    Completed without error       00%      9276         -
#19  Short offline       Completed without error       00%      9146         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       522 (0x020a)
Device State:                        Active (0)
Current Temperature:                    38 Celsius
Power Cycle Min/Max Temperature:     34/38 Celsius
Lifetime    Min/Max Temperature:     23/53 Celsius
Under/Over Temperature Limit Count:   0/2

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        10 minutes
Min/Max recommended Temperature:      0/ 0 Celsius
Min/Max Temperature Limit:            0/ 0 Celsius
Temperature History Size (Index):    128 (71)

Index    Estimated Time   Temperature Celsius
  72    2025-04-24 21:30    40  *********************
  73    2025-04-24 21:40    43  ************************
  74    2025-04-24 21:50    45  **************************
  75    2025-04-24 22:00    45  **************************
  76    2025-04-24 22:10    45  **************************
  77    2025-04-24 22:20    46  ***************************
  78    2025-04-24 22:30    46  ***************************
  79    2025-04-24 22:40     ?  -
  80    2025-04-24 22:50    45  **************************
  81    2025-04-24 23:00     ?  -
  82    2025-04-24 23:10    41  **********************
  83    2025-04-24 23:20    43  ************************
  84    2025-04-24 23:30    45  **************************
  85    2025-04-24 23:40    46  ***************************
 ...    ..(  9 skipped).    ..  ***************************
  95    2025-04-25 01:20    46  ***************************
  96    2025-04-25 01:30    45  **************************
 ...    ..(  2 skipped).    ..  **************************
  99    2025-04-25 02:00    45  **************************
 100    2025-04-25 02:10    44  *************************
 ...    ..(  4 skipped).    ..  *************************
 105    2025-04-25 03:00    44  *************************
 106    2025-04-25 03:10    43  ************************
 ...    ..(  2 skipped).    ..  ************************
 109    2025-04-25 03:40    43  ************************
 110    2025-04-25 03:50    42  ***********************
 111    2025-04-25 04:00    42  ***********************
 112    2025-04-25 04:10    43  ************************
 113    2025-04-25 04:20    43  ************************
 114    2025-04-25 04:30    43  ************************
 115    2025-04-25 04:40    44  *************************
 116    2025-04-25 04:50    45  **************************
 ...    ..(  2 skipped).    ..  **************************
 119    2025-04-25 05:20    45  **************************
 120    2025-04-25 05:30    46  ***************************
 121    2025-04-25 05:40    46  ***************************
 122    2025-04-25 05:50    45  **************************
 123    2025-04-25 06:00    45  **************************
 124    2025-04-25 06:10    45  **************************
 125    2025-04-25 06:20    44  *************************
 ...    ..(  4 skipped).    ..  *************************
   2    2025-04-25 07:10    44  *************************
   3    2025-04-25 07:20    45  **************************
 ...    ..(  7 skipped).    ..  **************************
  11    2025-04-25 08:40    45  **************************
  12    2025-04-25 08:50    46  ***************************
 ...    ..( 32 skipped).    ..  ***************************
  45    2025-04-25 14:20    46  ***************************
  46    2025-04-25 14:30     ?  -
  47    2025-04-25 14:40    38  *******************
  48    2025-04-25 14:50     ?  -
  49    2025-04-25 15:00    40  *********************
  50    2025-04-25 15:10     ?  -
  51    2025-04-25 15:20    41  **********************
  52    2025-04-25 15:30     ?  -
  53    2025-04-25 15:40    42  ***********************
  54    2025-04-25 15:50    44  *************************
  55    2025-04-25 16:00     ?  -
  56    2025-04-25 16:10    45  **************************
  57    2025-04-25 16:20     ?  -
  58    2025-04-25 16:30    46  ***************************
  59    2025-04-25 16:40     ?  -
  60    2025-04-25 16:50    47  ****************************
  61    2025-04-25 17:00     ?  -
  62    2025-04-25 17:10    47  ****************************
  63    2025-04-25 17:20     ?  -
  64    2025-04-25 17:30    47  ****************************
  65    2025-04-25 17:40     ?  -
  66    2025-04-25 17:50    28  *********
  67    2025-04-25 18:00     ?  -
  68    2025-04-25 18:10    31  ************
  69    2025-04-25 18:20     ?  -
  70    2025-04-25 18:30    34  ***************
  71    2025-04-25 18:40    37  ******************

SCT Error Recovery Control:
           Read:     80 (8.0 seconds)
          Write:     80 (8.0 seconds)

Device Statistics (GP/SMART Log 0x04) not supported

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2            2  Device-to-host register FISes sent due to a COMRESET
0x0001  2            0  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS

root@truenas[/home/admin]# smartctl -x /dev/sdm
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Constellation ES.3
Device Model:     ST4000NM0033-9ZM170
Serial Number:    Z1Z43PX7
LU WWN Device Id: 5 000c50 066ebacf6
Add. Product Id:  DELL(tm)
Firmware Version: GA0A
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5671
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Apr 25 18:48:39 2025 +08
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (   90) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 497) minutes.
Conveyance self-test routine
recommended polling time:        (   3) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--+  081   063   044    -    138900076
  3 Spin_Up_Time            PO----+  093   088   000    -    0
  4 Start_Stop_Count        -O--CK   099   099   020    -    2024
  5 Reallocated_Sector_Ct   PO--CK+  100   100   010    -    0
  7 Seek_Error_Rate         POSR--   093   060   030    -    1965892773
  9 Power_On_Hours          -O--CK   013   013   000    -    77062
 10 Spin_Retry_Count        PO--C-   100   100   097    -    0
 12 Power_Cycle_Count       -O--CK   100   100   020    -    988
184 End-to-End_Error        -O--CK   100   100   099    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   099   000    -    4295032834
189 High_Fly_Writes         -O-RCK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   058   045   045    Past 42 (Min/Max 36/42)
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    0
192 Power-Off_Retract_Count -O--CK   100   100   000    -    269
193 Load_Cycle_Count        -O--CK   096   096   000    -    8356
194 Temperature_Celsius     -O---K   042   055   000    -    42 (0 22 0 0 0)
195 Hardware_ECC_Recovered  -O-RC-   026   003   000    -    138900076
196 Reallocated_Event_Count -O--CK   000   000   000    -    0
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    1
240 Head_Flying_Hours       ------   100   253   000    -    75126 (5 172 0)
241 Total_LBAs_Written      ------   100   253   000    -    641642242351
242 Total_LBAs_Read         ------   100   253   000    -    7699694319300
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL,SL  R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    512  Current Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS      20  Device vendor specific log
0xa2       GPL     VS    4496  Device vendor specific log
0xa8       GPL,SL  VS     129  Device vendor specific log
0xa9       GPL,SL  VS       1  Device vendor specific log
0xb0       GPL     VS    5176  Device vendor specific log
0xbd       GPL     VS     512  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc1       GPL,SL  VS      10  Device vendor specific log
0xc2       GPL,SL  VS      50  Device vendor specific log
0xc3       GPL,SL  VS       8  Device vendor specific log
0xc4       GPL,SL  VS       5  Device vendor specific log
0xd1       GPL,SL  VS       8  Device vendor specific log
0xdf       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     11382         -
# 2  Short offline       Completed without error       00%     11214         -
# 3  Short offline       Completed without error       00%     11046         -
# 4  Extended offline    Completed without error       00%     10984         -
# 5  Short offline       Completed without error       00%     10894         -
# 6  Short offline       Completed without error       00%     10728         -
# 7  Short offline       Completed without error       00%     10565         -
# 8  Short offline       Completed without error       00%     10562         -
# 9  Short offline       Completed without error       00%     10394         -
#10  Short offline       Completed without error       00%     10226         -
#11  Short offline       Completed without error       00%     10058         -
#12  Extended offline    Completed without error       00%      9943         -
#13  Short offline       Completed without error       00%      9891         -
#14  Short offline       Completed without error       00%      9723         -
#15  Short offline       Completed without error       00%      9555         -
#16  Short offline       Completed without error       00%      9387         -
#17  Short offline       Completed without error       00%      9219         -
#18  Extended offline    Completed without error       00%      9181         -
#19  Short offline       Completed without error       00%      9051         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       522 (0x020a)
Device State:                        Active (0)
Current Temperature:                    40 Celsius
Power Cycle Min/Max Temperature:     36/40 Celsius
Lifetime    Min/Max Temperature:     22/55 Celsius
Under/Over Temperature Limit Count:   0/2

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        10 minutes
Min/Max recommended Temperature:      0/ 0 Celsius
Min/Max Temperature Limit:            0/ 0 Celsius
Temperature History Size (Index):    128 (14)

Index    Estimated Time   Temperature Celsius
  15    2025-04-24 21:30    45  **************************
  16    2025-04-24 21:40    45  **************************
  17    2025-04-24 21:50    45  **************************
  18    2025-04-24 22:00    44  *************************
 ...    ..(  2 skipped).    ..  *************************
  21    2025-04-24 22:30    44  *************************
  22    2025-04-24 22:40    45  **************************
 ...    ..( 10 skipped).    ..  **************************
  33    2025-04-25 00:30    45  **************************
  34    2025-04-25 00:40    44  *************************
  35    2025-04-25 00:50    44  *************************
  36    2025-04-25 01:00    43  ************************
 ...    ..(  5 skipped).    ..  ************************
  42    2025-04-25 02:00    43  ************************
  43    2025-04-25 02:10    42  ***********************
  44    2025-04-25 02:20    43  ************************
  45    2025-04-25 02:30    43  ************************
  46    2025-04-25 02:40    44  *************************
 ...    ..(  4 skipped).    ..  *************************
  51    2025-04-25 03:30    44  *************************
  52    2025-04-25 03:40    45  **************************
 ...    ..(  3 skipped).    ..  **************************
  56    2025-04-25 04:20    45  **************************
  57    2025-04-25 04:30    46  ***************************
 ...    ..(  2 skipped).    ..  ***************************
  60    2025-04-25 05:00    46  ***************************
  61    2025-04-25 05:10    47  ****************************
  62    2025-04-25 05:20    48  *****************************
  63    2025-04-25 05:30    48  *****************************
  64    2025-04-25 05:40    48  *****************************
  65    2025-04-25 05:50     ?  -
  66    2025-04-25 06:00    42  ***********************
  67    2025-04-25 06:10    44  *************************
  68    2025-04-25 06:20    45  **************************
  69    2025-04-25 06:30    46  ***************************
 ...    ..(  2 skipped).    ..  ***************************
  72    2025-04-25 07:00    46  ***************************
  73    2025-04-25 07:10    47  ****************************
 ...    ..(  3 skipped).    ..  ****************************
  77    2025-04-25 07:50    47  ****************************
  78    2025-04-25 08:00    48  *****************************
  79    2025-04-25 08:10    48  *****************************
  80    2025-04-25 08:20    48  *****************************
  81    2025-04-25 08:30    47  ****************************
  82    2025-04-25 08:40    47  ****************************
  83    2025-04-25 08:50    48  *****************************
 ...    ..(  3 skipped).    ..  *****************************
  87    2025-04-25 09:30    48  *****************************
  88    2025-04-25 09:40    47  ****************************
 ...    ..(  2 skipped).    ..  ****************************
  91    2025-04-25 10:10    47  ****************************
  92    2025-04-25 10:20    48  *****************************
  93    2025-04-25 10:30     ?  -
  94    2025-04-25 10:40    31  ************
  95    2025-04-25 10:50     ?  -
  96    2025-04-25 11:00    42  ***********************
  97    2025-04-25 11:10     ?  -
  98    2025-04-25 11:20    45  **************************
  99    2025-04-25 11:30     ?  -
 100    2025-04-25 11:40    44  *************************
 101    2025-04-25 11:50     ?  -
 102    2025-04-25 12:00    44  *************************
 103    2025-04-25 12:10     ?  -
 104    2025-04-25 12:20    46  ***************************
 105    2025-04-25 12:30     ?  -
 106    2025-04-25 12:40    47  ****************************
 107    2025-04-25 12:50    48  *****************************
 108    2025-04-25 13:00     ?  -
 109    2025-04-25 13:10    50  *******************************
 110    2025-04-25 13:20     ?  -
 111    2025-04-25 13:30    45  **************************
 112    2025-04-25 13:40    48  *****************************
 113    2025-04-25 13:50     ?  -
 114    2025-04-25 14:00    50  *******************************
 115    2025-04-25 14:10     ?  -
 116    2025-04-25 14:20    50  *******************************
 117    2025-04-25 14:30     ?  -
 118    2025-04-25 14:40    37  ******************
 119    2025-04-25 14:50     ?  -
 120    2025-04-25 15:00    30  ***********
 121    2025-04-25 15:10     ?  -
 122    2025-04-25 15:20    37  ******************
 123    2025-04-25 15:30     ?  -
 124    2025-04-25 15:40    42  ***********************
 125    2025-04-25 15:50    45  **************************
 126    2025-04-25 16:00     ?  -
 127    2025-04-25 16:10    47  ****************************
   0    2025-04-25 16:20     ?  -
   1    2025-04-25 16:30    48  *****************************
   2    2025-04-25 16:40     ?  -
   3    2025-04-25 16:50    49  ******************************
   4    2025-04-25 17:00     ?  -
   5    2025-04-25 17:10    50  *******************************
   6    2025-04-25 17:20     ?  -
   7    2025-04-25 17:30    50  *******************************
   8    2025-04-25 17:40     ?  -
   9    2025-04-25 17:50    28  *********
  10    2025-04-25 18:00     ?  -
  11    2025-04-25 18:10    32  *************
  12    2025-04-25 18:20     ?  -
  13    2025-04-25 18:30    36  *****************
  14    2025-04-25 18:40    38  *******************

SCT Error Recovery Control:
           Read:     80 (8.0 seconds)
          Write:     80 (8.0 seconds)

Device Statistics (GP/SMART Log 0x04) not supported

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2            3  Device-to-host register FISes sent due to a COMRESET
0x0001  2            0  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS

root@truenas[/home/admin]#

/dev/sde1: LABEL=“ThePool” UUID=“2403027406607053466” UUID_SUB=“8162898467031863576” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTUUID=“ae13e53b-c330-4a3f-877f-3209735dc59d”

/dev/sdm1: LABEL=“ThePool” UUID=“2403027406607053466” UUID_SUB=“3574681597018827698” BLOCK_SIZE=“4096” TYPE=“zfs_member” PARTUUID=“d486c78a-f360-4873-9132-9d2898f6c5e3”

zpool import
mirror-4
ae13e53b-c330-4a3f-877f-3209735dc59d ONLINE
d486c78a-f360-4873-9132-9d2898f6c5e3 ONLINE

9

@HoneyBadger ???

10

Looks like spacemap corruption trying to flush a pending transaction from the ZIL. We may be able to import this with aggressive rollback and/or discarding the ZIL.

Assuming you can hotplug the drives here, boot with enough of them detached to cause it to not import the pool and bring you to a regular webUI/prompt. Then drop to a shell and do the following (using sudo if you need to)

echo 1 > /sys/module/zfs/parameters/zil_replay_disable
echo 1 > /sys/module/zfs/parameters/zfs_recover

Once you’ve done that, hotplug your disks back in, and attempt an import from the CLI with

zpool import -R /mnt -f -o readonly=on ThePool

Assuming it doesn’t panic, do a zpool status to see if it can ID what’s what - from there, if you have the extra space, the safest thing is “back up to another pool and restore” but if you don’t, then we have to potentially look at importing R/W and permanently discarding some transactions with adding the -fFX or -T options to the command-line import.

Note that the zfs_recover and similar tunables will reset on every reboot - so if it does panic trying to do something after you put those in, you’ll have to do the same drive hotplug and enter them from CLI.

1 Like
11

i can see now its mounted at the last step i was performing lastime, trying to remove mirror-4. as im posting this, currently im copying from the pool to a newly created pool. thank you sensei @HoneyBadger and @Protopia for your kind help. i will update more once i have progress on the restore.

root@truenas[/home/admin]# echo 1 > /sys/module/zfs/parameters/zil_replay_disable
root@truenas[/home/admin]# echo 1 > /sys/module/zfs/parameters/zfs_recover
root@truenas[/home/admin]# 
root@truenas[/home/admin]# zpool import -R /mnt -f -o readonly=on ThePool
root@truenas[/home/admin]# 
root@truenas[/home/admin]# zpool status -v
  pool: ThePool
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: resilvered 4.14T in 12:56:09 with 3 errors on Wed Apr 23 17:10:50 2025
remove: Evacuation of mirror in progress since Thu Apr 24 01:05:16 2025
        604G copied out of 3.59T at 3.40M/s, 16.44% done, 257h5m to go
        46.8M memory used for removed device mappings
config:

        NAME                                      STATE     READ WRITE CKSUM
        ThePool                                   DEGRADED     0     0     0
          mirror-3                                ONLINE       0     0     0
            2a2eedf3-947d-44f1-be7a-218837ad9d1f  ONLINE       0     0     0
            ad597db7-b4c0-4032-a811-ba5c8b336248  ONLINE       0     0     0
          mirror-4                                ONLINE       0     0     0  (removing)
            ae13e53b-c330-4a3f-877f-3209735dc59d  ONLINE       0     0     0  (non-allocating)
            d486c78a-f360-4873-9132-9d2898f6c5e3  ONLINE       0     0     0  (non-allocating)
          mirror-5                                ONLINE       0     0     0
            850d4eb8-f5be-497c-93f9-873f84d32ad7  ONLINE       0     0     0
            f42cd54a-c9e7-4484-bd3c-996be41c6292  ONLINE       0     0     0
          mirror-6                                ONLINE       0     0     0
            34beb21d-18bf-4f19-b44f-688f4dab38d8  ONLINE       0     0     0
            9f55ee92-25d9-4069-bd39-b2b9a48a4568  ONLINE       0     0     0
          mirror-7                                ONLINE       0     0     0
            9c13fea5-8486-44d9-9fc5-0f7fec409119  ONLINE       0     0     0
            0bd5a09a-0a9c-402f-b5af-086c9bf08c73  ONLINE       0     0     0
          mirror-8                                DEGRADED     0     0     0
            93afe618-bf4a-49bd-96f8-6bf008a68312  ONLINE       0     0     0
            16447861092670188178                  UNAVAIL      0     0     0  was /dev/disk/by-partuuid/ef26faca-d964-468b-9a87-712fa02ceee0
          mirror-9                                ONLINE       0     0     0
            7868ef19-47e1-4647-af83-39f848b96ada  ONLINE       0     0     0
            3b8401eb-0714-4976-97be-ed7ccf8b3eae  ONLINE       0     0     0
          mirror-10                               ONLINE       0     0     0
            b16aa769-6f78-4757-80cb-b073e56f87ae  ONLINE       0     0     0
            4d255512-b2d5-4da4-b520-1287263d4d44  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        ThePool/Media:<0x9000>
        /mnt/ThePool/Media/TV Series/MasterChef Australia/Season 4/MasterChef Australia_S04E36_Masterclass.mp4
        ThePool/ix-apps/docker:/overlay2/a85c0f94ae28e52f2775a9549caedbd2a5012f09c4776f7472fe152d0b2be519/diff/etc/logrotate.d/nginx-proxy-manager

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:53 with 0 errors on Fri Apr 25 03:45:54 2025
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sdk3      ONLINE       0     0     0

errors: No known data errors
root@truenas[/home/admin]#

created new pool for restoration from the readonly pool. copy is in progress

root@truenas[/mnt/ThePool]# zpool status -v                          
  pool: ThePool
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: resilvered 4.14T in 12:56:09 with 3 errors on Wed Apr 23 17:10:50 2025
remove: Evacuation of mirror in progress since Thu Apr 24 01:05:16 2025
        604G copied out of 3.59T at 3.28M/s, 16.44% done, 266h0m to go
        46.8M memory used for removed device mappings
config:

        NAME                                      STATE     READ WRITE CKSUM
        ThePool                                   DEGRADED     0     0     0
          mirror-3                                ONLINE       0     0     0
            2a2eedf3-947d-44f1-be7a-218837ad9d1f  ONLINE       0     0     0
            ad597db7-b4c0-4032-a811-ba5c8b336248  ONLINE       0     0     0
          mirror-4                                ONLINE       0     0     0  (removing)
            ae13e53b-c330-4a3f-877f-3209735dc59d  ONLINE       0     0     0  (non-allocating)
            d486c78a-f360-4873-9132-9d2898f6c5e3  ONLINE       0     0     0  (non-allocating)
          mirror-5                                ONLINE       0     0     0
            850d4eb8-f5be-497c-93f9-873f84d32ad7  ONLINE       0     0     0
            f42cd54a-c9e7-4484-bd3c-996be41c6292  ONLINE       0     0     0
          mirror-6                                ONLINE       0     0     0
            34beb21d-18bf-4f19-b44f-688f4dab38d8  ONLINE       0     0     0
            9f55ee92-25d9-4069-bd39-b2b9a48a4568  ONLINE       0     0     0
          mirror-7                                ONLINE       0     0     0
            9c13fea5-8486-44d9-9fc5-0f7fec409119  ONLINE       0     0     0
            0bd5a09a-0a9c-402f-b5af-086c9bf08c73  ONLINE       0     0     0
          mirror-8                                DEGRADED     0     0     0
            93afe618-bf4a-49bd-96f8-6bf008a68312  ONLINE       0     0     0
            16447861092670188178                  UNAVAIL      0     0     0  was /dev/disk/by-partuuid/ef26faca-d964-468b-9a87-712fa02ceee0
          mirror-9                                ONLINE       0     0     0
            7868ef19-47e1-4647-af83-39f848b96ada  ONLINE       0     0     0
            3b8401eb-0714-4976-97be-ed7ccf8b3eae  ONLINE       0     0     0
          mirror-10                               ONLINE       0     0     0
            b16aa769-6f78-4757-80cb-b073e56f87ae  ONLINE       0     0     0
            4d255512-b2d5-4da4-b520-1287263d4d44  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        ThePool/Media:<0x9000>
        /mnt/ThePool/Media/TV Series/MasterChef Australia/Season 4/MasterChef Australia_S04E36_Masterclass.mp4
        ThePool/ix-apps/docker:/overlay2/a85c0f94ae28e52f2775a9549caedbd2a5012f09c4776f7472fe152d0b2be519/diff/etc/logrotate.d/nginx-proxy-manager

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:53 with 0 errors on Fri Apr 25 03:45:54 2025
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sdk3      ONLINE       0     0     0

errors: No known data errors

  pool: restoration_pool
 state: ONLINE
config:

        NAME                                      STATE     READ WRITE CKSUM
        restoration_pool                          ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            5d007127-491f-45a4-b3c5-49d399300dab  ONLINE       0     0     0
            72be99eb-7796-4dfb-ab73-3ec978d99b66  ONLINE       0     0     0
            fe77875d-5a39-495e-aadc-f4f3017bd0f5  ONLINE       0     0     0

errors: No known data errors
root@truenas[/mnt/ThePool]#
1 Like