Pool in faulted state, metadata is corrupted, I/O error

TrueNAS General
1

Hi Everyone,

Feeling kind of lost with this issue so any help would be appreciated. I am running this setup for last 10 years in Vmware Esxi 6.7. I recently needed to increase the space as I was at 85% capacity. The system had 4x 6TB HGST disks and I ordered 4 more to add to the pool. I started adding one disk at a time to the pool, which took 10+ hours each.

Anyway, here is my setup and output from various commands I’ve tried to troubleshoot this issue for last week or so:

My setup:
Vmware ESXi 6.7
-4U Supermicro CSE-846BA-R920B Case
-Supermicro X10DRi Server Motherboard
-Dual Intel Xeon E5-2683 v3 2.0 14C CPUs
-128GB ECC RAM (4x 32GB Sticks)
DELL PERC H730 Adapter SAS/SATA/SSD 12GB/S 1GB
-–Dual SSD 512GB RAID1 + DUAL HDD 6TB RAID1 (Local)
LSI 9207-8i SAS HBA (passthrough to TrueNas VM)
–4x HGST 6TB HDDs + 4x Western Digital 6TB HC310

Sudo zpool import

root@NAS04[~]# sudo zpool import

  pool: NAS04_VOL01
    id: 902485976026651984
 state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72
config:

        NAS04_VOL01                               FAULTED  corrupted data
          raidz2-0                                ONLINE
            sda2                                  ONLINE
            sdd2                                  ONLINE
            sdb2                                  ONLINE
            sdc2                                  ONLINE
            2ec5af01-18d9-4a36-93b0-9e87c0ac5221  ONLINE
            2b029f0b-48b0-4346-a18f-4357a9836164  ONLINE
            6cffcfba-8513-4bb4-9962-e17577bca519  ONLINE
            bee5c69c-0c7d-4c93-bdc0-d4c755ca5071  ONLINE


root@NAS04[~]# zdb -l /dev/disk/by-id/wwn-0x5000cca242c84085-part2
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'NAS04_VOL01'
    state: 0
    txg: 54314389
    pool_guid: 902485976026651984
    errata: 0
    hostid: 925905460
    hostname: 'NAS04'
    top_guid: 5829666678129938293
    guid: 10506769506302721900
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 5829666678129938293
        nparity: 2
        raidz_expand_txgs[0]: 54290756
        raidz_expand_txgs[1]: 54301985
        raidz_expand_txgs[2]: 54314233
        metaslab_array: 41
        metaslab_shift: 37
        ashift: 12
        asize: 41993141485568
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 10506769506302721900
            path: '/dev/sdb2'
            DTL: 335
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 18192056191162199573
            path: '/dev/sdd2'
            DTL: 334
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 1364921251938226754
            path: '/dev/sda2'
            DTL: 333
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 12288165796552003558
            path: '/dev/sdc2'
            DTL: 332
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3130116537688070235
            path: '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221'
            whole_disk: 0
            DTL: 595
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 17340071994985452923
            path: '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164'
            whole_disk: 0
            DTL: 12699
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 15081931530192833772
            path: '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519'
            whole_disk: 0
            DTL: 458
            create_txg: 4
        children[7]:
            type: 'disk'
            id: 7
            guid: 13464644164560822966
            path: '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
        org.openzfs:raidz_expansion
    labels = 0 1 2 3


root@NAS04[~]# lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID

NAME   MODEL                          ROTA PTTYPE TYPE   START          SIZE PARTTYPENAME             PARTUUID
sda    HGST HUS726060ALE610              1 gpt    disk         6001175126016
├─sda1                                   1 gpt    part    4096    2147483648 FreeBSD swap             f5022421-3061-11e7-bdf4-0050568776d5
└─sda2                                   1 gpt    part 4198400 5999024930816 FreeBSD ZFS              f51c316d-3061-11e7-bdf4-0050568776d5
sdb    HGST HUS726060ALE610              1 gpt    disk         6001175126016
├─sdb1                                   1 gpt    part    4096    2147483648 FreeBSD swap             f5877b26-3061-11e7-bdf4-0050568776d5
└─sdb2                                   1 gpt    part 4198400 5999024930816 FreeBSD ZFS              f5a0b0b7-3061-11e7-bdf4-0050568776d5
sdc    HGST HUS726060ALE610              1 gpt    disk         6001175126016
├─sdc1                                   1 gpt    part    4096    2147483648 FreeBSD swap             f5b25e93-3061-11e7-bdf4-0050568776d5
└─sdc2                                   1 gpt    part 4198400 5999024930816 FreeBSD ZFS              f5cfd24b-3061-11e7-bdf4-0050568776d5
sdd    HGST HUS726060ALE610              1 gpt    disk         6001175126016
├─sdd1                                   1 gpt    part    4096    2147483648 FreeBSD swap             f54a66a5-3061-11e7-bdf4-0050568776d5
└─sdd2                                   1 gpt    part 4198400 5999024930816 FreeBSD ZFS              f5644496-3061-11e7-bdf4-0050568776d5
sde    HGST HUS726T6TALE6L4              1 gpt    disk         6001175126016
└─sde1                                   1 gpt    part    2048 5999024930816 Solaris /usr & Apple ZFS 2b029f0b-48b0-4346-a18f-4357a9836164
sdf    HGST HUS726T6TALE6L4              1 gpt    disk         6001175126016
└─sdf1                                   1 gpt    part    2048 5999024930816 Solaris /usr & Apple ZFS 2ec5af01-18d9-4a36-93b0-9e87c0ac5221
sdg    HGST HUS726T6TALE6L4              1 gpt    disk         6001175126016
└─sdg1                                   1 gpt    part    2048 5999024930816 Solaris /usr & Apple ZFS 6cffcfba-8513-4bb4-9962-e17577bca519
sdh    HGST HUS726T6TALE6L4              1 gpt    disk         6001175126016
└─sdh1                                   1 gpt    part    2048 5999024930816 Solaris /usr & Apple ZFS bee5c69c-0c7d-4c93-bdc0-d4c755ca5071
sdi    Virtual disk                      1 gpt    disk           42949672960
├─sdi1                                   1 gpt    part      40        524288 BIOS boot                17198475-abd6-11e7-b861-0050568776d5
└─sdi2                                   1 gpt    part    1064   42949103616 FreeBSD ZFS              171edbf3-abd6-11e7-b861-0050568776d5
sdj    Virtual disk                      1 gpt    disk           42949672960
├─sdj1                                   1 gpt    part      40        524288 BIOS boot                1728a45c-abd6-11e7-b861-0050568776d5
└─sdj2                                   1 gpt    part    1064   42949103616 FreeBSD ZFS              172deb80-abd6-11e7-b861-0050568776d5
sr0    VMware Virtual IDE CDROM Drive    0        rom             1073741312


root@NAS04[~]# lspci
00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)
00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)
00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)
00:0f.0 VGA compatible controller: VMware SVGA II Adapter
00:10.0 SCSI storage controller: Broadcom / LSI 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 01)
00:11.0 PCI bridge: VMware PCI bridge (rev 02)
00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.7 PCI bridge: VMware PCI Express Root Port (rev 01)
03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
13:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)


root@NAS04[~]# sudo sas2flash -list

LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

        Adapter Selected is a LSI SAS: SAS2308_2(D1)

        Controller Number              : 0
        Controller                     : SAS2308_2(D1)
        PCI Address                    : 00:03:00:00
        SAS Address                    : 500605b-0-09a1-a350
        NVDATA Version (Default)       : 14.01.00.06
        NVDATA Version (Persistent)    : 14.01.00.06
        Firmware Product ID            : 0x2214 (IT)
        Firmware Version               : 20.00.07.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9207-8i
        BIOS Version                   : 07.39.02.00
        UEFI BSD Version               : 07.27.01.01
        FCODE Version                  : N/A
        Board Name                     : SAS9207-8i
        Board Assembly                 : H3-25412-00H
        Board Tracer Number            : SV44034615

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

I have tried resetting the cables, hard drives, HBA. All connections are solid.

I have about 9TB of data on this pool so any help here would be greatly appreciated. thanks

2

dbgmsg output

root@NAS04[~]# tail -n 200 /proc/spl/kstat/zfs/dbgmsg
1771577303   ffff9b2a83013300 dsl_scan.c:4500:dsl_scan_sync(): finish scan clearing for freenas-boot
1771577303   ffff9b2a83013300 dsl_scan.c:4619:dsl_scan_sync(): scan issuing complete txg 51972860 for freenas-boot
1771577303   ffff9b2a83013300 spa_history.c:306:spa_history_log_sync(): txg 51972860 scan done errors=0
1771577303   ffff9b2a83013300 spa.c:9245:spa_async_request(): spa=freenas-boot async request task=8
1771583892   ffff9b2a55becc80 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771583892   ffff9b2a55becc80 spa.c:6928:spa_tryimport(): spa_tryimport: importing NAS04_VOL01
1771583892   ffff9b2a55becc80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a55becc80-NAS04_VOL01, config trusted): LOADING
1771583892   ffff9b2a55efe600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771583892   ffff9b2a55efe600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771583892   ffff9b2a55efe600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771583892   ffff9b2a55efe600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771583892   ffff9b2a55efe600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771583892   ffff9b2a55efe600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f5cfd24b-3061-11e7-bdf4-0050568776d5': probe done, cant_read=0 cant_write=1
1771583892   ffff9b2a55efe600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771583892   ffff9b2a55efe600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f51c316d-3061-11e7-bdf4-0050568776d5': probe done, cant_read=0 cant_write=1
1771583892   ffff9b2a55becc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': best uberblock found for spa $import-ffff9b2a55becc80-NAS04_VOL01. txg 54314393
1771583892   ffff9b2a55becc80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a55becc80-NAS04_VOL01, config untrusted): using uberblock with txg=54314393
1771583892   ffff9b2a55becc80 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9b2a55becc80-NAS04_VOL01, config untrusted): FAILED: unable to open rootbp in ds                                                                                                                              l_pool_init [error=5]
1771583892   ffff9b2a55becc80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a55becc80-NAS04_VOL01, config untrusted): UNLOADING
1771583898   ffff9b2a83013300 metaslab.c:3721:metaslab_condense(): condensing: txg 51974148, msp[131] ffff9b2a595ad000, vdev id 0, spa freenas-boot, smp size 65832,                                                                                                                               segments 164, forcing condense=FALSE
1771583926   ffff9b2a103e1980 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771583926   ffff9b2a103e1980 spa.c:6928:spa_tryimport(): spa_tryimport: importing NAS04_VOL01
1771583926   ffff9b2a103e1980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a103e1980-NAS04_VOL01, config trusted): LOADING
1771583926   ffff9b2a55373300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771583926   ffff9b2a55373300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771583926   ffff9b2a55373300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771583926   ffff9b2a55373300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f51c316d-3061-11e7-bdf4-0050568776d5': probe done, cant_read=0 cant_write=1
1771583926   ffff9b2a55373300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771583926   ffff9b2a55373300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f5cfd24b-3061-11e7-bdf4-0050568776d5': probe done, cant_read=0 cant_write=1
1771583926   ffff9b2a55373300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771583926   ffff9b2a55373300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771583926   ffff9b2a103e1980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f51c316d-3061-11e7-bdf4-0050568776d5': best uberblock found for spa $import-                                                                                                                              ffff9b2a103e1980-NAS04_VOL01. txg 54314393
1771583926   ffff9b2a103e1980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a103e1980-NAS04_VOL01, config untrusted): using uberblock with txg=54314393
1771583926   ffff9b2a103e1980 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9b2a103e1980-NAS04_VOL01, config untrusted): FAILED: unable to open rootbp in ds                                                                                                                              l_pool_init [error=5]
1771583926   ffff9b2a103e1980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a103e1980-NAS04_VOL01, config untrusted): UNLOADING
1771584088   ffff9b2a55becc80 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771584088   ffff9b2a55becc80 spa.c:6928:spa_tryimport(): spa_tryimport: importing NAS04_VOL01
1771584088   ffff9b2a55becc80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a55becc80-NAS04_VOL01, config trusted): LOADING
1771584088   ffff9b2a819e4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771584088   ffff9b2a819e4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771584088   ffff9b2a819e4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771584088   ffff9b2a819e4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f5cfd24b-3061-11e7-bdf4-0050568776d5': probe done, cant_read=0 cant_write=1
1771584088   ffff9b2a819e4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771584088   ffff9b2a819e4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f51c316d-3061-11e7-bdf4-0050568776d5': probe done, cant_read=0 cant_write=1
1771584088   ffff9b2a819e4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771584088   ffff9b2a819e4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771584089   ffff9b2a55becc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f51c316d-3061-11e7-bdf4-0050568776d5': best uberblock found for spa $import-                                                                                                                              ffff9b2a55becc80-NAS04_VOL01. txg 54314393
1771584089   ffff9b2a55becc80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a55becc80-NAS04_VOL01, config untrusted): using uberblock with txg=54314393
1771584089   ffff9b2a55becc80 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9b2a55becc80-NAS04_VOL01, config untrusted): FAILED: unable to open rootbp in ds                                                                                                                              l_pool_init [error=5]
1771584089   ffff9b2a55becc80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a55becc80-NAS04_VOL01, config untrusted): UNLOADING
1771584740   ffff9b2a1e831980 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771584740   ffff9b2a1e831980 spa.c:6928:spa_tryimport(): spa_tryimport: importing NAS04_VOL01
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a1e831980-NAS04_VOL01, config trusted): LOADING
1771584740   ffff9b2a56759980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a56759980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a56759980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a56759980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a56759980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a56759980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a56759980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a56759980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a1e831980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': best uberblock found for spa $import-ffff9b2a1e831980-NAS04_VOL01. txg 54314393
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a1e831980-NAS04_VOL01, config untrusted): using uberblock with txg=54314393
1771584740   ffff9b2a1e831980 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9b2a1e831980-NAS04_VOL01, config untrusted): FAILED: unable to open rootbp in ds                                                                                                                              l_pool_init [error=5]
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a1e831980-NAS04_VOL01, config untrusted): UNLOADING
1771584740   ffff9b2a1e831980 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771584740   ffff9b2a1e831980 spa.c:6773:spa_import(): spa_import: importing NAS04_VOL01
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config trusted): LOADING
1771584740   ffff9b2a81fd0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a81fd0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a81fd0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a81fd0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a81fd0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a81fd0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a81fd0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a81fd0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a1e831980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': best uberblock found for spa NAS04_VOL01. txg 54314393
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): using uberblock with txg=54314393
1771584740   ffff9b2a1e831980 spa_misc.c:415:spa_load_failed(): spa_load(NAS04_VOL01, config untrusted): FAILED: unable to open rootbp in dsl_pool_init [error=5]
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): UNLOADING
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): spa_load_retry: rewind, max txg: 54314392
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): LOADING
1771584740   ffff9b2a57634c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57634c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57634c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57634c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57634c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57634c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57634c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57634c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a1e831980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': best uberblock found for spa NAS04_VOL01. txg 54314392
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): using uberblock with txg=54314392
1771584740   ffff9b2a1e831980 spa_misc.c:415:spa_load_failed(): spa_load(NAS04_VOL01, config untrusted): FAILED: unable to open rootbp in dsl_pool_init [error=5]
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): UNLOADING
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): spa_load_retry: rewind, max txg: 54314391
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): LOADING
1771584740   ffff9b2a57fd4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57fd4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57fd4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57fd4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57fd4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57fd4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57fd4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a57fd4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a1e831980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': best uberblock found for spa NAS04_VOL01. txg 54314391
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): using uberblock with txg=54314391
1771584740   ffff9b2a1e831980 spa_misc.c:415:spa_load_failed(): spa_load(NAS04_VOL01, config untrusted): FAILED: unable to open rootbp in dsl_pool_init [error=5]
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): UNLOADING
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): spa_load_retry: rewind, max txg: 54314390
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): LOADING
1771584740   ffff9b2a43e40000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a43e40000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a43e40000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a43e40000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a43e40000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a43e40000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a43e40000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a43e40000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771584740   ffff9b2a1e831980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': best uberblock found for spa NAS04_VOL01. txg 54314390
1771584740   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): using uberblock with txg=54314390
1771584740   ffff9b2a1e831980 spa_misc.c:415:spa_load_failed(): spa_load(NAS04_VOL01, config untrusted): FAILED: unable to open rootbp in dsl_pool_init [error=5]
1771584741   ffff9b2a1e831980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): UNLOADING
1771584998   ffff9b2a55c31980 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771584998   ffff9b2a55c31980 spa.c:6925:spa_tryimport(): spa_tryimport: importing NAS04_VOL01, max_txg=54314233
1771584998   ffff9b2a55c31980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a55c31980-NAS04_VOL01, config trusted): LOADING
1771584998   ffff9b2a1f5a9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a1f5a9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a1f5a9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a1f5a9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a1f5a9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a1f5a9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a1f5a9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a1f5a9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a55c31980 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9b2a55c31980-NAS04_VOL01, config untrusted): FAILED: no valid uberblock found
1771584998   ffff9b2a55c31980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a55c31980-NAS04_VOL01, config untrusted): UNLOADING
1771584998   ffff9b2a55c31980 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771584998   ffff9b2a55c31980 spa.c:6775:spa_import(): spa_import: importing NAS04_VOL01, max_txg=54314233 (RECOVERY MODE)
1771584998   ffff9b2a55c31980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config trusted): LOADING
1771584998   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771584998   ffff9b2a55c31980 spa_misc.c:415:spa_load_failed(): spa_load(NAS04_VOL01, config untrusted): FAILED: no valid uberblock found
1771584998   ffff9b2a55c31980 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): UNLOADING
1771585018   ffff9b2a57634c80 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771585018   ffff9b2a57634c80 spa.c:6925:spa_tryimport(): spa_tryimport: importing NAS04_VOL01, max_txg=54301985
1771585018   ffff9b2a57634c80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a57634c80-NAS04_VOL01, config trusted): LOADING
1771585018   ffff9b2abdcfb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2abdcfb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2abdcfb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2abdcfb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2abdcfb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2abdcfb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2abdcfb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2abdcfb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2a57634c80 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9b2a57634c80-NAS04_VOL01, config untrusted): FAILED: no valid uberblock found
1771585018   ffff9b2a57634c80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a57634c80-NAS04_VOL01, config untrusted): UNLOADING
1771585018   ffff9b2a57634c80 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771585018   ffff9b2a57634c80 spa.c:6775:spa_import(): spa_import: importing NAS04_VOL01, max_txg=54301985 (RECOVERY MODE)
1771585018   ffff9b2a57634c80 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config trusted): LOADING
1771585018   ffff9b2a219b8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2a219b8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2a219b8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2a219b8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2a219b8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2a219b8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2a219b8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2a219b8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771585018   ffff9b2a57634c80 spa_misc.c:415:spa_load_failed(): spa_load(NAS04_VOL01, config untrusted): FAILED: no valid uberblock found
1771585018   ffff9b2a57634c80 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): UNLOADING
1771585034   ffff9b2a43e43300 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771585034   ffff9b2a43e43300 spa.c:6925:spa_tryimport(): spa_tryimport: importing NAS04_VOL01, max_txg=54290756
1771585034   ffff9b2a43e43300 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a43e43300-NAS04_VOL01, config trusted): LOADING
1771585034   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a56cd9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a43e43300 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9b2a43e43300-NAS04_VOL01, config untrusted): FAILED: no valid uberblock found
1771585034   ffff9b2a43e43300 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9b2a43e43300-NAS04_VOL01, config untrusted): UNLOADING
1771585034   ffff9b2a43e43300 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1771585034   ffff9b2a43e43300 spa.c:6775:spa_import(): spa_import: importing NAS04_VOL01, max_txg=54290756 (RECOVERY MODE)
1771585034   ffff9b2a43e43300 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config trusted): LOADING
1771585034   ffff9b2a54ebb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a54ebb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a54ebb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a54ebb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a54ebb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a54ebb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a54ebb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a54ebb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1771585034   ffff9b2a43e43300 spa_misc.c:415:spa_load_failed(): spa_load(NAS04_VOL01, config untrusted): FAILED: no valid uberblock found
1771585034   ffff9b2a43e43300 spa_misc.c:429:spa_load_note(): spa_load(NAS04_VOL01, config untrusted): UNLOADING
3

Looks like you were expanding your 4 disk Z2 and not adding a second vdev to the pool is that correct?

4

Correct, just wanted to extend with additional 4 disks.

5

And you’re running TrueNAS as a VM and not baremetal right?

1 Like
6

Your best bet is most likely going to be to rewind to a recent TXG.

There are some good threads on the forums you can read through but tread carefully.

7

Yes, it is a VM.

I will explore txg rewind but not sure where to start with that and how to identify the most recent?

Here is output from some more commands:

root@NAS04[~]# sudo zdb -l /dev/sdj2
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'NAS04_VOL01'
    state: 0
    txg: 54314389
    pool_guid: 902485976026651984
    errata: 0
    hostid: 925905460
    hostname: 'NAS04'
    top_guid: 5829666678129938293
    guid: 10506769506302721900
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 5829666678129938293
        nparity: 2
        raidz_expand_txgs[0]: 54290756
        raidz_expand_txgs[1]: 54301985
        raidz_expand_txgs[2]: 54314233
        metaslab_array: 41
        metaslab_shift: 37
        ashift: 12
        asize: 41993141485568
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 10506769506302721900
            path: '/dev/sdb2'
            DTL: 335
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 18192056191162199573
            path: '/dev/sdd2'
            DTL: 334
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 1364921251938226754
            path: '/dev/sda2'
            DTL: 333
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 12288165796552003558
            path: '/dev/sdc2'
            DTL: 332
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3130116537688070235
            path: '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221'
            whole_disk: 0
            DTL: 595
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 17340071994985452923
            path: '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164'
            whole_disk: 0
            DTL: 12699
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 15081931530192833772
            path: '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519'
            whole_disk: 0
            DTL: 458
            create_txg: 4
        children[7]:
            type: 'disk'
            id: 7
            guid: 13464644164560822966
            path: '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
        org.openzfs:raidz_expansion
    labels = 0 1 2 3


root@NAS04[~]# sudo zdb -l /dev/sda2
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'NAS04_VOL01'
    state: 0
    txg: 54314389
    pool_guid: 902485976026651984
    errata: 0
    hostid: 925905460
    hostname: 'NAS04'
    top_guid: 5829666678129938293
    guid: 1364921251938226754
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 5829666678129938293
        nparity: 2
        raidz_expand_txgs[0]: 54290756
        raidz_expand_txgs[1]: 54301985
        raidz_expand_txgs[2]: 54314233
        metaslab_array: 41
        metaslab_shift: 37
        ashift: 12
        asize: 41993141485568
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 10506769506302721900
            path: '/dev/sdb2'
            DTL: 335
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 18192056191162199573
            path: '/dev/sdd2'
            DTL: 334
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 1364921251938226754
            path: '/dev/sda2'
            DTL: 333
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 12288165796552003558
            path: '/dev/sdc2'
            DTL: 332
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3130116537688070235
            path: '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221'
            whole_disk: 0
            DTL: 595
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 17340071994985452923
            path: '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164'
            whole_disk: 0
            DTL: 12699
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 15081931530192833772
            path: '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519'
            whole_disk: 0
            DTL: 458
            create_txg: 4
        children[7]:
            type: 'disk'
            id: 7
            guid: 13464644164560822966
            path: '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
        org.openzfs:raidz_expansion
    labels = 0 1 2 3



root@NAS04[~]# sudo zdb -l /dev/sdb2
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'NAS04_VOL01'
    state: 0
    txg: 54314389
    pool_guid: 902485976026651984
    errata: 0
    hostid: 925905460
    hostname: 'NAS04'
    top_guid: 5829666678129938293
    guid: 12288165796552003558
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 5829666678129938293
        nparity: 2
        raidz_expand_txgs[0]: 54290756
        raidz_expand_txgs[1]: 54301985
        raidz_expand_txgs[2]: 54314233
        metaslab_array: 41
        metaslab_shift: 37
        ashift: 12
        asize: 41993141485568
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 10506769506302721900
            path: '/dev/sdb2'
            DTL: 335
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 18192056191162199573
            path: '/dev/sdd2'
            DTL: 334
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 1364921251938226754
            path: '/dev/sda2'
            DTL: 333
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 12288165796552003558
            path: '/dev/sdc2'
            DTL: 332
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3130116537688070235
            path: '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221'
            whole_disk: 0
            DTL: 595
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 17340071994985452923
            path: '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164'
            whole_disk: 0
            DTL: 12699
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 15081931530192833772
            path: '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519'
            whole_disk: 0
            DTL: 458
            create_txg: 4
        children[7]:
            type: 'disk'
            id: 7
            guid: 13464644164560822966
            path: '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
        org.openzfs:raidz_expansion
    labels = 0 1 2 3


root@NAS04[~]# sudo zdb -l /dev/sdd2
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'NAS04_VOL01'
    state: 0
    txg: 54314389
    pool_guid: 902485976026651984
    errata: 0
    hostid: 925905460
    hostname: 'NAS04'
    top_guid: 5829666678129938293
    guid: 18192056191162199573
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 5829666678129938293
        nparity: 2
        raidz_expand_txgs[0]: 54290756
        raidz_expand_txgs[1]: 54301985
        raidz_expand_txgs[2]: 54314233
        metaslab_array: 41
        metaslab_shift: 37
        ashift: 12
        asize: 41993141485568
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 10506769506302721900
            path: '/dev/sdb2'
            DTL: 335
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 18192056191162199573
            path: '/dev/sdd2'
            DTL: 334
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 1364921251938226754
            path: '/dev/sda2'
            DTL: 333
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 12288165796552003558
            path: '/dev/sdc2'
            DTL: 332
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3130116537688070235
            path: '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221'
            whole_disk: 0
            DTL: 595
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 17340071994985452923
            path: '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164'
            whole_disk: 0
            DTL: 12699
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 15081931530192833772
            path: '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519'
            whole_disk: 0
            DTL: 458
            create_txg: 4
        children[7]:
            type: 'disk'
            id: 7
            guid: 13464644164560822966
            path: '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
        org.openzfs:raidz_expansion
    labels = 0 1 2 3



root@NAS04[~]# sudo zdb -l /dev/sdg1
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'NAS04_VOL01'
    state: 0
    txg: 54314389
    pool_guid: 902485976026651984
    errata: 0
    hostid: 925905460
    hostname: 'NAS04'
    top_guid: 5829666678129938293
    guid: 13464644164560822966
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 5829666678129938293
        nparity: 2
        raidz_expand_txgs[0]: 54290756
        raidz_expand_txgs[1]: 54301985
        raidz_expand_txgs[2]: 54314233
        metaslab_array: 41
        metaslab_shift: 37
        ashift: 12
        asize: 41993141485568
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 10506769506302721900
            path: '/dev/sdb2'
            DTL: 335
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 18192056191162199573
            path: '/dev/sdd2'
            DTL: 334
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 1364921251938226754
            path: '/dev/sda2'
            DTL: 333
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 12288165796552003558
            path: '/dev/sdc2'
            DTL: 332
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3130116537688070235
            path: '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221'
            whole_disk: 0
            DTL: 595
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 17340071994985452923
            path: '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164'
            whole_disk: 0
            DTL: 12699
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 15081931530192833772
            path: '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519'
            whole_disk: 0
            DTL: 458
            create_txg: 4
        children[7]:
            type: 'disk'
            id: 7
            guid: 13464644164560822966
            path: '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
        org.openzfs:raidz_expansion
    labels = 0 1 2 3


root@NAS04[~]# sudo zdb -l /dev/sdf1
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'NAS04_VOL01'
    state: 0
    txg: 54314389
    pool_guid: 902485976026651984
    errata: 0
    hostid: 925905460
    hostname: 'NAS04'
    top_guid: 5829666678129938293
    guid: 15081931530192833772
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 5829666678129938293
        nparity: 2
        raidz_expand_txgs[0]: 54290756
        raidz_expand_txgs[1]: 54301985
        raidz_expand_txgs[2]: 54314233
        metaslab_array: 41
        metaslab_shift: 37
        ashift: 12
        asize: 41993141485568
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 10506769506302721900
            path: '/dev/sdb2'
            DTL: 335
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 18192056191162199573
            path: '/dev/sdd2'
            DTL: 334
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 1364921251938226754
            path: '/dev/sda2'
            DTL: 333
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 12288165796552003558
            path: '/dev/sdc2'
            DTL: 332
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3130116537688070235
            path: '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221'
            whole_disk: 0
            DTL: 595
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 17340071994985452923
            path: '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164'
            whole_disk: 0
            DTL: 12699
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 15081931530192833772
            path: '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519'
            whole_disk: 0
            DTL: 458
            create_txg: 4
        children[7]:
            type: 'disk'
            id: 7
            guid: 13464644164560822966
            path: '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
        org.openzfs:raidz_expansion
    labels = 0 1 2 3



root@NAS04[~]# sudo zdb -l /dev/sde1
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'NAS04_VOL01'
    state: 0
    txg: 54314389
    pool_guid: 902485976026651984
    errata: 0
    hostid: 925905460
    hostname: 'NAS04'
    top_guid: 5829666678129938293
    guid: 3130116537688070235
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 5829666678129938293
        nparity: 2
        raidz_expand_txgs[0]: 54290756
        raidz_expand_txgs[1]: 54301985
        raidz_expand_txgs[2]: 54314233
        metaslab_array: 41
        metaslab_shift: 37
        ashift: 12
        asize: 41993141485568
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 10506769506302721900
            path: '/dev/sdb2'
            DTL: 335
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 18192056191162199573
            path: '/dev/sdd2'
            DTL: 334
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 1364921251938226754
            path: '/dev/sda2'
            DTL: 333
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 12288165796552003558
            path: '/dev/sdc2'
            DTL: 332
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3130116537688070235
            path: '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221'
            whole_disk: 0
            DTL: 595
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 17340071994985452923
            path: '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164'
            whole_disk: 0
            DTL: 12699
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 15081931530192833772
            path: '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519'
            whole_disk: 0
            DTL: 458
            create_txg: 4
        children[7]:
            type: 'disk'
            id: 7
            guid: 13464644164560822966
            path: '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
        org.openzfs:raidz_expansion
    labels = 0 1 2 3



root@NAS04[~]# sudo zdb -l /dev/sdc1
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'NAS04_VOL01'
    state: 0
    txg: 54314389
    pool_guid: 902485976026651984
    errata: 0
    hostid: 925905460
    hostname: 'NAS04'
    top_guid: 5829666678129938293
    guid: 17340071994985452923
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 5829666678129938293
        nparity: 2
        raidz_expand_txgs[0]: 54290756
        raidz_expand_txgs[1]: 54301985
        raidz_expand_txgs[2]: 54314233
        metaslab_array: 41
        metaslab_shift: 37
        ashift: 12
        asize: 41993141485568
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 10506769506302721900
            path: '/dev/sdb2'
            DTL: 335
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 18192056191162199573
            path: '/dev/sdd2'
            DTL: 334
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 1364921251938226754
            path: '/dev/sda2'
            DTL: 333
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 12288165796552003558
            path: '/dev/sdc2'
            DTL: 332
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3130116537688070235
            path: '/dev/disk/by-partuuid/2ec5af01-18d9-4a36-93b0-9e87c0ac5221'
            whole_disk: 0
            DTL: 595
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 17340071994985452923
            path: '/dev/disk/by-partuuid/2b029f0b-48b0-4346-a18f-4357a9836164'
            whole_disk: 0
            DTL: 12699
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 15081931530192833772
            path: '/dev/disk/by-partuuid/6cffcfba-8513-4bb4-9962-e17577bca519'
            whole_disk: 0
            DTL: 458
            create_txg: 4
        children[7]:
            type: 'disk'
            id: 7
            guid: 13464644164560822966
            path: '/dev/disk/by-partuuid/bee5c69c-0c7d-4c93-bdc0-d4c755ca5071'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
        org.openzfs:raidz_expansion
    labels = 0 1 2 3



root@NAS04[/mnt]# sudo zpool import -fF NAS04_VOL01 -R /mnt -o readonly=on
cannot import 'NAS04_VOL01': I/O error
        Destroy and re-create the pool from
        a backup source.


root@NAS04[/mnt]# sudo zpool import -fm -T 54314389 NAS04_VOL01 -o readonly=on
cannot import 'NAS04_VOL01': one or more devices is currently unavailable

root@NAS04[/]# zpool import -d /dev/disk/by-partuuid

  pool: NAS04_VOL01
    id: 902485976026651984
 state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72
config:

        NAS04_VOL01                               FAULTED  corrupted data
          raidz2-0                                ONLINE
            f51c316d-3061-11e7-bdf4-0050568776d5  ONLINE
            f5644496-3061-11e7-bdf4-0050568776d5  ONLINE
            f5a0b0b7-3061-11e7-bdf4-0050568776d5  ONLINE
            f5cfd24b-3061-11e7-bdf4-0050568776d5  ONLINE
            2ec5af01-18d9-4a36-93b0-9e87c0ac5221  ONLINE
            2b029f0b-48b0-4346-a18f-4357a9836164  ONLINE
            6cffcfba-8513-4bb4-9962-e17577bca519  ONLINE
            bee5c69c-0c7d-4c93-bdc0-d4c755ca5071  ONLINE

I am getting this error on one of the drives (smart tests show no errors, wtf?):

root@NAS04[/]# 2026 Feb 20 10:05:46 NAS04 Device: /dev/sdj [SAT], 24 Currently unreadable (pending) sectors
8

Ok this looks promising root@NAS04[/mnt]# sudo zpool import -fm -T 54314389 NAS04_VOL01 -o readonly=on
cannot import ‘NAS04_VOL01’: one or more devices is currently unavailable

But this not so much root@NAS04[/]# 2026 Feb 20 10:05:46 NAS04 Device: /dev/sdj [SAT], 24 Currently unreadable (pending) sectors

My guess is that this drive has caused the issues during expansion.

If you can get that drive to remap its bad sectors you may be able to get the pool back online.

Personally I’d be tempted to run a long smart test on that drive and once complete try your txg rewind command again.

9

Long smart test shows no errors but will try again.

i attached the drives to windows 10 VM to see if Klennet Recovery can help.

10

Klennet is at 4.8% after 10 hours, not going so great.

Can I use something like Cockpit ZFS Manager to import the pool on a linux VM and off load data?

11

If you’ve gone down that road you might want to let it finish.

If your pool could be imported then yes but currently it can’t so no.

12

Update on this: Klennet does not support extended vdevs so I am back to square one.

I will try hetman recovery software and see if there is any hope!

13

@HoneyBadger question on TXG and Raid-Z Expansion. There are a few listings in the ZDB data with raidz_expand_txgs. I think 4 drives total were being added. Could we try importing to raidz_expand_txgs[2]: 54314233 to get back to, guessing the third drive or is this the four addition?

14

I gave up on Klennet and tried Hetman but that is useless as well. It took 200 hours to scan the ZFS volume but totally different files and directory structure, none of the files are familiar or recoverable.

Any other ideas?

15

tried this as well

root@NAS04[~]# sudo zpool import -fm -T 54314233 NAS04_VOL01 -o readonly=on
cannot import 'NAS04_VOL01': one or more devices is currently unavailable
16

Did you ever try this?

17

not in that order. which txg should i use?

18

54314233 appears to be your most recent one.

Correction 54314389 is current. After the long SMART is complete I’d try that first.

19

What does smartctl -l error /dev/sdj show?

and smartctl -a /dev/sdj

20

smartctl -l error /dev/ output for 8 drives

root@NAS04[~]# smartctl -l error /dev/sde
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.15-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

root@NAS04[~]# smartctl -l error /dev/sdf
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.15-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

root@NAS04[~]# smartctl -l error /dev/sdg
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.15-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

root@NAS04[~]# smartctl -l error /dev/sdh
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.15-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

root@NAS04[~]# smartctl -l error /dev/sdc
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.15-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

root@NAS04[~]# smartctl -l error /dev/sdd
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.15-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

root@NAS04[~]# smartctl -l error /dev/sda
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.15-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

root@NAS04[~]# smartctl -l error /dev/sdb
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.15-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
ATA Error Count: 16 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 16 occurred at disk power-on lifetime: 16170 hours (673 days + 18 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 00 00 00 00 00  Error: UNC at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 18 3e 77 40 00   1d+05:54:18.301  READ FPDMA QUEUED
  60 e8 08 18 42 77 40 00   1d+05:54:15.506  READ FPDMA QUEUED
  60 88 08 48 3b 77 40 00   1d+05:54:15.442  READ FPDMA QUEUED
  60 00 00 48 37 77 40 00   1d+05:54:15.442  READ FPDMA QUEUED
  61 08 28 40 ef a0 40 00   1d+05:54:15.407  WRITE FPDMA QUEUED

Error 15 occurred at disk power-on lifetime: 16155 hours (673 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 00 00 00 00 00  Error: UNC at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 18 08 78 5e 23 40 00      15:01:08.830  READ FPDMA QUEUED
  60 08 08 e8 5b 23 40 00      15:01:05.985  READ FPDMA QUEUED
  60 d8 10 e8 39 23 40 00      15:01:05.984  READ FPDMA QUEUED
  60 20 00 a8 fb 22 40 00      15:01:05.973  READ FPDMA QUEUED
  60 48 08 80 e2 22 40 00      15:01:05.972  READ FPDMA QUEUED

Error 14 occurred at disk power-on lifetime: 16155 hours (673 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 00 00 00 00 00  Error: UNC at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 10 08 20 7d 17 40 00      15:00:51.438  READ FPDMA QUEUED
  60 08 08 b0 7a 17 40 00      15:00:48.515  READ FPDMA QUEUED
  60 08 08 50 78 17 40 00      15:00:48.505  READ FPDMA QUEUED
  60 08 08 40 27 17 40 00      15:00:48.490  READ FPDMA QUEUED
  60 48 10 40 21 17 40 00      15:00:48.489  READ FPDMA QUEUED

Error 13 occurred at disk power-on lifetime: 16155 hours (673 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 00 00 00 00 00  Error: UNC at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 08 10 c8 5a 6f 40 00      14:32:49.245  READ FPDMA QUEUED
  60 08 10 d8 58 6f 40 00      14:32:46.383  READ FPDMA QUEUED
  60 08 10 00 57 6f 40 00      14:32:46.368  READ FPDMA QUEUED
  60 10 08 c0 fb 6d 40 00      14:32:46.361  READ FPDMA QUEUED
  60 08 10 f8 fa 6b 40 00      14:32:46.359  READ FPDMA QUEUED

Error 12 occurred at disk power-on lifetime: 16154 hours (673 days + 2 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 00 00 00 00 00  Error: UNC at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 38 00 80 05 6b 40 00      14:07:53.235  READ FPDMA QUEUED
  60 38 00 b8 03 6b 40 00      14:07:49.977  READ FPDMA QUEUED
  60 20 08 38 fa 6a 40 00      14:07:49.943  READ FPDMA QUEUED
  60 08 00 08 95 1b 40 00      14:07:49.942  READ FPDMA QUEUED
  60 08 08 b0 ee 0b 40 00      14:07:49.940  READ FPDMA QUEUED