230 TB RAIDZ2 Pool Hangs During Normal Operation and Cannot Be Imported After Reboot or Reinstallation

tacioandrade · June 11, 2026, 12:25pm

Hello everyone,

I am facing a critical issue with a TrueNAS SCALE 25.10 system and would like to know if anyone has encountered similar behavior or can help identify the root cause.

Environment

Hardware

Server: Supermicro
CPU: Intel Xeon E5-2620 v4
RAM: 64 GB
HBA: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
Enclosure: Datacom
Drives: 16 × Toshiba MG10SCA20TE (20 TB SAS)
I checked the SMART data of all drives. None of them report any errors, warnings, or abnormal conditions. All drives are less than six months old.

ZFS Configuration

Pool size: approximately 230 TB usable
Pool used: approximately 130 TB
Layout: 2 × RAIDZ2 vdevs
Deduplication: disabled
SLOG: none
L2ARC: none

The system had been running in production for a long time without any apparent issues.

Problem Description

During normal business hours, SMB shares suddenly became unavailable.

Interestingly, the pool itself still appeared accessible locally. Through SSH, I was able to browse datasets and list files normally using commands such as ls, but SMB services stopped responding.

I attempted to restart the SMB service multiple times, but the stop/start operations would remain running indefinitely and never complete.

Since the issue could not be resolved, I rebooted the server. After the reboot, the system became stuck while importing the pool.

To rule out operating system or boot device issues, I performed a clean installation of TrueNAS SCALE on a different boot drive and restored the saved configuration. The behavior remained exactly the same: the pool could no longer be imported in read-write mode.

Recovery Attempts

My first attempt was:

zpool import -f Pool

The import process ran for several hours without completing.

During this time, iostat -xz 1 showed ongoing disk activity, indicating that the import process was still performing work.

While monitoring ZFS debug messages using:

cat /proc/spl/kstat/zfs/dbgmsg | tail -n 10

I repeatedly observed messages referencing the same logical block (blkid=4526), including zio_deadman events and hung I/O operations.

Log excerpt:

root@truenas-202[~]# cat /proc/spl/kstat/zfs/dbgmsg | tail -200
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d440160-f1c1-40ab-9559-d8a4a5c1aa83’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/2bd71fd7-d9ec-4b8c-b9a7-bafbec7d43bc’: probe done, cant_read=0 cant_write=1
1781128992 ffff8938819ecc80 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: best uberblock found for spa $import-ffff8938819ecc80-Pool. txg 4163040
1781128992 ffff8938819ecc80 spa_misc.c:430(): spa_load($import-ffff8938819ecc80-Pool, config untrusted): using uberblock with txg=4163040
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/efa79260-217d-4531-9eb6-8e6e34a5d509’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8667b66c-2a93-4d36-bd86-b1f1902b67d0’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/b3d59dea-2e12-4b9d-9315-9704cb8c1ae1’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/3fa197bc-3de9-4300-9de6-38a3c6536676’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d542a21-ebc9-46ca-962b-b411d000b6e8’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8fcb5470-a498-4c1c-911e-61cb2a106f79’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d0246346-7238-405a-9f15-e1a758d0cfe9’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d56d394f-4b07-422d-8d28-8dd555bc231d’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/18d0e239-9346-4af7-a58c-aefdae87d8c2’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/079ec043-8038-4842-9f58-195cea90cc5d’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/558fac31-bb78-4ebf-b40c-bcb9072bdaad’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/ceda4f91-2b33-488e-aafa-b4263f2f0b4e’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/2bd71fd7-d9ec-4b8c-b9a7-bafbec7d43bc’: probe done, cant_read=0 cant_write=1
1781128992 ffff893890040000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d440160-f1c1-40ab-9559-d8a4a5c1aa83’: probe done, cant_read=0 cant_write=1
1781128992 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Loading checkpoint txg
1781128992 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Loading indirect vdev metadata
1781128992 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Checking feature flags
1781128993 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Loading special MOS directories
1781128993 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Loading properties
1781128993 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Loading AUX vdevs
1781128993 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Loading vdev metadata
1781128993 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Loading dedup tables
1781128993 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Loading BRT
1781128993 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Verifying Log Devices
1781128993 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Verifying pool data
1781128993 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Calculating deflated space
1781128993 ffff8938819ecc80 spa_misc.c:2391(): ‘Pool’ Starting import
1781128993 ffff8938819ecc80 spa.c:9320(): spa=$import-ffff8938819ecc80-Pool async request task=2048
1781128993 ffff8938819ecc80 spa_misc.c:430(): spa_load($import-ffff8938819ecc80-Pool, config trusted): LOADED
1781128993 ffff8938819ecc80 spa_misc.c:430(): spa_load($import-ffff8938819ecc80-Pool, config trusted): UNLOADING
1781129029 ffff893860453300 metaslab.c:1789(): spa allocator: dynamic
1781129029 ffff893860453300 spa.c:6921(): spa_tryimport: importing Pool
1781129029 ffff893860453300 spa_misc.c:430(): spa_load($import-ffff893860453300-Pool, config trusted): LOADING
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/efa79260-217d-4531-9eb6-8e6e34a5d509’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8667b66c-2a93-4d36-bd86-b1f1902b67d0’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/b3d59dea-2e12-4b9d-9315-9704cb8c1ae1’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/079ec043-8038-4842-9f58-195cea90cc5d’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d0246346-7238-405a-9f15-e1a758d0cfe9’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/ceda4f91-2b33-488e-aafa-b4263f2f0b4e’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d56d394f-4b07-422d-8d28-8dd555bc231d’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/2bd71fd7-d9ec-4b8c-b9a7-bafbec7d43bc’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8fcb5470-a498-4c1c-911e-61cb2a106f79’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/3fa197bc-3de9-4300-9de6-38a3c6536676’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/18d0e239-9346-4af7-a58c-aefdae87d8c2’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d542a21-ebc9-46ca-962b-b411d000b6e8’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d440160-f1c1-40ab-9559-d8a4a5c1aa83’: probe done, cant_read=0 cant_write=1
1781129029 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/558fac31-bb78-4ebf-b40c-bcb9072bdaad’: probe done, cant_read=0 cant_write=1
1781129030 ffff893860453300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: best uberblock found for spa $import-ffff893860453300-Pool. txg 4163040
1781129030 ffff893860453300 spa_misc.c:430(): spa_load($import-ffff893860453300-Pool, config untrusted): using uberblock with txg=4163040
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/3fa197bc-3de9-4300-9de6-38a3c6536676’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d0246346-7238-405a-9f15-e1a758d0cfe9’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/2bd71fd7-d9ec-4b8c-b9a7-bafbec7d43bc’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d56d394f-4b07-422d-8d28-8dd555bc231d’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/b3d59dea-2e12-4b9d-9315-9704cb8c1ae1’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d542a21-ebc9-46ca-962b-b411d000b6e8’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8fcb5470-a498-4c1c-911e-61cb2a106f79’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/efa79260-217d-4531-9eb6-8e6e34a5d509’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/079ec043-8038-4842-9f58-195cea90cc5d’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d440160-f1c1-40ab-9559-d8a4a5c1aa83’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/ceda4f91-2b33-488e-aafa-b4263f2f0b4e’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/558fac31-bb78-4ebf-b40c-bcb9072bdaad’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8667b66c-2a93-4d36-bd86-b1f1902b67d0’: probe done, cant_read=0 cant_write=1
1781129030 ffff8938a1726600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/18d0e239-9346-4af7-a58c-aefdae87d8c2’: probe done, cant_read=0 cant_write=1
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Loading checkpoint txg
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Loading indirect vdev metadata
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Checking feature flags
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Loading special MOS directories
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Loading properties
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Loading AUX vdevs
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Loading vdev metadata
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Loading dedup tables
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Loading BRT
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Verifying Log Devices
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Verifying pool data
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Calculating deflated space
1781129030 ffff893860453300 spa_misc.c:2391(): ‘Pool’ Starting import
1781129030 ffff893860453300 spa.c:9320(): spa=$import-ffff893860453300-Pool async request task=2048
1781129030 ffff893860453300 spa_misc.c:430(): spa_load($import-ffff893860453300-Pool, config trusted): LOADED
1781129030 ffff893860453300 spa_misc.c:430(): spa_load($import-ffff893860453300-Pool, config trusted): UNLOADING
1781129061 ffff893860390000 metaslab.c:1789(): spa allocator: dynamic
1781129061 ffff893860390000 spa.c:6921(): spa_tryimport: importing Pool
1781129061 ffff893860390000 spa_misc.c:430(): spa_load($import-ffff893860390000-Pool, config trusted): LOADING
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8fcb5470-a498-4c1c-911e-61cb2a106f79’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/3fa197bc-3de9-4300-9de6-38a3c6536676’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/ceda4f91-2b33-488e-aafa-b4263f2f0b4e’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d56d394f-4b07-422d-8d28-8dd555bc231d’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/558fac31-bb78-4ebf-b40c-bcb9072bdaad’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/2bd71fd7-d9ec-4b8c-b9a7-bafbec7d43bc’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d440160-f1c1-40ab-9559-d8a4a5c1aa83’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d542a21-ebc9-46ca-962b-b411d000b6e8’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8667b66c-2a93-4d36-bd86-b1f1902b67d0’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/efa79260-217d-4531-9eb6-8e6e34a5d509’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/b3d59dea-2e12-4b9d-9315-9704cb8c1ae1’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/18d0e239-9346-4af7-a58c-aefdae87d8c2’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/079ec043-8038-4842-9f58-195cea90cc5d’: probe done, cant_read=0 cant_write=1
1781129061 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d0246346-7238-405a-9f15-e1a758d0cfe9’: probe done, cant_read=0 cant_write=1
1781129062 ffff893860390000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: best uberblock found for spa $import-ffff893860390000-Pool. txg 4163040
1781129062 ffff893860390000 spa_misc.c:430(): spa_load($import-ffff893860390000-Pool, config untrusted): using uberblock with txg=4163040
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/3fa197bc-3de9-4300-9de6-38a3c6536676’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/558fac31-bb78-4ebf-b40c-bcb9072bdaad’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d542a21-ebc9-46ca-962b-b411d000b6e8’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d56d394f-4b07-422d-8d28-8dd555bc231d’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d0246346-7238-405a-9f15-e1a758d0cfe9’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/efa79260-217d-4531-9eb6-8e6e34a5d509’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/b3d59dea-2e12-4b9d-9315-9704cb8c1ae1’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8fcb5470-a498-4c1c-911e-61cb2a106f79’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/2bd71fd7-d9ec-4b8c-b9a7-bafbec7d43bc’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/ceda4f91-2b33-488e-aafa-b4263f2f0b4e’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8667b66c-2a93-4d36-bd86-b1f1902b67d0’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/18d0e239-9346-4af7-a58c-aefdae87d8c2’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/079ec043-8038-4842-9f58-195cea90cc5d’: probe done, cant_read=0 cant_write=1
1781129062 ffff8938a136e600 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d440160-f1c1-40ab-9559-d8a4a5c1aa83’: probe done, cant_read=0 cant_write=1
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading checkpoint txg
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading indirect vdev metadata
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Checking feature flags
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading special MOS directories
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading properties
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading AUX vdevs
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading vdev metadata
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading dedup tables
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading BRT
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Verifying Log Devices
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Verifying pool data
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Calculating deflated space
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Starting import
1781129062 ffff893860390000 spa.c:9320(): spa=$import-ffff893860390000-Pool async request task=2048
1781129062 ffff893860390000 spa_misc.c:430(): spa_load($import-ffff893860390000-Pool, config trusted): LOADED
1781129062 ffff893860390000 spa_misc.c:430(): spa_load($import-ffff893860390000-Pool, config trusted): UNLOADING
1781129062 ffff893860390000 metaslab.c:1789(): spa allocator: dynamic
1781129062 ffff893860390000 spa.c:6766(): spa_import: importing Pool
1781129062 ffff893860390000 spa_misc.c:430(): spa_load(Pool, config trusted): LOADING
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/b3d59dea-2e12-4b9d-9315-9704cb8c1ae1’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/ceda4f91-2b33-488e-aafa-b4263f2f0b4e’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d440160-f1c1-40ab-9559-d8a4a5c1aa83’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/efa79260-217d-4531-9eb6-8e6e34a5d509’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d0246346-7238-405a-9f15-e1a758d0cfe9’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/18d0e239-9346-4af7-a58c-aefdae87d8c2’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/3fa197bc-3de9-4300-9de6-38a3c6536676’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8fcb5470-a498-4c1c-911e-61cb2a106f79’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/079ec043-8038-4842-9f58-195cea90cc5d’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8667b66c-2a93-4d36-bd86-b1f1902b67d0’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d56d394f-4b07-422d-8d28-8dd555bc231d’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/558fac31-bb78-4ebf-b40c-bcb9072bdaad’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/2bd71fd7-d9ec-4b8c-b9a7-bafbec7d43bc’: probe done, cant_read=0 cant_write=1
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d542a21-ebc9-46ca-962b-b411d000b6e8’: probe done, cant_read=0 cant_write=1
1781129062 ffff893860390000 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: best uberblock found for spa Pool. txg 4163040
1781129062 ffff893860390000 spa_misc.c:430(): spa_load(Pool, config untrusted): using uberblock with txg=4163040
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/3fa197bc-3de9-4300-9de6-38a3c6536676’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/558fac31-bb78-4ebf-b40c-bcb9072bdaad’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d542a21-ebc9-46ca-962b-b411d000b6e8’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8667b66c-2a93-4d36-bd86-b1f1902b67d0’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/b3d59dea-2e12-4b9d-9315-9704cb8c1ae1’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/8fcb5470-a498-4c1c-911e-61cb2a106f79’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d56d394f-4b07-422d-8d28-8dd555bc231d’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/ceda4f91-2b33-488e-aafa-b4263f2f0b4e’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/efa79260-217d-4531-9eb6-8e6e34a5d509’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/18d0e239-9346-4af7-a58c-aefdae87d8c2’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/d0246346-7238-405a-9f15-e1a758d0cfe9’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/9d440160-f1c1-40ab-9559-d8a4a5c1aa83’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/2bd71fd7-d9ec-4b8c-b9a7-bafbec7d43bc’: probe done, cant_read=0 cant_write=0
1781129062 ffff893852edb300 vdev.c:184(): disk vdev ‘/dev/disk/by-partuuid/079ec043-8038-4842-9f58-195cea90cc5d’: probe done, cant_read=0 cant_write=0
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading checkpoint txg
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading indirect vdev metadata
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Checking feature flags
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading special MOS directories
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading properties
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading AUX vdevs
1781129062 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading vdev metadata
1781129096 ffff893860390000 spa_misc.c:430(): spa_load(Pool, config trusted): Read 49 log space maps (3620 total blocks - blksz = 131072 bytes) in 9759 ms
1781129096 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading dedup tables
1781129096 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Loading BRT
1781129096 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Verifying Log Devices
1781129096 ffff893858f56600 metaslab.c:2602(): metaslab_load: txg 0, spa Pool, vdev_id 0, ms_id 4350, smp_length 205544, unflushed_allocs 0, unflushed_frees 49152, freed 0, defer 0 + 0, unloaded time 249746 ms, loading_time 52 ms, ms_max_size 14720389120, max size error 14720364544, old_weight 840000000000001, new_weight 840000000000001
1781129096 ffff89389145e600 metaslab.c:2602(): metaslab_load: txg 0, spa Pool, vdev_id 1, ms_id 1617, smp_length 296888, unflushed_allocs 5098967040, unflushed_frees 313737216, freed 0, defer 0 + 0, unloaded time 249710 ms, loading_time 131 ms, ms_max_size 2374193152, max size error 2373959680, old_weight 7c0000000000001, new_weight 7c0000000000001
1781129096 ffff893858f56600 metaslab.c:2602(): metaslab_load: txg 0, spa Pool, vdev_id 1, ms_id 4290, smp_length 205344, unflushed_allocs 49152, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 249829 ms, loading_time 50 ms, ms_max_size 11268026368, max size error 11268026368, old_weight 840000000000001, new_weight 840000000000001
1781129097 ffff8938e8b64c80 metaslab.c:2602(): metaslab_load: txg 0, spa Pool, vdev_id 0, ms_id 142, smp_length 4288, unflushed_allocs 3823607808, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 249949 ms, loading_time 114 ms, ms_max_size 2147405824, max size error 2147405824, old_weight 780000000000001, new_weight 780000000000001
1781129097 ffff8938e8b64c80 metaslab.c:2602(): metaslab_load: txg 0, spa Pool, vdev_id 0, ms_id 194, smp_length 7008, unflushed_allocs 301129728, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 250231 ms, loading_time 26 ms, ms_max_size 3821215744, max size error 3821215744, old_weight 7c0000000000001, new_weight 7c0000000000001
1781129097 ffff8938e8b64c80 metaslab.c:2602(): metaslab_load: txg 0, spa Pool, vdev_id 0, ms_id 263, smp_length 3248, unflushed_allocs 6160883712, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 250310 ms, loading_time 11 ms, ms_max_size 2147332096, max size error 2147332096, old_weight 780000000000002, new_weight 780000000000002
1781129097 ffff8938e8b64c80 metaslab.c:2602(): metaslab_load: txg 0, spa Pool, vdev_id 1, ms_id 1631, smp_length 661240, unflushed_allocs 484638720, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 250651 ms, loading_time 115 ms, ms_max_size 2247553024, max size error 2247553024, old_weight 7c0000000000001, new_weight 7c0000000000001
1781129098 ffff893926471980 metaslab.c:2602(): metaslab_load: txg 0, spa Pool, vdev_id 1, ms_id 4324, smp_length 6712, unflushed_allocs 3804831744, unflushed_frees 12288, freed 0, defer 0 + 0, unloaded time 250806 ms, loading_time 126 ms, ms_max_size 10535833600, max size error 10535821312, old_weight 840000000000001, new_weight 840000000000001
1781129100 ffff893926471980 metaslab.c:2602(): metaslab_load: txg 0, spa Pool, vdev_id 0, ms_id 4312, smp_length 9640, unflushed_allocs 0, unflushed_frees 24576, freed 0, defer 0 + 0, unloaded time 252966 ms, loading_time 16 ms, ms_max_size 17179869184, max size error 17179856896, old_weight 880000000000001, new_weight 880000000000001
1781129109 ffff893860390000 spa_misc.c:2391(): ‘Pool’ Verifying pool data
1781130293 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa276c0 timestamp=895715127216 delta=550499256513 queued=0 io=0 path=/dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c last=1446210870664 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=20480 error=0
1781130293 ffff89389075cc80 zio.c:2371:zio_deadman(): zio_wait waiting for hung I/O to pool ‘Pool’
1781130354 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa204c0 timestamp=895715140704 delta=611943287814 queued=0 io=0 path=/dev/disk/by-partuuid/d0246346-7238-405a-9f15-e1a758d0cfe9 last=1507651131254 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636689920 size=24576 error=0
1781130354 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa20980 timestamp=895715138740 delta=611943366098 queued=0 io=0 path=/dev/disk/by-partuuid/079ec043-8038-4842-9f58-195cea90cc5d last=1507658163157 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636689920 size=24576 error=0
1781130354 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa22140 timestamp=895715136829 delta=611943413515 queued=0 io=0 path=/dev/disk/by-partuuid/18d0e239-9346-4af7-a58c-aefdae87d8c2 last=1507652385992 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636689920 size=24576 error=0
1781130354 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa25a40 timestamp=895715134983 delta=611943487665 queued=0 io=0 path=/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6 last=1507653001723 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=24576 error=0
1781130354 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa23dc0 timestamp=895715132632 delta=611943530589 queued=0 io=0 path=/dev/disk/by-partuuid/efa79260-217d-4531-9eb6-8e6e34a5d509 last=1507658582184 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=20480 error=0
1781130354 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa24740 timestamp=895715130827 delta=611943578611 queued=0 io=0 path=/dev/disk/by-partuuid/8667b66c-2a93-4d36-bd86-b1f1902b67d0 last=1507651107524 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=20480 error=0
1781130354 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa22600 timestamp=895715129073 delta=611943616885 queued=0 io=0 path=/dev/disk/by-partuuid/b3d59dea-2e12-4b9d-9315-9704cb8c1ae1 last=1507658619815 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=20480 error=0
1781130354 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa276c0 timestamp=895715127216 delta=611943652802 queued=0 io=0 path=/dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c last=1507655494413 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=20480 error=0
1781130354 ffff89389075cc80 zio.c:2371:zio_deadman(): zio_wait waiting for hung I/O to pool ‘Pool’
1781130416 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa204c0 timestamp=895715140704 delta=673379692761 queued=0 io=0 path=/dev/disk/by-partuuid/d0246346-7238-405a-9f15-e1a758d0cfe9 last=1569091384467 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636689920 size=24576 error=0
1781130416 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa20980 timestamp=895715138740 delta=673379767982 queued=0 io=0 path=/dev/disk/by-partuuid/079ec043-8038-4842-9f58-195cea90cc5d last=1569091251900 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636689920 size=24576 error=0
1781130416 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa22140 timestamp=895715136829 delta=673379815743 queued=0 io=0 path=/dev/disk/by-partuuid/18d0e239-9346-4af7-a58c-aefdae87d8c2 last=1569094668948 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636689920 size=24576 error=0
1781130416 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa25a40 timestamp=895715134983 delta=673379858423 queued=0 io=0 path=/dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6 last=1569092193425 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=24576 error=0
1781130416 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa23dc0 timestamp=895715132632 delta=673379943907 queued=0 io=0 path=/dev/disk/by-partuuid/efa79260-217d-4531-9eb6-8e6e34a5d509 last=1569092275402 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=20480 error=0
1781130416 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa24740 timestamp=895715130827 delta=673379981742 queued=0 io=0 path=/dev/disk/by-partuuid/8667b66c-2a93-4d36-bd86-b1f1902b67d0 last=1569081617250 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=20480 error=0
1781130416 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa22600 timestamp=895715129073 delta=673380019009 queued=0 io=0 path=/dev/disk/by-partuuid/b3d59dea-2e12-4b9d-9315-9704cb8c1ae1 last=1569090087382 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=20480 error=0
1781130416 ffff89389075cc80 zio.c:2313:zio_deadman_impl(): slow zio[2]: zio=ffff89378aa276c0 timestamp=895715127216 delta=673380059976 queued=0 io=0 path=/dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c last=1569091873164 type=1 priority=0 flags=0x300190 stage=0x200000 pipeline=0x4e00000 pipeline-trace=0x200001 objset=9276 object=0 level=18446744073709551614 blkid=4525 offset=9341636694016 size=20480 error=0
1781130416 ffff89389075cc80 zio.c:2371:zio_deadman(): zio_wait waiting for hung I/O to pool ‘Pool’

The logs appear to indicate that the import process becomes stuck while attempting to access the same block during metadata reads.

I also attempted to import the pool using the following recovery-related settings:

echo 0 > /sys/module/zfs/parameters/zfs_bclone_enabled
echo 1 > /sys/module/zfs/parameters/zfs_recover
echo 1 > /sys/module/zfs/parameters/zil_replay_disable

zpool import -F -f Pool

Even with these settings, the process remained indefinitely in D state (uninterruptible sleep) with no observable progress.

Current Status

After performing a forced shutdown, I was able to import the pool only in read-only mode:

zpool import -o readonly=on -R /mnt -f Pool

In this state:

The pool imports successfully.
All datasets are accessible.
I can browse directories and access files normally.
No obvious errors occur during the read-only import process.

However:

The pool cannot be imported in read-write mode.
The behavior persists after a complete TrueNAS reinstallation.
All drives continue to be detected normally by the system.

Questions

Has anyone encountered a situation where a pool imports successfully in read-only mode but completely hangs when imported read-write?
Are there any additional diagnostic procedures I should perform before considering more invasive recovery options?
Do the repeated events involving the same blkid=4526 suggest metadata corruption, hardware issues, or a known OpenZFS bug?
Is there any way to identify exactly which object or metadata structure is associated with this block?

My primary concern at this point is understanding the root cause of the issue. Even if I end up migrating the data and recreating the pool, I would like to avoid having the same problem occur again in the future.

Below are some relevant logs that may help diagnose the problem and identify the root cause:

# sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 16.00.00.00 (2017.04.26)
Copyright (c) 2009-2017 Avago Technologies. All rights reserved.

Read configuration has been initiated for controller 0

Controller information

Controller type                         : SAS3008
BIOS version                            : 8.37.00.00
Firmware version                        : 16.00.10.00
Channel description                     : 1 Serial Attached SCSI
Initiator ID                            : 0
Maximum physical devices                : 1023
Concurrent commands supported           : 9664
Slot                                    : 1
Segment                                 : 0
Bus                                     : 8
Device                                  : 0
Function                                : 0
RAID Support                            : No

IR Volume information




Physical device information

Initiator at ID #0

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 1
SAS Address                             : 5000039-d-982b-6b6f
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : X4M0A01LF3XJ
Unit Serial No(VPD)                     : X4M0A01LF3XJ
GUID                                    : 5000039d982b6b6d
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 2
SAS Address                             : 5000039-d-c801-1367
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : 1550A002F3XJ
Unit Serial No(VPD)                     : 1550A002F3XJ
GUID                                    : 5000039dc8011365
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 3
SAS Address                             : 5000039-d-b839-242b
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4L0A00YF3XJ
Unit Serial No(VPD)                     : Z4L0A00YF3XJ
GUID                                    : 5000039db8392429
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 4
SAS Address                             : 5000039-d-c801-71c7
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : 1550A00VF3XJ
Unit Serial No(VPD)                     : 1550A00VF3XJ
GUID                                    : 5000039dc80171c5
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 5
SAS Address                             : 5000039-d-b822-368f
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4D0A0QDF3XJ
Unit Serial No(VPD)                     : Z4D0A0QDF3XJ
GUID                                    : 5000039db822368d
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 6
SAS Address                             : 5000039-d-c801-3623
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : 1550A00AF3XJ
Unit Serial No(VPD)                     : 1550A00AF3XJ
GUID                                    : 5000039dc8013621
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 7
SAS Address                             : 5000039-d-b822-36d3
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4D0A0QMF3XJ
Unit Serial No(VPD)                     : Z4D0A0QMF3XJ
GUID                                    : 5000039db82236d1
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 8
SAS Address                             : 5000039-d-b822-386f
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4D0A0TCF3XJ
Unit Serial No(VPD)                     : Z4D0A0TCF3XJ
GUID                                    : 5000039db822386d
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 9
SAS Address                             : 5000039-d-b822-1fbb
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4D0A0DDF3XJ
Unit Serial No(VPD)                     : Z4D0A0DDF3XJ
GUID                                    : 5000039db8221fb9
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 10
SAS Address                             : 5000039-d-b822-36ef
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4D0A0QVF3XJ
Unit Serial No(VPD)                     : Z4D0A0QVF3XJ
GUID                                    : 5000039db82236ed
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 11
SAS Address                             : 5000039-d-982a-d9db
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : X4M0A00GF3XJ
Unit Serial No(VPD)                     : X4M0A00GF3XJ
GUID                                    : 5000039d982ad9d9
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 12
SAS Address                             : 5000039-d-b822-33e3
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4D0A0LNF3XJ
Unit Serial No(VPD)                     : Z4D0A0LNF3XJ
GUID                                    : 5000039db82233e1
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 13
SAS Address                             : 5000039-d-b821-34b3
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4D0A00HF3XJ
Unit Serial No(VPD)                     : Z4D0A00HF3XJ
GUID                                    : 5000039db82134b1
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 14
SAS Address                             : 5000039-d-b822-2e17
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4D0A0JTF3XJ
Unit Serial No(VPD)                     : Z4D0A0JTF3XJ
GUID                                    : 5000039db8222e15
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 15
SAS Address                             : 5000039-d-b821-7bcf
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4D0A00MF3XJ
Unit Serial No(VPD)                     : Z4D0A00MF3XJ
GUID                                    : 5000039db8217bcd
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Hard disk
Enclosure #                             : 2
Slot #                                  : 16
SAS Address                             : 5000039-d-b822-3843
State                                   : Ready (RDY)
Size (in MB)/(in sectors)               : 19074047/39063650303
Manufacturer                            : TOSHIBA
Model Number                            : MG10SCA20TE
Firmware Revision                       : 0102
Serial No                               : Z4D0A0T7F3XJ
Unit Serial No(VPD)                     : Z4D0A0T7F3XJ
GUID                                    : 5000039db8223841
Protocol                                : SAS
Drive Type                              : SAS_HDD

Device is a Enclosure services device
Enclosure #                             : 2
Slot #                                  : 25
SAS Address                             : 50015b2-3-6900-193d
State                                   : Standby (SBY)
Manufacturer                            : AIC
Model Number                            : J4024-04 P1
Firmware Revision                       : 0c3b
Serial No                               : 50015B236900193F
Unit Serial No(VPD)                     : 50015B236900193F
GUID                                    : N/A
Protocol                                : SAS
Device Type                             : Enclosure services device

Enclosure information

Enclosure#                              : 1
Logical ID                              : 500605b0:0a076440
Numslots                                : 8
StartSlot                               : 0
Enclosure#                              : 2
Logical ID                              : 50015b23:68ff803f
Numslots                                : 25
StartSlot                               : 1

SAS3IRCU: Command DISPLAY Completed Successfully.
SAS3IRCU: Utility Completed Successfully.

Any guidance or recommendations would be greatly appreciated.

Farout · June 11, 2026, 2:28pm

I would not try to force import the Pool with the -f or -F flag until the more knowledgable users here chime in.

~~In the meantime, your SAS controller seems to be in IR mode instead of IT mode. Which is not great but also not terrible.~~

Check with
sas2flash -list

And post the output of

sudo zpool status -v while imported as readonly.

Is this bare metal or virtualized ?

tacioandrade · June 11, 2026, 2:52pm

That’s right! I already tried that, because before posting here I had tried practically everything (during desperation amidst the pressure we used what we knew) and it didn’t work.

# sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

        Controller Number              : 0
        Controller                     : SAS3008(C0)
        PCI Address                    : 00:08:00:00
        SAS Address                    : 500605b-0-0a07-6440
        NVDATA Version (Default)       : 0e.01.00.08
        NVDATA Version (Persistent)    : 0e.01.00.08
        Firmware Product ID            : 0x2221 (IT)
        Firmware Version               : 16.00.10.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9300-8e
        BIOS Version                   : 08.37.00.00
        UEFI BSD Version               : N/A
        FCODE Version                  : N/A
        Board Name                     : SAS9300-8e
        Board Assembly                 : N/A
        Board Tracer Number            : N/A

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.

zpool status -v Pool

pool: Pool
state: ONLINE
scan: scrub repaired 0B in 1 days 13:51:37 with 0 errors on Mon May 25 13:51:40 2026
config:

    NAME                                      STATE     READ WRITE CKSUM
    Pool                                     ONLINE       0     0     0
      raidz2-0                                ONLINE       0     0     0
        5e8f013c-1ea6-4e31-ba74-eca0de2983e6  ONLINE       0     0     0
        efa79260-217d-4531-9eb6-8e6e34a5d509  ONLINE       0     0     0
        8667b66c-2a93-4d36-bd86-b1f1902b67d0  ONLINE       0     0     0
        b3d59dea-2e12-4b9d-9315-9704cb8c1ae1  ONLINE       0     0     0
        f407220e-1532-4380-86e6-abfa62b6301c  ONLINE       0     0     0
        d0246346-7238-405a-9f15-e1a758d0cfe9  ONLINE       0     0     0
        079ec043-8038-4842-9f58-195cea90cc5d  ONLINE       0     0     0
        18d0e239-9346-4af7-a58c-aefdae87d8c2  ONLINE       0     0     0
      raidz2-1                                ONLINE       0     0     0
        3fa197bc-3de9-4300-9de6-38a3c6536676  ONLINE       0     0     0
        9d542a21-ebc9-46ca-962b-b411d000b6e8  ONLINE       0     0     0
        558fac31-bb78-4ebf-b40c-bcb9072bdaad  ONLINE       0     0     0
        d56d394f-4b07-422d-8d28-8dd555bc231d  ONLINE       0     0     0
        8fcb5470-a498-4c1c-911e-61cb2a106f79  ONLINE       0     0     0
        2bd71fd7-d9ec-4b8c-b9a7-bafbec7d43bc  ONLINE       0     0     0
        ceda4f91-2b33-488e-aafa-b4263f2f0b4e  ONLINE       0     0     0
        9d440160-f1c1-40ab-9559-d8a4a5c1aa83  ONLINE       0     0     0


errors: No known data errors

Bare metal

Johnny_Fartpants · June 11, 2026, 3:03pm

Just to confirm you have a separate Server Head externally connected to a JBOD right?

If so what’s the make and model of the JBOD and how is it currently connected, single SAS cable, dual (wide-port or multipath) etc.

My immediate guess would be a connection issue to the JBOD or the JBOD itself. Do you have another chassis you could move the drives over to to test?

1781130354  zio=ffff89378aa204c0  /dev/disk/by-partuuid/d0246346-7238-405a-9f15-e1a758d0cfe9   blkid=4525 offset=9341636689920 size=24576
1781130354  zio=ffff89378aa20980  /dev/disk/by-partuuid/079ec043-8038-4842-9f58-195cea90cc5d   blkid=4525 offset=9341636689920 size=24576
1781130354  zio=ffff89378aa22140  /dev/disk/by-partuuid/18d0e239-9346-4af7-a58c-aefdae87d8c2   blkid=4525 offset=9341636689920 size=24576
1781130354  zio=ffff89378aa25a40  /dev/disk/by-partuuid/5e8f013c-1ea6-4e31-ba74-eca0de2983e6   blkid=4525 offset=9341636694016 size=24576
1781130354  zio=ffff89378aa23dc0  /dev/disk/by-partuuid/efa79260-217d-4531-9eb6-8e6e34a5d509   blkid=4525 offset=9341636694016 size=20480
1781130354  zio=ffff89378aa24740  /dev/disk/by-partuuid/8667b66c-2a93-4d36-bd86-b1f1902b67d0   blkid=4525 offset=9341636694016 size=20480
1781130354  zio=ffff89378aa22600  /dev/disk/by-partuuid/b3d59dea-2e12-4b9d-9315-9704cb8c1ae1   blkid=4525 offset=9341636694016 size=20480
1781130354  zio=ffff89378aa276c0  /dev/disk/by-partuuid/f407220e-1532-4380-86e6-abfa62b6301c   blkid=4525 offset=9341636694016 size=20480
1781130354   ffff89389075cc80 zio.c:2371:zio_deadman(): zio_wait waiting for hung I/O to pool 'Pool'

Interesting looks like 8 of your disks (one vdev I would guess) are having issues. Unlikely to be the disks themselves so I would say connectivity.

tacioandrade · June 11, 2026, 3:20pm

Exactly! It’s a Supermicro server with a Datacom Enclosure/JBOD.

Although both have two Mini SAS ports, I ended up connecting only one, after some friends who use TrueNAS Scale recommended using only one instead of two with multipath because they had problems with that before.

However, I ruled out a problem with the JBOD/Enclosure/SAS cable, because the IOPS when using iostat -xz 1 were very good and the errors during import were jumping from disk to disk, not a specific disk.

I don’t have one, and to make matters worse, the disks are SAS, so even a small makeshift solution to check if it would matter by connecting directly to a server with 2 HBA cards + multiple power supplies ends up not being possible.

Arwen · June 11, 2026, 10:15pm

Two things:

Is your memory ECC?
And if not, have you run a few hours of memory test recently?
You could check the ZFS write transaction group number for each member disk. See below. They should be identical. If not, that is a problem. While this is unlikely, it is reasonably easy and fast to do.

To get the ZFS TXG for this pool’s member devices, first get the raw device path. Either of these 2 should do it, (with the first only working if you imported the pool).

zpool status -L Pool
lsblk -o NAME,MODEL,SERIAL,LABEL,UUID,PARTUUID,TYPE

Next, run this command against each device:

zdb -l /dev/DISK_PART | grep txg | head -1

As I said, they should be identical, (and probably are). But, we have seen ZFS pool import problems where there are differences due to hardware faults, where some disks end up not getting written to.

Topic		Replies	Views
ZFS import crashing system TrueNAS General SCALE , Hardware , Import-problem	44	333	November 20, 2025
Pool in faulted state, metadata is corrupted, I/O error TrueNAS General Import-problem	34	303	March 7, 2026
Cannot Import Pool to new system after hardware failure TrueNAS General	29	400	November 24, 2025
Help recovering pool (if possible) TrueNAS General SCALE	26	208	April 30, 2025
TrueNAS stuck with some services after a disk has been replaced TrueNAS General SCALE , ZFS	37	403	June 20, 2025

230 TB RAIDZ2 Pool Hangs During Normal Operation and Cannot Be Imported After Reboot or Reinstallation

Environment

Problem Description

Current Status

Questions

Related topics