I agree with @Protopia, option 1 seems best because you have a RAID-Z2 vDev. Removing 1 disk does not remove all redundancy.
After the pool is imported, you can replace the missing disk.
option 1 didn’t work:
admin@truenas[~]$ sudo zpool import -R /mnt -f tank
[sudo] password for admin:
cannot import 'tank': one or more devices is currently unavailable
Can you run the following commands again pls so we can check that the correct disk was removed:
lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAMEsudo zpool import
and then:
sudo zdb -l /dev/sdXNfor each of the partitions in the tank pool that are still installed…
Thanks.
1 Like
Sure!!!
admin@truenas[~]$ lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME
NAME LABEL MAJ:MIN TRAN ROTA ZONED VENDOR MODEL SERIAL PARTUUID START SIZE PARTTYPENAME
loop1 7:1 0 none 383078400
sda 8:0 sata 0 none ATA MTFDDAK256MAM-1K12 14080C02410B 256060514304
├─sda1 8:1 0 none 7bce92bb-c57e-48a7-90a2-18316f48df4d 4096 1048576 BIOS boot
├─sda2 EFI 8:2 0 none 076662d1-13a3-4d32-9cd3-0b39f25bd7eb 6144 536870912 EFI System
├─sda3 boot-pool 8:3 0 none 93d10795-bf3b-4fef-8966-8baa5df3fc65 34609152 238340611584 Solaris /usr & Apple ZFS
└─sda4 8:4 0 none 21f56f6c-2a4c-4fe0-85f2-1ddbcbdd0696 1054720 17179869184 Linux swap
sdb 8:16 sata 1 none ATA ST4000NT001-3M2101 WX11JZB9 4000787030016
└─sdb1 tank 8:17 1 none 58a99133-4ca3-4ae2-afc3-f88ad96f3d68 2048 3998639460352 Solaris /usr & Apple ZFS
sdc 8:32 sata 1 none ATA ST4000VN006-3CW104 ZW601MD7 4000787030016
├─sdc1 8:33 1 none 4f2144fc-2192-11ef-8ffc-bc2411747b27 128 2147483648 FreeBSD swap
└─sdc2 tank 8:34 1 none 4f7421bf-2192-11ef-8ffc-bc2411747b27 4194432 3998639460352 FreeBSD ZFS
sdd 8:48 sata 1 none ATA ST4000VN006-3CW104 WW60CTGN 4000787030016
├─sdd1 8:49 1 none 4f0242d0-2192-11ef-8ffc-bc2411747b27 128 2147483648 FreeBSD swap
└─sdd2 tank 8:50 1 none 4f584c22-2192-11ef-8ffc-bc2411747b27 4194432 3998639460352 FreeBSD ZFS
sde 8:64 sata 1 none ATA ST8000VN002-2ZM188 ZPV0G93G 8001563222016
└─sde1 datos 8:65 1 none 8cfc82bf-1706-4c57-a307-c796ed8216c7 4096 8001560773120 Solaris /usr & Apple ZFS
nvme0n1 259:0 nvme 0 none Lexar SSD NM620 512GB MJN315Q003738 512110190592
├─nvme0n1p1 259:1 nvme 0 none 30b828ac-2234-11ef-8ffc-bc2411747b27 128 2147483648 FreeBSD swap
└─nvme0n1p2 apps 259:2 nvme 0 none 30bb9738-2234-11ef-8ffc-bc2411747b27 4194432 509962620928 FreeBSD ZFS
admin@truenas[~]$ sudo zpool import
[sudo] password for admin:
pool: tank
id: 13400593935322567235
state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:
tank DEGRADED
raidz2-0 DEGRADED
4f584c22-2192-11ef-8ffc-bc2411747b27 ONLINE
4f7421bf-2192-11ef-8ffc-bc2411747b27 ONLINE
4f4aa40e-2192-11ef-8ffc-bc2411747b27 UNAVAIL
58a99133-4ca3-4ae2-afc3-f88ad96f3d68 ONLINE
admin@truenas[~]$ sudo zdb -l /dev/sdb1
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'tank'
state: 0
txg: 6606762
pool_guid: 13400593935322567235
errata: 0
hostid: 590394936
hostname: 'truenas'
top_guid: 15550163770109457767
guid: 4243709192505670205
vdev_children: 1
vdev_tree:
type: 'raidz'
id: 0
guid: 15550163770109457767
nparity: 2
metaslab_array: 65
metaslab_shift: 34
ashift: 12
asize: 15994538950656
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 9392911267891898184
path: '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27'
DTL: 73904
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 5149461304993027350
path: '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27'
DTL: 73903
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 15599814676261111453
path: '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27'
DTL: 73902
create_txg: 4
removed: 1
children[3]:
type: 'disk'
id: 3
guid: 4243709192505670205
path: '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68'
whole_disk: 0
DTL: 771
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 0 1 2 3
admin@truenas[~]$ sudo zdb -l /dev/sdc1
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3
admin@truenas[~]$ sudo zdb -l /dev/sdc2
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'tank'
state: 0
txg: 6606762
pool_guid: 13400593935322567235
errata: 0
hostid: 590394936
hostname: 'truenas'
top_guid: 15550163770109457767
guid: 5149461304993027350
vdev_children: 1
vdev_tree:
type: 'raidz'
id: 0
guid: 15550163770109457767
nparity: 2
metaslab_array: 65
metaslab_shift: 34
ashift: 12
asize: 15994538950656
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 9392911267891898184
path: '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27'
DTL: 73904
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 5149461304993027350
path: '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27'
DTL: 73903
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 15599814676261111453
path: '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27'
DTL: 73902
create_txg: 4
removed: 1
children[3]:
type: 'disk'
id: 3
guid: 4243709192505670205
path: '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68'
whole_disk: 0
DTL: 771
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 0 1 2 3
admin@truenas[~]$ sudo zdb -l /dev/sdd1
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3
admin@truenas[~]$ sudo zdb -l /dev/sdd2
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'tank'
state: 0
txg: 6606762
pool_guid: 13400593935322567235
errata: 0
hostid: 590394936
hostname: 'truenas'
top_guid: 15550163770109457767
guid: 9392911267891898184
vdev_children: 1
vdev_tree:
type: 'raidz'
id: 0
guid: 15550163770109457767
nparity: 2
metaslab_array: 65
metaslab_shift: 34
ashift: 12
asize: 15994538950656
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 9392911267891898184
path: '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27'
DTL: 73904
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 5149461304993027350
path: '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27'
DTL: 73903
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 15599814676261111453
path: '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27'
DTL: 73902
create_txg: 4
removed: 1
children[3]:
type: 'disk'
id: 3
guid: 4243709192505670205
path: '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68'
whole_disk: 0
DTL: 771
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 0 1 2 3
Ok now we have confirmed that the correct drives are still installed, let’s try to do a force import specifying the 3 drives explicitly:
sudo zpool import -R /mnt -f -d /dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27 -d /dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27 -d /dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68 tank
If that doesn’t work then we should start to test (but not do) an import with -Fn:
sudo zpool import -R /mnt -fFn tank
and see what the output from this is, but either way after that we will need input from HoneyBadger I think.
2 Likes
admin@truenas[~]$ sudo zpool import -R /mnt -f -d /dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27 -d /dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27 -d /dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68 tank
[sudo] password for admin:
cannot import 'tank': one or more devices is currently unavailable
admin@truenas[~]$ sudo zpool import -R /mnt -fFn tank
admin@truenas[~]$
Ok - well the bad news is that it is disappointing that nothing we seem to be able to do allows us to import the pool in the consistent state it should be in.
But the good news is that it seems like if we try to import the pool with -fF then it should import - but this means that ZFS will roll back one or more TXGs and @Jorge_Fernandez might lose data.
Before we do this, I think that we should ask @HoneyBadger to review this thread and see what he thinks.
2 Likes
Let’s see the contents of the /proc/spl/kstat/zfs/dbgmsg file. It’s long, so the last 200 lines or so should be enough.
tail -n 200 /proc/spl/kstat/zfs/dbgmsg
here you go…
admin@truenas[/proc/spl/kstat/zfs]$ sudo tail -n 200 /proc/spl/kstat/zfs/dbgmsg
[sudo] password for admin:
1751468707 ffff9621cf40ce00 spa.c:9245:spa_async_request(): spa=$import-ffff9621cf40ce00-tank async request task=2048
1751468707 ffff9621cf40ce00 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9621cf40ce00-tank, config trusted): LOADED
1751468707 ffff9621cf40ce00 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9621cf40ce00-tank, config trusted): UNLOADING
1751472133 ffff9621b4bb3400 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1751472133 ffff9621b4bb3400 spa.c:6928:spa_tryimport(): spa_tryimport: importing tank
1751472133 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9621b4bb3400-tank, config trusted): LOADING
1751472133 ffff9621232c3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472133 ffff9621232c3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751472133 ffff9621232c3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472134 ffff9627c298b400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1013238205/1000000000
1751472134 ffff9621b4bb3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': best uberblock found for spa $import-ffff9621b4bb3400-tank. txg 6608975
1751472134 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9621b4bb3400-tank, config untrusted): using uberblock with txg=6608975
1751472134 ffff9621232c3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751472134 ffff9621232c3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472134 ffff9621232c3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472135 ffff9621cfaf9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1010351749/1000000000
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1751472135 ffff9621cfdb8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a9893000
1751472135 ffff96211c0e0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a988d000
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Calculating deflated space
1751472135 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Starting import
1751472135 ffff9621b4bb3400 spa.c:9245:spa_async_request(): spa=$import-ffff9621b4bb3400-tank async request task=2048
1751472135 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9621b4bb3400-tank, config trusted): LOADED
1751472135 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9621b4bb3400-tank, config trusted): UNLOADING
1751472135 ffff9621b4bb3400 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1751472135 ffff9621b4bb3400 spa.c:6773:spa_import(): spa_import: importing tank
1751472135 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): LOADING
1751472135 ffff9621ce8d4e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472135 ffff9621ce8d4e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472135 ffff9621ce8d4e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751472136 ffff9621492e1a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1009598506/1000000000
1751472136 ffff9621b4bb3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': best uberblock found for spa tank. txg 6608975
1751472136 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=6608975
1751472136 ffff9621108f9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=2459/1000000000
1751472136 ffff9621ce8d4e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=0
1751472136 ffff9621ce8d4e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=0
1751472136 ffff9621ce8d4e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=0
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1751472136 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): Read 7 log space maps (7 total blocks - blksz = 131072 bytes) in 0 ms
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1751472136 ffff9621b4bb3400 spa_misc.c:415:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_check_logs failed
1751472136 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): UNLOADING
1751472136 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): spa_load_retry: rewind, max txg: 6608974
1751472136 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): LOADING
1751472136 ffff962316d68000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=3801/1000000000
1751472136 ffff96213d698000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472136 ffff96213d698000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751472136 ffff96213d698000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472136 ffff9621b4bb3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': best uberblock found for spa tank. txg 6608968
1751472136 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=6608968
1751472136 ffff9621cd984e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1857/1000000000
1751472136 ffff96213d698000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=0
1751472136 ffff96213d698000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=0
1751472136 ffff96213d698000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=0
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1751472136 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): Read 7 log space maps (7 total blocks - blksz = 131072 bytes) in 0 ms
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1751472136 ffff9621b4bb3400 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1751472136 ffff9621b4ba4e00 metaslab.c:2495:metaslab_load_impl(): metaslab_load: txg 0, spa tank, vdev_id 0, ms_id 680, smp_length 503392, unflushed_allocs 36864, unflushed_frees 36864, freed 0, defer 0 + 0, unloaded time 40667794 ms, loading_time 9 ms, ms_max_size 4935270400, max size error 4935233536, old_weight 800000000000001, new_weight 800000000000001
1751472136 ffff9621b4bb3400 spa_misc.c:415:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_check_logs failed
1751472136 ffff9621b4bb3400 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): UNLOADING
1751472153 ffff9621107b4e00 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1751472153 ffff9621107b4e00 spa.c:6928:spa_tryimport(): spa_tryimport: importing tank
1751472153 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9621107b4e00-tank, config trusted): LOADING
1751472153 ffff962156dfb400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472153 ffff962156dfb400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751472153 ffff962156dfb400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472154 ffff9621cee09a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1012428705/1000000000
1751472154 ffff9621107b4e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': best uberblock found for spa $import-ffff9621107b4e00-tank. txg 6608975
1751472154 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9621107b4e00-tank, config untrusted): using uberblock with txg=6608975
1751472154 ffff962156dfb400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751472154 ffff962156dfb400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472154 ffff962156dfb400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472155 ffff9621cf2f4e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1014661388/1000000000
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1751472155 ffff9621ce90b400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a9893000
1751472155 ffff9621ce90b400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a988d000
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Calculating deflated space
1751472155 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Starting import
1751472155 ffff9621107b4e00 spa.c:9245:spa_async_request(): spa=$import-ffff9621107b4e00-tank async request task=2048
1751472155 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9621107b4e00-tank, config trusted): LOADED
1751472155 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9621107b4e00-tank, config trusted): UNLOADING
1751472155 ffff9621107b4e00 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1751472155 ffff9621107b4e00 spa.c:6773:spa_import(): spa_import: importing tank
1751472155 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): LOADING
1751472155 ffff9621cd984e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472155 ffff9621cd984e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472155 ffff9621cd984e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751472156 ffff9621cfdb9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1010289310/1000000000
1751472156 ffff9621107b4e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': best uberblock found for spa tank. txg 6608975
1751472156 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=6608975
1751472156 ffff9621cfaf9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=3597/1000000000
1751472156 ffff9621cd984e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=0
1751472156 ffff9621cd984e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=0
1751472156 ffff9621cd984e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=0
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1751472156 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): Read 7 log space maps (7 total blocks - blksz = 131072 bytes) in 0 ms
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1751472156 ffff9621107b4e00 spa_misc.c:415:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_check_logs failed
1751472156 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): UNLOADING
1751472156 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): spa_load_retry: rewind, max txg: 6608974
1751472156 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): LOADING
1751472156 ffff9621cf1f9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1918/1000000000
1751472156 ffff9621cec53400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751472156 ffff9621cec53400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472156 ffff9621cec53400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472156 ffff9621107b4e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': best uberblock found for spa tank. txg 6608968
1751472156 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=6608968
1751472156 ffff96211f200000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=3380/1000000000
1751472156 ffff9621cec53400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=0
1751472156 ffff9621cec53400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=0
1751472156 ffff9621cec53400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=0
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1751472156 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): Read 7 log space maps (7 total blocks - blksz = 131072 bytes) in 0 ms
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1751472156 ffff9621107b4e00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1751472156 ffff9621ce8d4e00 metaslab.c:2495:metaslab_load_impl(): metaslab_load: txg 0, spa tank, vdev_id 0, ms_id 680, smp_length 503392, unflushed_allocs 36864, unflushed_frees 36864, freed 0, defer 0 + 0, unloaded time 40687498 ms, loading_time 9 ms, ms_max_size 4935270400, max size error 4935233536, old_weight 800000000000001, new_weight 800000000000001
1751472156 ffff9621107b4e00 spa_misc.c:415:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_check_logs failed
1751472156 ffff9621107b4e00 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): UNLOADING
1751472378 ffff96211e511a00 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1751472378 ffff96211e511a00 spa.c:6928:spa_tryimport(): spa_tryimport: importing tank
1751472378 ffff96211e511a00 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96211e511a00-tank, config trusted): LOADING
1751472378 ffff96218bcc0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472378 ffff96218bcc0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472378 ffff96218bcc0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751472379 ffff9621cd984e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1013978547/1000000000
1751472379 ffff96211e511a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': best uberblock found for spa $import-ffff96211e511a00-tank. txg 6608975
1751472379 ffff96211e511a00 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96211e511a00-tank, config untrusted): using uberblock with txg=6608975
1751472379 ffff96218bcc0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751472379 ffff96218bcc0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472379 ffff96218bcc0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751472380 ffff96218c370000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1015452119/1000000000
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1751472380 ffff9621ce4a9a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a9893000
1751472380 ffff9621105a8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a988d000
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Calculating deflated space
1751472380 ffff96211e511a00 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Starting import
1751472380 ffff96211e511a00 spa.c:9245:spa_async_request(): spa=$import-ffff96211e511a00-tank async request task=2048
1751472380 ffff96211e511a00 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96211e511a00-tank, config trusted): LOADED
1751472380 ffff96211e511a00 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96211e511a00-tank, config trusted): UNLOADING
1751472156 ffff9621107b4e00 spa_misc.c:415:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_check_logs failed
We don’t have a SLOG device so I’m confused as to why it’s going to throw a failure here 
I wonder if adding the -m to tell it explicitly to discard ZIL contents is necessary here.
Importing with -fF as suggested should allow a rollback to a previous transaction group since it isn’t throwing an alert on the -n noop version. You may lose the last few seconds of data though. You can try importing with:
sudo zpool import -R /mnt -fF tank -o readonly=on
This will mount it read-only so changes would not be committed.
admin@truenas[~]$ sudo zpool import -R /mnt -fF tank -o readonly=on
[sudo] password for admin:
cannot import 'tank': I/O error
Destroy and re-create the pool from
a backup source.
in this situation, the latest changes don’t matter at all 
Let’s try explicitly telling it to disregard any log device it might think is present with -m
sudo zpool import -R /mnt -fF tank -o readonly=on -m
admin@truenas[~]$ sudo zpool import -R /mnt -fF tank -o readonly=on -m
cannot import 'tank': I/O error
Destroy and re-create the pool from
a backup source.
Let’s add the -X for eXtreme rewinds.
sudo zpool import -R /mnt -fFX tank -o readonly=on
I would say it is doing something, but I don’t know what … 
I will keep you informed!!!
You should be able to open a second SSH session (assuming you’re doing these commands from one, and not the webshell!) and probe iostat to see disk activity, as well as tail the end of the dbgmsg log file. If it’s still stuck in a loop of “error” and walking backwards through transaction groups that is not ideal.
admin@truenas[~]$ iostat
Linux 6.12.15-production+truenas (truenas) 07/02/25 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.02 0.14 0.93 0.10 0.00 97.80
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
loop1 0.03 1.24 0.00 0.00 73051 0 0
nvme0n1 6.41 16.21 64.74 0.00 957216 3822420 0
sda 13.99 9.76 190.90 0.00 576242 11272176 0
sdb 14.89 5264.01 0.01 0.00 310824052 432 0
sde 0.03 0.86 0.06 0.00 50826 3492 0
sdf 0.00 0.00 0.00 0.00 64 0 0
sdg 0.00 0.00 0.00 0.00 64 0 0
admin@truenas[~]$ tail -n 200 /proc/spl/kstat/zfs/dbgmsg
tail: cannot open '/proc/spl/kstat/zfs/dbgmsg' for reading: Permission denied
admin@truenas[~]$ sudo tail -n 200 /proc/spl/kstat/zfs/dbgmsg
[sudo] password for admin:
1751481077 ffff9621cfdb8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751481077 ffff9621cfdb8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751481077 ffff9621cfdb8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751481078 ffff9621cf2d9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1001480703/1000000000
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1751481078 ffff96211304b400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 13030021000
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1751481078 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1751481078 ffff96211c0f1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a9806000
1751481078 ffff9621b4bb0000 spa_misc.c:415:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_load_verify failed [error=5]
1751481078 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): UNLOADING
1751481078 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): spa_load_retry: rewind, max txg: 6608955
1751481078 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): LOADING
1751481078 ffff96220b4f9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751481078 ffff96220b4f9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751481078 ffff96220b4f9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751481079 ffff962317923400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1008739741/1000000000
1751481079 ffff9621b4bb0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': best uberblock found for spa tank. txg 6608955
1751481079 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=6608955
1751481079 ffff9621cf0e9a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97f4000
1751481079 ffff96220b4f9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751481079 ffff96220b4f9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751481079 ffff96220b4f9a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751481080 ffff9621b66c3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1004712256/1000000000
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1751481080 ffff96220b4f8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=0 off 231a97f8000
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1751481080 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1751481080 ffff9621cf0e9a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=0 off 231a97ef000
1751481080 ffff9621cee43400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302fe5f000
1751481080 ffff9621b4bb0000 spa_misc.c:415:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_load_verify failed [error=5]
1751481080 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): UNLOADING
1751481080 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): spa_load_retry: rewind, max txg: 6608954
1751481080 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): LOADING
1751481080 ffff9621cf731a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751481080 ffff9621cf731a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751481080 ffff9621cf731a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751481081 ffff962317924e00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1008572166/1000000000
1751481081 ffff9621b4bb0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': best uberblock found for spa tank. txg 6608954
1751481081 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=6608954
1751481081 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97d3000
1751481081 ffff9621cf731a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751481081 ffff9621cf731a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751481081 ffff9621cf731a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=0 cant_write=1
1751481082 ffff96220b4f8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1006279352/1000000000
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1751481082 ffff9621cd9c3400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302fffa000
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1751481082 ffff9621b4bb0000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1751481082 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302fe5f000
1751481082 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
(spa_load_verify_data=1, spa_load_verify_metadata=1)
1751481082 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 232a6019000
1751481082 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97d3000
1751481082 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a97de000
1751481082 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a97d8000
1751481082 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 1302fe61000
1751481082 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302fffa000
1751481082 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 1302fff9000
1751481083 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 232a5ed0000
1751481083 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 232a6016000
1751481084 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97e8000
1751481085 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97b2000
1751481085 ffff9621cd9c3400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97d9000
1751481085 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 233b3240000
1751481085 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 233b34ab000
1751481085 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 233b349b000
1751481085 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 233b349f000
1751481085 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 233b34a2000
1751481085 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 233b3499000
1751481085 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 233b3493000
1751481086 ffff9621cd9c3400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 2269f67a000
1751481086 ffff9621cd9c3400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 2269f648000
1751481086 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 2269f67b000
1751481090 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 233b349e000
1751481095 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a9799000
1751481095 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a97db000
1751481095 ffff9621cd9c3400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a97e7000
1751481095 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a97d2000
1751481095 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97e8000
1751481095 ffff9621cd9c3400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97d9000
1751481095 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97df000
1751481095 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97c4000
1751481095 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 2269f650000
1751481096 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 233b3438000
1751481098 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 2269f67d000
1751481098 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 2269f674000
1751481099 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 232a5fe0000
1751481101 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a9355000
1751481101 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97b2000
1751481101 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a97ac000
1751481101 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 232a5ff5000
1751481101 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 232a601c000
1751481101 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 232a601f000
1751481101 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 232a6013000
1751481101 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 232a5c1b000
1751481101 ffff9621cd9c3400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 232a6014000
1751481101 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 233b2f5e000
1751481101 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 233b3474000
1751481102 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 233b2f5f000
1751481102 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 1302ffae000
1751481103 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 233b344d000
1751481104 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 2339116f000
1751481107 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 1302fffc000
1751481107 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 2269f668000
1751481107 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a969a000
1751481108 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 233b3498000
1751481108 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 233b34a1000
1751481108 ffff9621cd9c3400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 233b3492000
1751481108 ffff9621cd9c3400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302fe5f000
1751481108 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302ffee000
1751481108 ffff9621cd9c3400 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=0 off 2269f004000
1751481108 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=0 off 2269f003000
1751481108 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 2319b843000
1751481109 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 2328ce3b000
1751481109 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 232a5f12000
1751481109 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 233b323e000
1751481109 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 233b3331000
1751481109 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 233b349c000
1751481109 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 233b3490000
1751481117 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a977c000
1751481117 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302ffaf000
1751481117 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 1302ffb4000
1751481117 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a977e000
1751481118 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 2269f63e000
1751481118 ffff9621ce514e00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302fbaa000
1751481118 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a945d000
1751481119 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302fbaa000
1751481119 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302ffaf000
1751481119 ffff9621ce514e00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 2269f2a9000
1751481119 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 2269f645000
1751481119 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 2269f63f000
1751481119 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a977c000
1751481119 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 232a5fcc000
1751481119 ffff9621ce514e00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 232a5fc5000
1751481121 ffff9627c2741a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a964a000
1751481121 ffff9621ce514e00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 2269f560000
1751481121 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 231a9649000
1751481121 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302febc000
1751481121 ffff9621ce514e00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 232a5ecf000
1751481121 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 231a964a000
1751481121 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302febc000
1751481123 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302c61f000
1751481128 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 1302febb000
1751481130 ffff96211c3c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 2269f55e000
1751481130 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302c61f000
1751481130 ffff9621ceca0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 1302c604000
1751481130 ffff9621ce514e00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 226853ec000
1751481130 ffff9621ce52ce00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=1 off 232a5ecd000
1751481130 ffff9621ceca1a00 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=1 off 1300ace2000
1751483652 ffff9621cf731a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=1 cant_write=1
1751483652 ffff9621cf731a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': failed probe
1751483652 ffff9621cf731a00 spa.c:9245:spa_async_request(): spa=tank async request task=4
1751483652 ffff9621cf731a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': probe done, cant_read=1 cant_write=1
1751483652 ffff9621cf731a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': failed probe
1751483652 ffff9621cf731a00 spa.c:9245:spa_async_request(): spa=tank async request task=4
1751483708 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): spa_load_verify found 6 metadata errors and 1313 data errors
1751483708 ffff9621b4bb0000 spa_misc.c:415:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_load_verify failed [error=6]
1751483708 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): UNLOADING
1751483708 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): spa_load_retry: rewind, max txg: 6608953
1751483708 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): LOADING
1751483708 ffff96227b74b400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751483709 ffff96214aa59a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1001175437/1000000000
1751483709 ffff9621cde79a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1013191041/1000000000
1751483709 ffff9621cdfc3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1013186318/1000000000
1751483709 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): vdev tree has 1 missing top-level vdevs.
1751483709 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
1751483709 ffff9621b4bb0000 spa_misc.c:415:spa_load_failed(): spa_load(tank, config untrusted): FAILED: unable to open vdev tree [error=2]
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 0: root, guid: 13400593935322567235, path: N/A, can't open
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 0: raidz, guid: 15550163770109457767, path: N/A, can't open
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 0: disk, guid: 9392911267891898184, path: /dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27, can't open
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 1: disk, guid: 5149461304993027350, path: /dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27, can't open
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 2: disk, guid: 15599814676261111453, path: /dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27, can't open
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 3: disk, guid: 4243709192505670205, path: /dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68, healthy
1751483709 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): UNLOADING
1751487743 ffff962123933400 metaslab.c:3721:metaslab_condense(): condensing: txg 6610402, msp[82] ffff9621551f5000, vdev id 0, spa boot-pool, smp size 524552, segments 354, forcing condense=FALSE
1751483708 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): spa_load_retry: rewind, max txg: 6608953
1751483708 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config trusted): LOADING
1751483708 ffff96227b74b400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68': probe done, cant_read=0 cant_write=1
1751483709 ffff96214aa59a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1001175437/1000000000
1751483709 ffff9621cde79a00 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1013191041/1000000000
1751483709 ffff9621cdfc3400 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27': open error=2 timeout=1013186318/1000000000
1751483709 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): vdev tree has 1 missing top-level vdevs.
1751483709 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
1751483709 ffff9621b4bb0000 spa_misc.c:415:spa_load_failed(): spa_load(tank, config untrusted): FAILED: unable to open vdev tree [error=2]
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 0: root, guid: 13400593935322567235, path: N/A, can't open
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 0: raidz, guid: 15550163770109457767, path: N/A, can't open
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 0: disk, guid: 9392911267891898184, path: /dev/disk/by-partuuid/4f584c22-2192-11ef-8ffc-bc2411747b27, can't open
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 1: disk, guid: 5149461304993027350, path: /dev/disk/by-partuuid/4f7421bf-2192-11ef-8ffc-bc2411747b27, can't open
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 2: disk, guid: 15599814676261111453, path: /dev/disk/by-partuuid/4f4aa40e-2192-11ef-8ffc-bc2411747b27, can't open
1751483709 ffff9621b4bb0000 vdev.c:235:vdev_dbgmsg_print_tree(): vdev 3: disk, guid: 4243709192505670205, path: /dev/disk/by-partuuid/58a99133-4ca3-4ae2-afc3-f88ad96f3d68, healthy
1751483709 ffff9621b4bb0000 spa_misc.c:429:spa_load_note(): spa_load(tank, config untrusted): UNLOADING
I’m wondering why it’s trying to enter at txg 6608953 which is further ahead than what appears to be the latest on-disk value of 6606762 from your post above:
So… it does not look good? Should I stop it?
Just double checking. This is a bare metal install of TrueNAS and not TrueNAS running as a VM