Without the space is correct, you’re combining a force import with Force rewind, eXtreme rewind, and a specific Transaction group - so -fFXT
Fingers crossed.
1 Like
No luck. it failed after some activity. zpool import doesnt show
this is from file vi /proc/spl/kstat/zfs/dbgmsg . Im not sure if im looking at right place but i still see 2 disk is waiting to be added to pool in the dashboard and not imported
1732574634 spa.c:6771:spa_tryimport(): spa_tryimport: importing tank
1732574634 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADING
1732574634 metaslab.c:3939:metaslab_flush(): flushing: txg 3718696, spa ssd, vdev_id 0, ms_id 75, unflushed_allocs 401408, unflushed_frees 512000, appended 464 bytes
1732574635 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': open error=2 timeout=1000001027/1000000000
1732574635 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': best uberblock found for spa $import. txg 3583088
1732574635 spa_misc.c:418:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=3583088
1732574636 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': open error=2 timeout=1000000597/1000000000
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1732574636 dprintf: brt.c:694:brt_vdevs_expand(): BRT VDEVs expanded from 0 to 1.
1732574636 dprintf: brt.c:543:brt_vdev_realloc(): BRT VDEV 0 initiated.
1732574636 dprintf: brt.c:602:brt_vdev_load(): MOS BRT VDEV com.fudosecurity:brt:vdev:0 loaded: mos_brtvdev=273, mos_entries=272
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Calculating deflated space
1732574636 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Starting import
1732574636 spa.c:8925:spa_async_request(): spa=$import async request task=2048
1732574636 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADED
1732574636 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): UNLOADING
1732574636 dprintf: brt.c:625:brt_vdev_dealloc(): BRT VDEV 0 deallocated.
1732574639 metaslab.c:3939:metaslab_flush(): flushing: txg 3718697, spa ssd, vdev_id 0, ms_id 109, unflushed_allocs 36864, unflushed_frees 16384, appended 32 bytes
1732574644 metaslab.c:3939:metaslab_flush(): flushing: txg 3718698, spa ssd, vdev_id 0, ms_id 45, unflushed_allocs 425984, unflushed_frees 430080, appended 392 bytes
1732574649 metaslab.c:3939:metaslab_flush(): flushing: txg 3718699, spa ssd, vdev_id 0, ms_id 79, unflushed_allocs 3690496, unflushed_frees 3493888, appended 944 bytes
1732574654 metaslab.c:3939:metaslab_flush(): flushing: txg 3718700, spa ssd, vdev_id 0, ms_id 70, unflushed_allocs 339968, unflushed_frees 348160, appended 392 bytes
1732574659 metaslab.c:3939:metaslab_flush(): flushing: txg 3718701, spa ssd, vdev_id 0, ms_id 80, unflushed_allocs 0, unflushed_frees 16384, appended 40 bytes
1732574665 metaslab.c:3939:metaslab_flush(): flushing: txg 3718702, spa ssd, vdev_id 0, ms_id 82, unflushed_allocs 524288, unflushed_frees 495616, appended 744 bytes
1732574670 metaslab.c:3939:metaslab_flush(): flushing: txg 3718703, spa ssd, vdev_id 0, ms_id 81, unflushed_allocs 0, unflushed_frees 135168, appended 32 bytes
1732574675 metaslab.c:3939:metaslab_flush(): flushing: txg 3718704, spa ssd, vdev_id 0, ms_id 72, unflushed_allocs 393216, unflushed_frees 786432, appended 624 bytes
Odd:
1732574635 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': best uberblock found for spa $import. txg 3583088
You specified to import at the earlier 3583040 but it still tried 3583088?
This is the final in the display (monitor) before it restarted
EDIT: Should i go back to another txg or try with one disk ? not sure if that will even work. just throwing the idea.
i want to load once so i can take a copy and then i can probably redo the pool. may be 1 pool with 2 disk and another pool with another 2 disk. one as backup to other.
lol. i never thought i will be in this situation even with 3 disk in mirror
EDIT2: Trying another one. fingers crossed.
sudo zpool import -fFXT 3583074 -R /mnt -o readonly=on tank
EDIT3: Crashed immediately
anyone have any more ideas to get me out of this issue ? @HoneyBadger any inputs ? TIA
1 Like
Let’s try disconnecting everything except for the one drive with S/N 8CJVUUNE and then attempting this one:
sudo zpool import -fFXT 3583047 -R /mnt -o readonly=on tank -d /dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28
One of the drives has the others tagged with expand_time in the label, the other has one tagged with removed - this drive seems to be the “newest” but yet has this one txg sitting there chilling out from a previous situation.
didnt work
root@truenas[/home/admin]# sudo lsblk -o name,serial,uuid,partuuid,label
NAME SERIAL UUID PARTUUID LABEL
sda TPBF2312190010103863
└─sda1 9570678906628884325 d4c7f156-eff2-410c-b410-46c884c268f8 ssd
sdb 8CJVUUNE
└─sdb1 5087050144587203797 a9e51b2f-2df8-48ae-9c3e-6303e9256c28 tank
sdd 2L3329S8HKF7
├─sdd1 f831f3ef-03d6-48a2-bb72-a2fb6791da77
├─sdd2 44F1-C597 ee924cc1-87b4-4de0-a8bd-0fec0097dc63 EFI
├─sdd3 1696277370963779199 408a753f-3d74-4d2a-be6a-77cf0c711051 boot-pool
└─sdd4 cea312e0-c0f5-4a8d-bdf9-8d84ef58ca88
└─sdd4 003b4203-a009-4361-baa0-4c2137f4aab8
sde MAB446W102869
└─sde1 9570678906628884325 334cbbbb-e0d5-40c6-8462-5f82b0033ad7 ssd
root@truenas[/home/admin]# sudo zpool import -fFXT 3583047 -R /mnt -o readonly=on tank -d /dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28
cannot import 'tank': one or more devices is currently unavailable
root@truenas[/home/admin]#
Try dropping the poolname of tank from the command. It seems odd that it’s still looking for other devices.
The expand and removed makes me a bit puzzled - it’s almost like it’s looking for an indirection table (from a removed vdev) that’s not been properly stored on one drive or another.
with tank removed i got a different message. zpool status doesnt show it still ? is it imported ?
root@truenas[/home/admin]# sudo zpool import -fFXT 3583047 -R /mnt -o readonly=on -d /dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28
pool: tank
id: 5087050144587203797
state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:
tank DEGRADED
mirror-0 DEGRADED
a9e51b2f-2df8-48ae-9c3e-6303e9256c28 ONLINE
d08bcfdf-d4fc-4094-8978-7f29368f737f UNAVAIL
a3573b00-d380-4587-806a-8a1a50690002 UNAVAIL
root@truenas[/home/admin]# zpool status
pool: boot-pool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:01:56 with 0 errors on Fri Nov 22 03:46:58 2024
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
sdd3 ONLINE 0 0 0
errors: No known data errors
pool: ssd
state: ONLINE
scan: scrub repaired 0B in 00:10:41 with 0 errors on Sun Nov 24 00:10:42 2024
config:
NAME STATE READ WRITE CKSUM
ssd ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
334cbbbb-e0d5-40c6-8462-5f82b0033ad7 ONLINE 0 0 0
d4c7f156-eff2-410c-b410-46c884c268f8 ONLINE 0 0 0
errors: No known data errors
root@truenas[/home/admin]#
No, it seems like it wasn’t sure what to do without the poolname. Let’s move the numeric ID first.
sudo zpool import 5087050144587203797 -fFXT 3583047 -R /mnt -o readonly=on -d /dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28
didnt work. same error as previous time
root@truenas[/home/admin]# sudo zpool import 5087050144587203797 -fFXT 3583047 -R /mnt -o readonly=on -d /dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28
cannot import 'tank': one or more devices is currently unavailable
root@truenas[/home/admin]#
Over the internet, I am yelling at your system “What you do you bloody mean ‘one or more devices’ it’s a mirror3!”
Can you perform the bpftrace command again in another SSH session:
bpftrace -e 'kretprobe:zil_read_log_block { printf("rlb: %d\n", retval); } kretprobe:zil_claim_log_block { printf("clb: %d\n", retval); } kretprobe:zil_claim_log_record { printf("clr: %d\n", r
as well as the contents of /proc/spl/kstat/zfs/dbgmsg after the import fails?
1 Like
that probe didnt get any results
this is the dbgmsg output.
1732819564 spa.c:6768:spa_tryimport(): spa_tryimport: importing tank, max_txg=3583047
1732819564 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADING
1732819565 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000001466/1000000000
1732819565 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000000396/1000000000
1732819565 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': best uberblock found for spa $import. txg 3583047
1732819565 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': label discarded as txg is too large (3583088 > 3583047)
1732819565 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': failed to read label config. Trying again without txg restrictions.
1732819565 spa_misc.c:418:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=3583047
1732819566 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000000846/1000000000
1732819566 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000001226/1000000000
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1732819566 dprintf: brt.c:694:brt_vdevs_expand(): BRT VDEVs expanded from 0 to 1.
1732819566 dprintf: brt.c:543:brt_vdev_realloc(): BRT VDEV 0 initiated.
1732819566 dprintf: brt.c:602:brt_vdev_load(): MOS BRT VDEV com.fudosecurity:brt:vdev:0 loaded: mos_brtvdev=273, mos_entries=272
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Calculating deflated space
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Starting import
1732819566 spa.c:8925:spa_async_request(): spa=$import async request task=2048
1732819566 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADED
1732819566 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): UNLOADING
1732819566 dprintf: brt.c:625:brt_vdev_dealloc(): BRT VDEV 0 deallocated.
1732819566 spa.c:6625:spa_import(): spa_import: importing tank, max_txg=3583047 (RECOVERY MODE)
1732819566 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): LOADING
1732819567 metaslab.c:3939:metaslab_flush(): flushing: txg 3766471, spa ssd, vdev_id 0, ms_id 46, unflushed_allocs 172032, unflushed_frees 167936, appended 192 bytes
1732819567 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000002926/1000000000
1732819567 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000001656/1000000000
1732819567 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': best uberblock found for spa tank. txg 3583047
1732819567 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': label discarded as txg is too large (3583088 > 3583047)
1732819567 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': failed to read label config. Trying again without txg restrictions.
1732819567 spa_misc.c:418:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=3583047
1732819568 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000001656/1000000000
1732819568 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000000326/1000000000
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1732819568 dprintf: brt.c:694:brt_vdevs_expand(): BRT VDEVs expanded from 0 to 1.
1732819568 dprintf: brt.c:543:brt_vdev_realloc(): BRT VDEV 0 initiated.
1732819568 dprintf: brt.c:602:brt_vdev_load(): MOS BRT VDEV com.fudosecurity:brt:vdev:0 loaded: mos_brtvdev=273, mos_entries=272
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1732819568 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1732819569 spa_misc.c:404:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_load_verify failed [error=52]
1732819569 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): UNLOADING
1732819569 dprintf: brt.c:625:brt_vdev_dealloc(): BRT VDEV 0 deallocated.
1732819569 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): spa_load_retry: rewind, max txg: 3583046
1732819569 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): LOADING
1732819570 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000001876/1000000000
1732819570 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000000996/1000000000
1732819570 spa_misc.c:404:spa_load_failed(): spa_load(tank, config untrusted): FAILED: no valid uberblock found
1732819570 spa_misc.c:418:spa_load_note(): spa_load(tank, config untrusted): UNLOADING
1732819572 metaslab.c:3939:metaslab_flush(): flushing: txg 3766472, spa ssd, vdev_id 0, ms_id 68, unflushed_allocs 516096, unflushed_frees 520192, appended 376 bytes
1732819577 metaslab.c:3939:metaslab_flush(): flushing: txg 3766473, spa ssd, vdev_id 0, ms_id 79, unflushed_allocs 102400, unflushed_frees 102400, appended 128 bytes
1732819582 metaslab.c:3939:metaslab_flush(): flushing: txg 3766474, spa ssd, vdev_id 0, ms_id 80, unflushed_allocs 143360, unflushed_frees 147456, appended 120 bytes
1732819587 spa.c:6768:spa_tryimport(): spa_tryimport: importing tank, max_txg=3583047
1732819587 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADING
1732819587 metaslab.c:3939:metaslab_flush(): flushing: txg 3766475, spa ssd, vdev_id 0, ms_id 49, unflushed_allocs 294912, unflushed_frees 299008, appended 184 bytes
1732819588 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000002126/1000000000
1732819588 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000000736/1000000000
1732819588 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': best uberblock found for spa $import. txg 3583047
1732819588 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': label discarded as txg is too large (3583088 > 3583047)
1732819588 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': failed to read label config. Trying again without txg restrictions.
1732819588 spa_misc.c:418:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=3583047
1732819589 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000000366/1000000000
1732819589 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000000286/1000000000
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1732819589 dprintf: brt.c:694:brt_vdevs_expand(): BRT VDEVs expanded from 0 to 1.
1732819589 dprintf: brt.c:543:brt_vdev_realloc(): BRT VDEV 0 initiated.
1732819589 dprintf: brt.c:602:brt_vdev_load(): MOS BRT VDEV com.fudosecurity:brt:vdev:0 loaded: mos_brtvdev=273, mos_entries=272
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Calculating deflated space
1732819589 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Starting import
1732819589 spa.c:8925:spa_async_request(): spa=$import async request task=2048
1732819589 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADED
1732819589 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): UNLOADING
1732819589 dprintf: brt.c:625:brt_vdev_dealloc(): BRT VDEV 0 deallocated.
1732819589 spa.c:6625:spa_import(): spa_import: importing tank, max_txg=3583047 (RECOVERY MODE)
1732819589 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): LOADING
1732819590 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000395937/1000000000
1732819590 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000411957/1000000000
1732819590 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': best uberblock found for spa tank. txg 3583047
1732819590 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': label discarded as txg is too large (3583088 > 3583047)
1732819590 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': failed to read label config. Trying again without txg restrictions.
1732819590 spa_misc.c:418:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=3583047
1732819591 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000002776/1000000000
1732819591 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000000636/1000000000
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1732819591 dprintf: brt.c:694:brt_vdevs_expand(): BRT VDEVs expanded from 0 to 1.
1732819591 dprintf: brt.c:543:brt_vdev_realloc(): BRT VDEV 0 initiated.
1732819591 dprintf: brt.c:602:brt_vdev_load(): MOS BRT VDEV com.fudosecurity:brt:vdev:0 loaded: mos_brtvdev=273, mos_entries=272
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1732819591 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1732819592 spa_misc.c:404:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_load_verify failed [error=52]
1732819592 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): UNLOADING
1732819592 dprintf: brt.c:625:brt_vdev_dealloc(): BRT VDEV 0 deallocated.
1732819592 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): spa_load_retry: rewind, max txg: 3583046
1732819592 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): LOADING
1732819592 metaslab.c:3939:metaslab_flush(): flushing: txg 3766476, spa ssd, vdev_id 0, ms_id 75, unflushed_allocs 299008, unflushed_frees 299008, appended 184 bytes
1732819593 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000001396/1000000000
1732819593 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000001976/1000000000
1732819593 spa_misc.c:404:spa_load_failed(): spa_load(tank, config untrusted): FAILED: no valid uberblock found
1732819593 spa_misc.c:418:spa_load_note(): spa_load(tank, config untrusted): UNLOADING
one time i see this message
1732819566 spa_misc.c:2311:spa_import_progress_set_notes_impl(): ‘tank’ Starting import
but not able to understand why it is failing
Well, we’re at the point now of attempting dangerous recoveries.
Keep only one disk attached and check the below:
Warning - These commands will disable the ZFS SPA (Storage Pool Allocator) verify passes. This is intended only for last-resort recovery and data integrity will no longer be guaranteed.
These commands must be run from root prompt. You cannot sudo them.
sudo -s
echo 0 > /sys/module/zfs/parameters/spa_load_verify_data
echo 0 > /sys/module/zfs/parameters/spa_load_verify_metadata
exit
Once those steps are complete, try the same zpool import command as before.
3 Likes
yes only one disk connected from the previous attempt.
is this right ? i feel may be i didnt execute it correctly ? but result is same.
root@truenas[/home/admin]# sudo -s
root@truenas[/home/admin]# echo 0 > /sys/module/zfs/parameters/spa_load_verify_data
root@truenas[/home/admin]# echo 0 > /sys/module/zfs/parameters/spa_load_verify_metadata
root@truenas[/home/admin]# exit
root@truenas[/home/admin]# sudo zpool import 5087050144587203797 -fFXT 3583047 -R /mnt -o readonly=on -d /dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28
cannot import 'tank': one or more devices is currently unavailable
root@truenas[/home/admin]#
1732829855 spa.c:8925:spa_async_request(): spa=$import async request task=2048
1732829855 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADED
1732829855 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): UNLOADING
1732829855 dprintf: brt.c:625:brt_vdev_dealloc(): BRT VDEV 0 deallocated.
1732829855 spa.c:6625:spa_import(): spa_import: importing tank, max_txg=3583047 (RECOVERY MODE)
1732829855 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): LOADING
1732829856 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000000226/1000000000
1732829856 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000001326/1000000000
1732829856 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': best uberblock found for spa tank. txg 3583047
1732829856 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': label discarded as txg is too large (3583088 > 3583047)
1732829856 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a9e51b2f-2df8-48ae-9c3e-6303e9256c28': failed to read label config. Trying again without txg restrictions.
1732829856 spa_misc.c:418:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=3583047
1732829857 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000001286/1000000000
1732829857 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000001486/1000000000
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading checkpoint txg
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading indirect vdev metadata
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Checking feature flags
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading special MOS directories
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading properties
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading AUX vdevs
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading vdev metadata
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading dedup tables
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Loading BRT
1732829857 dprintf: brt.c:694:brt_vdevs_expand(): BRT VDEVs expanded from 0 to 1.
1732829857 dprintf: brt.c:543:brt_vdev_realloc(): BRT VDEV 0 initiated.
1732829857 dprintf: brt.c:602:brt_vdev_load(): MOS BRT VDEV com.fudosecurity:brt:vdev:0 loaded: mos_brtvdev=273, mos_entries=272
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying Log Devices
1732829857 spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'tank' Verifying pool data
1732829858 spa_misc.c:404:spa_load_failed(): spa_load(tank, config trusted): FAILED: spa_load_verify failed [error=52]
1732829858 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): UNLOADING
1732829858 dprintf: brt.c:625:brt_vdev_dealloc(): BRT VDEV 0 deallocated.
1732829858 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): spa_load_retry: rewind, max txg: 3583046
1732829858 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): LOADING
1732829858 metaslab.c:3939:metaslab_flush(): flushing: txg 3768481, spa ssd, vdev_id 0, ms_id 80, unflushed_allocs 184320, unflushed_frees 196608, appended 168 bytes
1732829859 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d08bcfdf-d4fc-4094-8978-7f29368f737f': open error=2 timeout=1000422567/1000000000
1732829859 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a3573b00-d380-4587-806a-8a1a50690002': open error=2 timeout=1000722387/1000000000
1732829859 spa_misc.c:404:spa_load_failed(): spa_load(tank, config untrusted): FAILED: no valid uberblock found
1732829859 spa_misc.c:418:spa_load_note(): spa_load(tank, config untrusted): UNLOADING
Let me know if i should try this with another disk and another txg ? Thanks for helping here @HoneyBadger
@truenasrooks were these drives used by a different system beforehand? we’ve just squashed an odd bug on the back end that has to do with drives not being properly recognized under certain circumstances if they came with an existing filesystem.
If they were, please post the output of sudo wipefs -n /dev/sdb1 for each of the partitions (sdb1 sdc1 sdf1) - the -n will explicitly tell the command not to do anything, it doesn’t by default, but this will ensure it doesn’t. 
sure let me get that data but these drives were bought from ebay from old Datacenter HDD.
but it was wiped off earlier by the seller i assume. After i bought it i created the pool and thats it. never did anything. Also, didnt use anything apart from truenas.
This machine has windows os in another SSD but its not always connected. couple of times i booted but that long long time ago.
But even if they were wiped after usage in the datacenter, it’s not completely unreasonable to think that someone, likely the seller, put them in a Windows PC and formatted them to NTFS before running CrystalDiskInfo on them so they can show prospective buyers their status.
A pass of badblocks burn-in should take care of that… 
2 Likes