-fFX can take a long time to run as it is searching for hopeful uberblocks and walking/verifying metadata.
To capture the logs on CORE you’ll need a second SSH session to run
dtrace -n 'zfs-dbgmsg{trace(stringof(arg0))}' | tee dbgmsg.txt
Then attempt the import, and it will dump the results to the dbgmsg.txt as well as showing them live on the screen.
Will the prompt return immediately as it runs in the background or will it appear to “hang” without returning the prompt?
It’s not a backgrounded job, the SSH session attempting to import will stall/hang and not respond until there is a success/fail condition.
@JGordonT could still open a separate SSH session now and start the dtrace command above to see if it continues to throw repeated iterations of “attempting to fall back to TXG (x) - fail”
The dtrace command showed “dtrace: description ‘zfs-dbgmsg’ matched 1 probe”
then hung
It’s likely not hung, it’s just waiting to show more messages.
Can you cancel out of the import in the other window and try again with just -fF and not the X?
Is there potentially an issue with a console session timing out whilst it tries the import then?
# zpool import -fF -d /dev/gptid -R /mnt 14223652181009904092
cannot import 'Pool2': insufficient replicas
Destroy and re-create the pool from
a backup source.
When I ran the -fF import there was outpuy under the dtrace command as follows
# dtrace -n 'zfs-dbgmsg{trace(stringof(arg0))}' | tee dbgmsg.txt
dtrace: description 'zfs-dbgmsg' matched 1 probe
CPU ID FUNCTION:NAME
1 84255 none:zfs-dbgmsg spa.c:6771:spa_tryimport(): spa_tryimport: importing AriseLD
1 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): LOADING
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
1 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa $import. txg 17236786
1 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=17236786
1 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load($import, config untrusted): FAILED: unable to retrieve MOS config
1 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): UNLOADING
0 84255 none:zfs-dbgmsg spa.c:6626:spa_import(): spa_import: importing AriseLD, max_txg=-1 (RECOVERY MODE)
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(AriseLD, config trusted): LOADING
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa AriseLD. txg 17236786
6 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(AriseLD, config untrusted): using uberblock with txg=17236786
5 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(AriseLD, config untrusted): FAILED: unable to retrieve MOS config
5 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(AriseLD, config untrusted): UNLOADING
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(AriseLD, config untrusted): spa_load_retry: rewind, max txg: 17236785
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(AriseLD, config untrusted): LOADING
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
1 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa AriseLD. txg 17236781
1 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(AriseLD, config untrusted): using uberblock with txg=17236781
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
1 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading checkpoint txg
1 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading indirect vdev metadata
1 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Checking feature flags
1 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading special MOS directories
1 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading properties
1 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading AUX vdevs
1 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading vdev metadata
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=97]
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=5]
0 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config trusted): FAILED: vdev_load failed [error=97]
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): UNLOADING
I canceled out the dtrace command then restarted it. I then restarted the import with the -fFX option and this is what I got on the dtrace before it stopped
The import -fFX appears to have also hung or waiting on additional data
dtrace -n 'zfs-dbgmsg{trace(stringof(arg0))}' | tee dbgmsg.txt
dtrace: description 'zfs-dbgmsg' matched 1 probe
CPU ID FUNCTION:NAME
1 84255 none:zfs-dbgmsg spa.c:6771:spa_tryimport(): spa_tryimport: importing Pool2
1 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): LOADING
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
4 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa $import. txg 17236786
4 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=17236786
4 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load($import, config untrusted): FAILED: unable to retrieve MOS config
4 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): UNLOADING
7 84255 none:zfs-dbgmsg spa.c:6626:spa_import(): spa_import: importing Pool2, max_txg=-1 (RECOVERY MODE)
7 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): LOADING
5 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
4 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236786
4 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236786
4 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config untrusted): FAILED: unable to retrieve MOS config
4 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): UNLOADING
7 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): spa_load_retry: rewind, max txg: 17236785
7 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): LOADING
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236781
7 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236781
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading checkpoint txg
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading indirect vdev metadata
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Checking feature flags
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading special MOS directories
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading properties
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading AUX vdevs
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading vdev metadata
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=97]
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=5]
2 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config trusted): FAILED: vdev_load failed [error=97]
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): UNLOADING
4 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): spa_load_retry: rewind, max txg: 17236780
4 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): LOADING
3 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
4 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236780
3 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236780
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading checkpoint txg
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading indirect vdev metadata
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Checking feature flags
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading special MOS directories
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading properties
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading AUX vdevs
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading vdev metadata
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=97]
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=5]
4 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config trusted): FAILED: vdev_load failed [error=97]
4 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): UNLOADING
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): spa_load_retry: rewind, max txg: 17236779
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): LOADING
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
0 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236779
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236779
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading checkpoint txg
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading indirect vdev metadata
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Checking feature flags
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading special MOS directories
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading properties
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading AUX vdevs
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading vdev metadata
3 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=97]
3 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=5]
2 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config trusted): FAILED: vdev_load failed [error=97]
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): UNLOADING
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): spa_load_retry: rewind, max txg: 17236778
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): LOADING
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236778
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236778
3 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading checkpoint txg
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading indirect vdev metadata
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Checking feature flags
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading special MOS directories
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading properties
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading AUX vdevs
3 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading vdev metadata
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=97]
3 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=5]
4 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config trusted): FAILED: vdev_load failed [error=97]
4 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): UNLOADING
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): spa_load_retry: rewind, max txg: 17236777
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): LOADING
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236777
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': label discarded as txg is too large (17236778 > 17236777)
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': failed to read label config. Trying again without txg restrictions.
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236777
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading checkpoint txg
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading indirect vdev metadata
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Checking feature flags
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading special MOS directories
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading properties
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading AUX vdevs
2 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading vdev metadata
0 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=97]
0 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=5]
3 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config trusted): FAILED: vdev_load failed [error=97]
3 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): UNLOADING
5 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): spa_load_retry: rewind, max txg: 17236776
5 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): LOADING
0 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236776
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': label discarded as txg is too large (17236778 > 17236776)
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': failed to read label config. Trying again without txg restrictions.
6 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236776
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading checkpoint txg
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading indirect vdev metadata
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Checking feature flags
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading special MOS directories
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading properties
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading AUX vdevs
5 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading vdev metadata
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=97]
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_load: vdev_dtl_load failed [error=5]
3 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config trusted): FAILED: vdev_load failed [error=97]
3 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): UNLOADING
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): spa_load_retry: rewind, max txg: 17236775
2 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config trusted): LOADING
0 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236775
1 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': label discarded as txg is too large (17236778 > 17236775)
1 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': failed to read label config. Trying again without txg restrictions.
1 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236775
5 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=97]
5 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): UNLOADING
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): spa_load_retry: rewind, max txg: 17236774
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): LOADING
5 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
0 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236774
0 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': label discarded as txg is too large (17236778 > 17236774)
0 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': failed to read label config. Trying again without txg restrictions.
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236774
0 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=97]
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): UNLOADING
6 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): spa_load_retry: rewind, max txg: 17236773
6 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): LOADING
3 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236773
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': label discarded as txg is too large (17236778 > 17236773)
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': failed to read label config. Trying again without txg restrictions.
6 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236773
6 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=97]
6 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): UNLOADING
7 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): spa_load_retry: rewind, max txg: 17236772
7 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): LOADING
2 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236772
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': label discarded as txg is too large (17236778 > 17236772)
7 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': failed to read label config. Trying again without txg restrictions.
7 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236772
5 84255 none:zfs-dbgmsg spa_misc.c:405:spa_load_failed(): spa_load(Pool2, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=97]
5 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): UNLOADING
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): spa_load_retry: rewind, max txg: 17236771
0 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): LOADING
5 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': best uberblock found for spa Pool2. txg 17236767
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': label discarded as txg is too large (17236778 > 17236767)
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8c9a8d5-ffd5-11ec-b16d-002590d59089': failed to read label config. Trying again without txg restrictions.
6 84255 none:zfs-dbgmsg spa_misc.c:419:spa_load_note(): spa_load(Pool2, config untrusted): using uberblock with txg=17236767
6 84255 none:zfs-dbgmsg vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/gptid/d8dba0c2-ffd5-11ec-b16d-002590d59089': vdev_geom_open: failed to open [error=2]
4 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading checkpoint txg
4 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading indirect vdev metadata
4 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Checking feature flags
4 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading special MOS directories
4 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading properties
4 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading AUX vdevs
4 84255 none:zfs-dbgmsg spa_misc.c:2311:spa_import_progress_set_notes_impl(): 'Pool2' Loading vdev metadata
One of my ssh connections had an error Network err caused connection abort
I’m not sure which one the import -fFX or the dtrace
This command can take hours to return. Give it time, plenty of time.
Maybe use a tmux session.
Well it seems that saving my data won’t happen. I do have backups and have access to almost all of what I need. I also hope that the data I posted can give some insights.
I believe that when my core 13.0 system went into an endless boot loop, that the boot-pool somehow got corrupted. The were several files open from Pool2 so it probably also got corrupted. I’ve been using FreeNAS which became TrueNAS for at least 15 years and this is the first real problem I’ve had. I had time available so I wanted to try all options to recover.
Now I have to wipe and create a new pool and I have to make a choice which platform. There were several things I liked about Scale 24.10.2.1 and I think Linux is the direction TrueNAS is headed.
What are your thoughts!
Also I appreciate all of the time and effort everyone put in to truly try to help me. Thank You! Thank You! Thank You!!
That wouldn’t corrupt a pool or make it unimportable. You can kill the power of a running ZFS server while it is in the middle of writing data, yet it still won’t corrupt a pool or make it unimportable.
Why do you say this? What happened with the recovery import attempt? Did it return anything after some hours? Do you try it in a tmux session so that you don’t need to leave the window open?
The recovery import -fFX ran for about 20-30 minutes and then the ssh session aborted. When that happened previously I always got a message in the ssh window. That didn’t happen yesterday (I ran the import -fFX twice). I posted above the dtrace that ran until the session aborted. One of the posts I made yesterday stated the pool data was corrupted. I left both ssh windows open overnight and what I posted above from the dtrace was ALL that posted.
If you think there is a chance to recover the data I’m all in. I just need directions and specific commands to use. I trust you guys.
Using tmux would prevent it from dying if you lose the SSH session or close the window.
Do you mean that the command eventually finished and returned that the pool was not importable?
No, when the ssh window aborted there was no more output, the dtrace was still open and this morning there was no additional data.
I don’t know how to use tmux
Also Core 13.3 has a short autologoff with no setting that can be changed
You don’t use the GUI.
Login to the server with SSH.
Start a new tmux session named “import” with:
tmux new -t import
Run the emergency import in this session.
Detach from the session with CTRL + B, then press D.
Start another tmux session named “dtrace” with:
tmux new -t dtrace
Run the dtrace command in this session.
Detach from the session with CTRL + B, then press D.
Now you have two tmux sessions running (named “import” and “dtrace”), which you can list with:
tmux ls
To attach to one or the other, use this:
tmux a -t <name>
Like this:
tmux a -t dtrace
or
tmux a -t import
To detatch again, just do this:
CTRL + B, then press D.
do I continue with the same single disk connected or switch back to the first disk I tried or both disks
Maybe both drives, on the chance that ZFS metadata is corrupt on one or the other?
@HoneyBadger does that sound okay?
Try to do the emergency -FfX in a tmux session with both drives plugged in, and a dtrace in the other tmux session?
Did the previous dtrace show any hints?