TrueNAS stuck with some services after a disk has been replaced

This parameter will cause ZFS to ignore a whole lot of errors, which might be worse than just allowing a rollback.

Did you ever run the zpool import -f -F FreeNAS command to attempt to roll back to a previous transaction?

Did you ever run the zpool import -f -F FreeNAS command to attempt to roll back to a previous transaction?

Yes I tried, but got only a prompt that never came back…

How long did you wait after the command zpool import -f -F FreeNAS or are you still waiting? I’m not sure if this is one of those things which could take time, I suspect it would as it looks for a good restore point.

Sorry to abandon you earlier, I didn’t see any notifications that you had posted.

To make a copy of your TrueNAS configuration files:

  1. Open the TrueNAS GUI
  2. System
  3. General Settings
  4. Manage Configuration
  5. Download File
  6. Export Secret Seed checked.
  7. Save

This will download a copy of your configuration as a .tar file. Ensure you actually have it and can open it. Hopefully you will not need to use it.

I don’t recall, do you have a copy of your files? Maybe while the pool is in Read Only, you could grab anything important, just in case. But with the possibility of errors in your data, you might be better off to see what @HoneyBadger has to say next. He’s a pretty smart badger.

How long did you wait after the command zpool import -f -F FreeNAS or are you still waiting?

Lot of hours… I remember trying that more than once and one attempt, about a night… My print screen taken 2 hours ago, show that it still the same now.
Thanks for the way to backup the configuration files, in the worst case scenario, I’ll maybe destroy the pool and start a new one with the same config !?!?

Sorry to hear that. Let’s see what the badger has to say.

2 Likes

You may need to open a separate or additional shell session through SSH so that you can monitor the processes and log files.

If zfs_recover allows for a mount in read-only, the hope is that the file /proc/spl/kstat/zfs/dbgmsg has some manner of detail about what previously-fatal and now-ignored errors were blocking the mount.

Warning: If you enable zfs_recover and importing without the readonly parameter then you’ll be telling ZFS that whatever errors are in place are acceptable. For corrupted data this might be okay and you can restore whatever was lost from backup - but for corrupted metadata this can result in a permanently unusable pool.

the hope is that the file /proc/spl/kstat/zfs/dbgmsg has some manner of detail about what previously-fatal and now-ignored errors were blocking the mount

The file exist but it’s little big to copy-paste here… Is there any grep I can do to filter only the good interest things? I see that there METADATA got 0 errors but some DATA errors…

You may need to open a separate or additional shell session through SSH so that you can monitor the processes and log files.

I don’t know what is the log file…
But I tried to import without readonly but I put an & at the end of my command, so I get the prompt but, it freezes anyway and the process can’t be killed with a kill -9…
And I also got this from the kernel :
Jun 12 11:12:16 truenas kernel: WARNING: Pool ‘FreeNAS’ has encountered an uncorrectable I/O failure and has been suspended.

You might be able to paste it into a text file and then attach said text file - we really only need the most recent lines from the last import attempts. Even a tail -n 200 might do it, unless it’s just flooded with metaslab_load and _unload statements.

ok there what I got with

tail -n 200 /proc/spl/kstat/zfs/dbgmsg | grep -v metaslab

root@truenas[~]# tail -n 200 /proc/spl/kstat/zfs/dbgmsg | grep -v metaslab
timestamp message
1749752907 ffff92cb665b9840 spa.c:6928:spa_tryimport(): spa_tryimport: importing boot-pool
1749752907 ffff92cb665b9840 spa_misc.c:429:spa_load_note(): spa_load($import-ffff92cb665b9840-boot-pool, config trusted): LOADING
1749752907 ffff92cb6646e100 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-id/ata-SanDisk_SSD_U100_32GB_130646400964-part2’: probe done, cant_read=0 cant_write=1
1749752907 ffff92cb665b9840 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-id/ata-SanDisk_SSD_U100_32GB_130646400964-part2’: best uberblock found for spa $import-ffff92cb665b9840-boot-pool. txg 5251323
1749752907 ffff92cb665b9840 spa_misc.c:429:spa_load_note(): spa_load($import-ffff92cb665b9840-boot-pool, config untrusted): using uberblock with txg=5251323
1749752907 ffff92cb6646e100 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-id/ata-SanDisk_SSD_U100_32GB_130646400964-part2’: probe done, cant_read=0 cant_write=1
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading checkpoint txg
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading indirect vdev metadata
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Checking feature flags
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading special MOS directories
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading properties
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading AUX vdevs
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading vdev metadata
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading dedup tables
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading BRT
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Verifying Log Devices
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Verifying pool data
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Calculating deflated space
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Starting import
1749752907 ffff92cb665b9840 spa.c:9245:spa_async_request(): spa=$import-ffff92cb665b9840-boot-pool async request task=2048
1749752907 ffff92cb665b9840 spa_misc.c:429:spa_load_note(): spa_load($import-ffff92cb665b9840-boot-pool, config trusted): LOADED
1749752907 ffff92cb665b9840 spa_misc.c:429:spa_load_note(): spa_load($import-ffff92cb665b9840-boot-pool, config trusted): UNLOADING
1749752907 ffff92cb665b9840 spa.c:6773:spa_import(): spa_import: importing boot-pool
1749752907 ffff92cb665b9840 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): LOADING
1749752907 ffff92cb665e0000 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-id/ata-SanDisk_SSD_U100_32GB_130646400964-part2’: probe done, cant_read=0 cant_write=1
1749752907 ffff92cb665b9840 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-id/ata-SanDisk_SSD_U100_32GB_130646400964-part2’: best uberblock found for spa boot-pool. txg 5251323
1749752907 ffff92cb665b9840 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config untrusted): using uberblock with txg=5251323
1749752907 ffff92cb665e0000 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-id/ata-SanDisk_SSD_U100_32GB_130646400964-part2’: probe done, cant_read=0 cant_write=0
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading checkpoint txg
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading indirect vdev metadata
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Checking feature flags
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading special MOS directories
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading properties
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading AUX vdevs
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading vdev metadata
1749752907 ffff92cb665b9840 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): Read 3 log space maps (3 total blocks - blksz = 131072 bytes) in 1 ms
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading dedup tables
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Loading BRT
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Verifying Log Devices
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Verifying pool data
1749752907 ffff92cb665b9840 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): spa_load_verify found 0 metadata errors and 6 data errors
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Calculating deflated space
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Starting import
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Claiming ZIL blocks
1749752907 ffff92cb665b9840 mmp.c:239:mmp_thread_start(): MMP thread started pool ‘boot-pool’ gethrtime 8160508735
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Syncing ZIL claims
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Updating configs
1749752907 ffff92cb665b9840 spa.c:9245:spa_async_request(): spa=boot-pool async request task=1
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Starting resilvers
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Restarting device removals
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Cleaning up inconsistent objsets
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Cleaning up temporary userrefs
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Restarting initialize
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Restarting TRIM
1749752907 ffff92cb665b9840 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘boot-pool’ Finished importing
1749752907 ffff92cb665b9840 spa.c:9245:spa_async_request(): spa=boot-pool async request task=2048
1749752907 ffff92cb665b9840 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): LOADED
1749752907 ffff92cb665cc8c0 spa_history.c:306:spa_history_log_sync(): txg 5251325 open pool version 5000; software version zfs-2.1.99-3167-g1e0563f82; uts (none) 6.12.15-production+truenas #1 SMP PREEMPT_DYNAMIC Mon May 26 13:44:31 UTC 2025 x86_64
1749752907 ffff92cb665b9840 spa.c:9245:spa_async_request(): spa=boot-pool async request task=32
1749752907 ffff92cb665cc8c0 spa_history.c:306:spa_history_log_sync(): txg 5251327 import pool version 5000; software version zfs-2.1.99-3167-g1e0563f82; uts (none) 6.12.15-production+truenas #1 SMP PREEMPT_DYNAMIC Mon May 26 13:44:31 UTC 2025 x86_64
1749752909 ffff92cb665cc8c0 spa_history.c:293:spa_history_log_sync(): command: zpool import -N -f boot-pool
1749754805 ffff92cb6106e100 spa.c:6928:spa_tryimport(): spa_tryimport: importing FreeNAS
1749754805 ffff92cb6106e100 spa_misc.c:429:spa_load_note(): spa_load($import-ffff92cb6106e100-FreeNAS, config trusted): LOADING
1749754805 ffff92cb60b61840 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/4bc11e54-f900-428c-ad19-a9d2433d1401’: probe done, cant_read=0 cant_write=1
1749754805 ffff92cb60b61840 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/0d391935-b9b7-4d39-a6ca-04d54781f7ce’: probe done, cant_read=0 cant_write=1
1749754806 ffff92cb60b61840 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/c97fab7f-8a17-4c56-8e98-201619a832a7’: probe done, cant_read=0 cant_write=1
1749754806 ffff92cb6106e100 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/4bc11e54-f900-428c-ad19-a9d2433d1401’: best uberblock found for spa $import-ffff92cb6106e100-FreeNAS. txg 748075
1749754806 ffff92cb6106e100 spa_misc.c:429:spa_load_note(): spa_load($import-ffff92cb6106e100-FreeNAS, config untrusted): using uberblock with txg=748075
1749754806 ffff92cb60b61840 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/4bc11e54-f900-428c-ad19-a9d2433d1401’: probe done, cant_read=0 cant_write=1
1749754806 ffff92cb60b61840 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/c97fab7f-8a17-4c56-8e98-201619a832a7’: probe done, cant_read=0 cant_write=1
1749754806 ffff92cb60b61840 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/0d391935-b9b7-4d39-a6ca-04d54781f7ce’: probe done, cant_read=0 cant_write=1
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading checkpoint txg
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading indirect vdev metadata
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Checking feature flags
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading special MOS directories
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading properties
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading AUX vdevs
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading vdev metadata
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading dedup tables
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading BRT
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Verifying Log Devices
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Verifying pool data
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Calculating deflated space
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Starting import
1749754806 ffff92cb6106e100 spa.c:9245:spa_async_request(): spa=$import-ffff92cb6106e100-FreeNAS async request task=2048
1749754806 ffff92cb6106e100 spa_misc.c:429:spa_load_note(): spa_load($import-ffff92cb6106e100-FreeNAS, config trusted): LOADED
1749754806 ffff92cb6106e100 spa_misc.c:429:spa_load_note(): spa_load($import-ffff92cb6106e100-FreeNAS, config trusted): UNLOADING
1749754806 ffff92cb6106e100 spa.c:6775:spa_import(): spa_import: importing FreeNAS, max_txg=-1 (RECOVERY MODE)
1749754806 ffff92cb6106e100 spa_misc.c:429:spa_load_note(): spa_load(FreeNAS, config trusted): LOADING
1749754806 ffff92cb51b2b080 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/4bc11e54-f900-428c-ad19-a9d2433d1401’: probe done, cant_read=0 cant_write=1
1749754806 ffff92cb51b2b080 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/c97fab7f-8a17-4c56-8e98-201619a832a7’: probe done, cant_read=0 cant_write=1
1749754806 ffff92cb51b2b080 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/0d391935-b9b7-4d39-a6ca-04d54781f7ce’: probe done, cant_read=0 cant_write=1
1749754806 ffff92cb6106e100 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/4bc11e54-f900-428c-ad19-a9d2433d1401’: best uberblock found for spa FreeNAS. txg 748075
1749754806 ffff92cb6106e100 spa_misc.c:429:spa_load_note(): spa_load(FreeNAS, config untrusted): using uberblock with txg=748075
1749754806 ffff92cb51b2b080 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/c97fab7f-8a17-4c56-8e98-201619a832a7’: probe done, cant_read=0 cant_write=0
1749754806 ffff92cb51b2b080 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/4bc11e54-f900-428c-ad19-a9d2433d1401’: probe done, cant_read=0 cant_write=0
1749754806 ffff92cb51b2b080 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/0d391935-b9b7-4d39-a6ca-04d54781f7ce’: probe done, cant_read=0 cant_write=0
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading checkpoint txg
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading indirect vdev metadata
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Checking feature flags
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading special MOS directories
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading properties
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading AUX vdevs
1749754806 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading vdev metadata
1749754808 ffff92cb6106e100 spa_misc.c:429:spa_load_note(): spa_load(FreeNAS, config trusted): Read 6 log space maps (6 total blocks - blksz = 131072 bytes) in 8 ms
1749754808 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading dedup tables
1749754808 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Loading BRT
1749754808 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Verifying Log Devices
1749754808 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Verifying pool data
1749754808 ffff92cb6106e100 spa_misc.c:429:spa_load_note(): spa_load(FreeNAS, config trusted): spa_load_verify found 0 metadata errors and 3 data errors
1749754808 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Calculating deflated space
1749754808 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Starting import
1749754808 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Claiming ZIL blocks
1749754808 ffff92cb6106e100 mmp.c:239:mmp_thread_start(): MMP thread started pool ‘FreeNAS’ gethrtime 1909677300991
1749754808 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Syncing ZIL claims
root@truenas[~]#

Don’t filter anything with grep, in case we do get useful info in a line with metaslab present, but I do see

1749754808 ffff92cb6106e100 spa_misc.c:429:spa_load_note(): spa_load(FreeNAS, config trusted): spa_load_verify found 0 metadata errors and 3 data errors

And then we choke after these lines:

1749754808 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Claiming ZIL blocks
1749754808 ffff92cb6106e100 mmp.c:239:mmp_thread_start(): MMP thread started pool ‘FreeNAS’ gethrtime 1909677300991
1749754808 ffff92cb6106e100 spa_misc.c:2376:spa_import_progress_set_notes_impl(): ‘FreeNAS’ Syncing ZIL claims

It looks like it’s experiencing issues trying to replay the ZIL … -F allows rollback to discard the ZIL but it might be hitting the I/O error in the active one and hanging something there - which is likely why importing -o readonly=on works, because it won’t be replaying the ZIL in that mode.

You may need to forcibly restart the system to stop the hung import process.

After that, we might need to specify a transaction in the past to stop it from stalling. Since it’s trying to import at 748075 now, let’s just N-1 manually:

zpool import -fFXT 748074 FreeNAS -R /mnt -n

The separate lowercase -n is telling it to “attempt import, but don’t actually do it” - if it returns to a prompt, and the dbgmsg doesn’t show contraindications, you should be able to drop the -n and import it for real.

2 Likes

I tried the command but it hang… :frowning:

But my dbgmsg show this:

1749754806 ffff92cb6106e100 vdev.c:183:vdev_dbgmsg(): disk vdev ‘/dev/disk/by-partuuid/4bc11e54-f900-428c-ad19-a9d2433d1401’: best uberblock found for spa $import-ffff92cb6106e100-FreeNAS. txg 748075

I can confirm that the disk 4bc11e54-f900-428c-ad19-a9d2433d1401 is out of the pool.
It’s the best clue I can find…

zpool import -fFXT 748074 FreeNAS -R /mnt -n

I remember when I remove my old disk, try to add the new one before re-silvering the pool. What if I use N-2 so 748073 ?

The import process should be automatically “walking backwards” from the targeted TXG. Even the -n hangs and doesn’t go further than that line in your dbgmsg log?

Even the -n hangs and doesn’t go further

Correct… :frowning:

I’m puzzled that it tried to go with entering at 748075 when we explicitly told it to start at -T 748074 instead

Have you rebooted the system since the last import attempt with zfs_recover enabled?

Have you rebooted the system since the last import attempt with zfs_recover enabled?

Yess many times… Is it matter ?

This thread can be deleted… I decided to start from scratch…
Thanks to all who really tried to help!