Cannot Import Pool to new system after hardware failure

chereszabor · October 16, 2025, 4:02am

Hello,
My primary truenas system failed due to a hardware fault, while it may have been resilvering one or two of the drives.

I have a raidz2 with a log drive, so assume the pool was still functional after the crash.

I moved all the relevant drives into a new system, and tried to import the pool, however I get the following error:

I’ve tried to import the pool via the WebUI and with the -f and -fF flags in cli, in all cases i get the following:

Failed to import 'FNMedia' pool: cannot import 'FNMedia' as 'FNMedia': one or more devices is currently unavailable

If I run zpool import, I get the following:

root@truenas[~]# zpool import
  pool: FNMedia
    id: 16375726539085097560
 state: DEGRADED
status: One or more devices were being resilvered.
action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
config:

        FNMedia                                   DEGRADED
          raidz2-0                                ONLINE
            68239595-97a6-11ed-b959-ac1f6b483888  ONLINE
            67e1a3fa-97a6-11ed-b959-ac1f6b483888  ONLINE
            685381f3-97a6-11ed-b959-ac1f6b483888  ONLINE
            68089bab-97a6-11ed-b959-ac1f6b483888  ONLINE
            683c9144-97a6-11ed-b959-ac1f6b483888  ONLINE
            683f34aa-97a6-11ed-b959-ac1f6b483888  ONLINE
            f23f9038-da26-4973-952c-9e839b3e594b  ONLINE
            efb8e117-7881-48fb-9226-f3d6f8171109  ONLINE
        cache
          66dd4eea-97a6-11ed-b959-ac1f6b483888
        logs
          66ad85b6-97a6-11ed-b959-ac1f6b483888    UNAVAIL

If running If I run zpool import -f -m , I get the following:

cannot import 'FNMedia': I/O error
        Destroy and re-create the pool from
        a backup source.

This is a very large pool and though i have backups of the critical data, I would prefer not to recreate it. Any Assistance would be appriciated.

Farout · October 16, 2025, 5:14am

What does
zpool status -v show ?

What was the log device ?

Bare metal or virtualised? Pls post your complete setup.

etorix · October 16, 2025, 6:31am

Error messages are clear: You need to put back the SLOG drive.

Arwen · October 16, 2025, 9:04am

Actually not. The pool import option “-m” allows doing imports with missing log devices. If their was pending data, it will be lost.

 -m    Allows a pool to import when there is a missing log device.  Recent transactions can be lost
       because the log device will be discarded.

What does the following command say for each of this pools devices?

zdb -l /dev/XXX | egrep 'txg|labels'

Replace XXX with the appropriate disk and the pool’s partition.

chereszabor · October 16, 2025, 1:20pm

Both systems are bare metal

I’ve also been able to reattach the log device. Though still getting an I/O error during import attempt.

root@truenas[~]# zpool import
  pool: FNMedia
    id: 16375726539085097560
 state: ONLINE
status: One or more devices were being resilvered.
action: The pool can be imported using its name or numeric identifier.
config:

        FNMedia                                   ONLINE
          raidz2-0                                ONLINE
            68239595-97a6-11ed-b959-ac1f6b483888  ONLINE
            67e1a3fa-97a6-11ed-b959-ac1f6b483888  ONLINE
            685381f3-97a6-11ed-b959-ac1f6b483888  ONLINE
            68089bab-97a6-11ed-b959-ac1f6b483888  ONLINE
            683c9144-97a6-11ed-b959-ac1f6b483888  ONLINE
            683f34aa-97a6-11ed-b959-ac1f6b483888  ONLINE
            f23f9038-da26-4973-952c-9e839b3e594b  ONLINE
            efb8e117-7881-48fb-9226-f3d6f8171109  ONLINE
        cache
          66dd4eea-97a6-11ed-b959-ac1f6b483888
        logs
          66ad85b6-97a6-11ed-b959-ac1f6b483888    ONLINE

For zdb -l /dev/XXX | egrep ‘txg|labels’, here are my results:

root@truenas[~]# zdb -l /dev/sda2 | egrep 'txg|labels'
    txg: 16951679
        raidz_expand_txgs[0]: 16777575
        raidz_expand_txgs[1]: 16799595
        create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdb2 | egrep 'txg|labels'
    txg: 16951677
        raidz_expand_txgs[0]: 16777575
        raidz_expand_txgs[1]: 16799595
        create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdc2 | egrep 'txg|labels'
    txg: 16951679
        raidz_expand_txgs[0]: 16777575
        raidz_expand_txgs[1]: 16799595
        create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdd2 | egrep 'txg|labels'
    txg: 16951677
        raidz_expand_txgs[0]: 16777575
        raidz_expand_txgs[1]: 16799595
        create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sde2 | egrep 'txg|labels'
    txg: 16951679
        raidz_expand_txgs[0]: 16777575
        raidz_expand_txgs[1]: 16799595
        create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdf2 | egrep 'txg|labels'
    txg: 16951679
        raidz_expand_txgs[0]: 16777575
        raidz_expand_txgs[1]: 16799595
        create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdg1 | egrep 'txg|labels' 
    txg: 16951679
        raidz_expand_txgs[0]: 16777575
        raidz_expand_txgs[1]: 16799595
        create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdj1 | egrep 'txg|labels'
    txg: 16951679
        raidz_expand_txgs[0]: 16777575
        raidz_expand_txgs[1]: 16799595
        create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
            create_txg: 4
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdi1 | egrep 'txg|labels'
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/nvme0n1p1 | egrep 'txg|labels' 
    txg: 16951677
        create_txg: 4
    labels = 0 1 2 3

etorix · October 16, 2025, 1:41pm

And with zpool import FNMedia ?

chereszabor · October 16, 2025, 2:08pm

root@truenas[~]# zpool import FNMedia
cannot import 'FNMedia': pool was previously in use from another system.
Last accessed by truenas (hostid=273b7a88) at Wed Oct 15 16:47:06 2025
The pool can be imported, use 'zpool import -f' to import the pool.

with -f the result is:

root@truenas[~]# zpool import -f FNMedia
cannot import 'FNMedia': I/O error
        Destroy and re-create the pool from
        a backup source.

etorix · October 16, 2025, 2:22pm

Some drives are 2 txg ahead of the others.
Pending better advice from @Arwen or @HoneyBadger I’d suggest to try a dry run
zpool import -fFn FNMedia
and if there are no errors (no output) to remove the -n flag.

Is this all media storage (with a SLOG? NFS?)? No zvol which could get corrupted by losing ten seconds of the last transactions?

chereszabor · October 16, 2025, 2:28pm

Appreciate the input.

zpool import -fFn FNMedia does not return any errors or any information at al.

And yes, this is a purely media storage pool, there was some data that was being copied at the time of failure via replication, but that can be copied back in the case of corruption.

HoneyBadger · October 16, 2025, 2:30pm

2 TXGs should be within the window of acceptable rewind for the -fF import.

@chereszabor since -fFn did not return any errors let’s try just -fF and see if it throws up a problem at you.

If it doesn’t throw an error, then zpool export your pool from the CLI and reboot - it should hopefully grab it from middleware then.

If it does throw an error, let’s see the output of the last 100-200 lines of /proc/spl/kstat/zfs/dbgmsg between code blocks (triple backticks " ``` ") or use the “preformatted text” option </> ?

chereszabor · October 16, 2025, 3:10pm

with -fF I get the following:

root@truenas[~]# zpool import -fF FNMedia 
cannot import 'FNMedia': one or more devices is currently unavailable

for dbgmsg i have the following

760623704   ffff9c83b3bf8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83b3bf8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83b3bf8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83b3bf8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83b3bf8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83b3bf8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83b3bf8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83b3bf8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83b3bf8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760623704   ffff9c8314b24c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951677
1760623704   ffff9c8314b24c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951677)
1760623704   ffff9c8314b24c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760623704   ffff9c8314b24c80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951677
1760623704   ffff9c8314b24c80 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760623704   ffff9c8314b24c80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING
1760623704   ffff9c8314b24c80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): spa_load_retry: rewind, max txg: 16951676
1760623704   ffff9c8314b24c80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): LOADING
1760623704   ffff9c83e2cf4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83e2cf4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83e2cf4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83e2cf4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83e2cf4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83e2cf4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83e2cf4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83e2cf4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c83e2cf4c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760623704   ffff9c8314b24c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951676
1760623704   ffff9c8314b24c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951676)
1760623704   ffff9c8314b24c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760623704   ffff9c8314b24c80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951676
1760623704   ffff9c8314b24c80 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760623704   ffff9c8314b24c80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING
1760624686   ffff9ca2a3d13300 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1760624686   ffff9ca2a3d13300 spa.c:6928:spa_tryimport(): spa_tryimport: importing FNMedia
1760624686   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ca2a3d13300-FNMedia, config trusted): LOADING
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa $import-ffff9ca2a3d13300-FNMedia. txg 16951679
1760624686   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ca2a3d13300-FNMedia, config untrusted): using uberblock with txg=16951679
1760624686   ffff9c8314a3e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 1a1eba50000
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading checkpoint txg
1760624686   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading indirect vdev metadata
1760624686   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Checking feature flags
1760624686   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading special MOS directories
1760624686   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading properties
1760624686   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading AUX vdevs
1760624686   ffff9ca2880b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66dd4eea-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624686   ffff9ca2a3d13300 spa.c:9245:spa_async_request(): spa=$import-ffff9ca2a3d13300-FNMedia async request task=4096
1760624686   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading vdev metadata
1760624686   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading dedup tables
1760624687   ffff9c83149ee600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 1a1ed0e5000
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading BRT
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying Log Devices
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying pool data
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Calculating deflated space
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Starting import
1760624687   ffff9ca2a3d13300 spa.c:9245:spa_async_request(): spa=$import-ffff9ca2a3d13300-FNMedia async request task=2048
1760624687   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ca2a3d13300-FNMedia, config trusted): LOADED
1760624687   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ca2a3d13300-FNMedia, config trusted): UNLOADING
1760624687   ffff9ca2a3d13300 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1760624687   ffff9ca2a3d13300 spa.c:6773:spa_import(): spa_import: importing FNMedia
1760624687   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): LOADING
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760624687   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951679
1760624687   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951679
1760624687   ffff9c83e26b8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 1a1eba50000
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=0
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=0
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading checkpoint txg
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading indirect vdev metadata
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Checking feature flags
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading special MOS directories
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading properties
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading AUX vdevs
1760624687   ffff9c84098b9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66dd4eea-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760624687   ffff9ca2a3d13300 spa.c:9245:spa_async_request(): spa=FNMedia async request task=4096
1760624687   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading vdev metadata
1760624702   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): Read 188 log space maps (188 total blocks - blksz = 131072 bytes) in 1044 ms
1760624702   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading dedup tables
1760624718   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading BRT
1760624718   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying Log Devices
1760624719   ffff9ca2a3d13300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying pool data
1760624727   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): spa_load_verify found 7 metadata errors and 0 data errors
1760624727   ffff9ca2a3d13300 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config trusted): FAILED: spa_load_verify failed [error=5]
1760624727   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): UNLOADING
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): spa_load_retry: rewind, max txg: 16951678
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): LOADING
1760624729   ffff9ca2a0181980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a0181980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a0181980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a0181980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a0181980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a0181980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a0181980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a0181980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a0181980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951678
1760624729   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951678)
1760624729   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951678
1760624729   ffff9ca2a3d13300 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): spa_load_retry: rewind, max txg: 16951677
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): LOADING
1760624729   ffff9c8437a61980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c8437a61980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c8437a61980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c8437a61980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c8437a61980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c8437a61980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c8437a61980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c8437a61980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760624729   ffff9c8437a61980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951677
1760624729   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951677)
1760624729   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951677
1760624729   ffff9ca2a3d13300 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): spa_load_retry: rewind, max txg: 16951676
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): LOADING
1760624729   ffff9c84098bb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c84098bb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c84098bb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c84098bb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c84098bb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c84098bb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9c84098bb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760624729   ffff9c84098bb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760624729   ffff9c84098bb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760624729   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951676
1760624729   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951676)
1760624729   ffff9ca2a3d13300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951676
1760624729   ffff9ca2a3d13300 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760624729   ffff9ca2a3d13300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING
1760627210   ffff9ca2bc87b300 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1760627210   ffff9ca2bc87b300 spa.c:6928:spa_tryimport(): spa_tryimport: importing FNMedia
1760627210   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ca2bc87b300-FNMedia, config trusted): LOADING
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa $import-ffff9ca2bc87b300-FNMedia. txg 16951679
1760627210   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ca2bc87b300-FNMedia, config untrusted): using uberblock with txg=16951679
1760627210   ffff9c8437a88000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 1a1eba50000
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading checkpoint txg
1760627210   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading indirect vdev metadata
1760627210   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Checking feature flags
1760627210   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading special MOS directories
1760627210   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading properties
1760627210   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading AUX vdevs
1760627210   ffff9ca2bc756600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66dd4eea-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627210   ffff9ca2bc87b300 spa.c:9245:spa_async_request(): spa=$import-ffff9ca2bc87b300-FNMedia async request task=4096
1760627210   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading vdev metadata
1760627210   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading dedup tables
1760627211   ffff9ca2bc663300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 1a1ed0e5000
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading BRT
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying Log Devices
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying pool data
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Calculating deflated space
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Starting import
1760627211   ffff9ca2bc87b300 spa.c:9245:spa_async_request(): spa=$import-ffff9ca2bc87b300-FNMedia async request task=2048
1760627211   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ca2bc87b300-FNMedia, config trusted): LOADED
1760627211   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ca2bc87b300-FNMedia, config trusted): UNLOADING
1760627211   ffff9ca2bc87b300 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1760627211   ffff9ca2bc87b300 spa.c:6775:spa_import(): spa_import: importing FNMedia, max_txg=-1 (RECOVERY MODE)
1760627211   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): LOADING
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760627211   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951679
1760627211   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951679
1760627211   ffff9ca2b013b300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 1a1eba50000
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=0
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=0
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading checkpoint txg
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading indirect vdev metadata
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Checking feature flags
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading special MOS directories
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading properties
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading AUX vdevs
1760627211   ffff9ca2b013e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66dd4eea-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=0
1760627211   ffff9ca2bc87b300 spa.c:9245:spa_async_request(): spa=FNMedia async request task=4096
1760627211   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading vdev metadata
1760627226   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): Read 188 log space maps (188 total blocks - blksz = 131072 bytes) in 1108 ms
1760627226   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading dedup tables
1760627242   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading BRT
1760627242   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying Log Devices
1760627242   ffff9ca2bc87b300 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying pool data
1760627251   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): spa_load_verify found 7 metadata errors and 0 data errors
1760627251   ffff9ca2bc87b300 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config trusted): FAILED: spa_load_verify failed [error=5]
1760627251   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): UNLOADING
1760627252   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): spa_load_retry: rewind, max txg: 16951678
1760627252   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): LOADING
1760627252   ffff9c8482764c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627252   ffff9c8482764c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760627252   ffff9c8482764c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760627252   ffff9c8482764c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627252   ffff9c8482764c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627252   ffff9c8482764c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627252   ffff9c8482764c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627252   ffff9c8482764c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9c8482764c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951678
1760627253   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951678)
1760627253   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760627253   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951678
1760627253   ffff9ca2bc87b300 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760627253   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING
1760627253   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): spa_load_retry: rewind, max txg: 16951677
1760627253   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): LOADING
1760627253   ffff9ca34a726600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca34a726600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca34a726600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca34a726600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca34a726600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca34a726600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca34a726600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca34a726600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca34a726600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951677
1760627253   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951677)
1760627253   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760627253   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951677
1760627253   ffff9ca2bc87b300 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760627253   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING
1760627253   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): spa_load_retry: rewind, max txg: 16951676
1760627253   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): LOADING
1760627253   ffff9ca2a2a4e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2a2a4e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2a2a4e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2a2a4e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2a2a4e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2a2a4e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2a2a4e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2a2a4e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2a2a4e600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760627253   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951676
1760627253   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951676)
1760627253   ffff9ca2bc87b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760627253   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951676
1760627253   ffff9ca2bc87b300 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760627253   ffff9ca2bc87b300 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING

chereszabor · October 17, 2025, 12:30pm

@HoneyBadger and @Arwen,
Any thoughts on the dbgmsg output? can’t really make heads or tails of it…

Thanks!

etorix · October 17, 2025, 1:10pm

ZFS tries to rewind to 16951677, fails, goes one step further to 16951676 and again complains that this is too far from 16951679. Which is… puzzling, there should be überblocks available for these small rewinds.
From the above labels, 16951677 should be good to go but I’d prefer to have a second opinion before resorting to the mosT eXtremes measures.

chereszabor · October 17, 2025, 1:42pm

Not sure if this helps, but i did throw the log into Claude, it identified Disk 3 as having some corruption, which tracks with the fact that a drive dropped from the array and was being resilvered at the time of the crash…

Arwen · October 18, 2025, 4:59am

If the pool won’t rewind to a common TXG, then I don’t understand the problem well enough to give good advice.

Some bad advice could be to remove the SLOG / Log device, remove the 2 disks with the older TXGs, (sdb, sdd), and then try to import the pool. I’d try Read Only first:

zpool import -fm -o readonly=on FNMedia

A test would be to read all the data. Not sure you can run a Scrub on a R/O pool. But, a simple find and cat file >/dev/null would likely read much of the pool. Then check for errors in zpool status FNMedia.

If that worked and the data seemed intact, you could then see about:

Backing up the data
Exporting the pool
Re-import the pool read / write
Re-verify the pool seems healthy
If healthy, wipe the removed disks, and use them for replacing the missing disks

Importing the pool Read Write will likely cause the pool to move forward on TXGs, which will make any recovery with the missing disks harder. So, it’s a case of last resort. I take ZERO responsibility for any outcome, (well, I will take praise for a good outcome…) A better idea would be using a paid recovery service, or recovery tool like Klennet:

chereszabor · October 18, 2025, 2:02pm

Tried the drive pull method, that didn’t yield any results unfortunately. Still getting an I/O error.
dbgmsg output below:

760794866   ffff8e77621dcc80 spa.c:9245:spa_async_request(): spa=boot-pool async request task=32
1760794866   ffff8e7760fce600 spa_history.c:306:spa_history_log_sync(): txg 11783 import pool version 5000; software version zfs-2.1.99-3167-gbf2b1f142; uts (none) 6.12.15-production+truenas #1 SMP PREEMPT_DYNAMIC Mon Sep  8 18:50:34 UTC 2025 x86_64
1760794867   ffff8e7760fce600 spa_history.c:293:spa_history_log_sync(): command: zpool import -N -f boot-pool
1760795351   ffff8e96d33f8000 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1760795351   ffff8e96d33f8000 spa.c:6928:spa_tryimport(): spa_tryimport: importing FNMedia
1760795351   ffff8e96d33f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e96d33f8000-FNMedia, config trusted): LOADING
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795351   ffff8e96d33f8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa $import-ffff8e96d33f8000-FNMedia. txg 16951679
1760795351   ffff8e96d33f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e96d33f8000-FNMedia, config untrusted): using uberblock with txg=16951679
1760795351   ffff8e7763459980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 1a1eba50000
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795351   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading checkpoint txg
1760795351   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading indirect vdev metadata
1760795351   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Checking feature flags
1760795351   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading special MOS directories
1760795351   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading properties
1760795351   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading AUX vdevs
1760795351   ffff8e77a0459980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66dd4eea-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795351   ffff8e96d33f8000 spa.c:9245:spa_async_request(): spa=$import-ffff8e96d33f8000-FNMedia async request task=4096
1760795351   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading vdev metadata
1760795351   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading dedup tables
1760795352   ffff8e96d4614c80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 1a1ed0e5000
1760795352   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading BRT
1760795352   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying Log Devices
1760795352   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying pool data
1760795352   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Calculating deflated space
1760795352   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Starting import
1760795352   ffff8e96d33f8000 spa.c:9245:spa_async_request(): spa=$import-ffff8e96d33f8000-FNMedia async request task=2048
1760795352   ffff8e96d33f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e96d33f8000-FNMedia, config trusted): LOADED
1760795352   ffff8e96d33f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e96d33f8000-FNMedia, config trusted): UNLOADING
1760795405   ffff8e96d33f8000 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1760795405   ffff8e96d33f8000 spa.c:6928:spa_tryimport(): spa_tryimport: importing FNMedia
1760795405   ffff8e96d33f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e96d33f8000-FNMedia, config trusted): LOADING
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96d33f8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa $import-ffff8e96d33f8000-FNMedia. txg 16951679
1760795405   ffff8e96d33f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e96d33f8000-FNMedia, config untrusted): using uberblock with txg=16951679
1760795405   ffff8e7807e54c80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 1a1eba50000
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading checkpoint txg
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading indirect vdev metadata
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Checking feature flags
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading special MOS directories
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading properties
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading AUX vdevs
1760795405   ffff8e96c5c11980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66dd4eea-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795405   ffff8e96d33f8000 spa.c:9245:spa_async_request(): spa=$import-ffff8e96d33f8000-FNMedia async request task=4096
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading vdev metadata
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading dedup tables
1760795405   ffff8e78071d3300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 1a1ed0e5000
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading BRT
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying Log Devices
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying pool data
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Calculating deflated space
1760795405   ffff8e96d33f8000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Starting import
1760795405   ffff8e96d33f8000 spa.c:9245:spa_async_request(): spa=$import-ffff8e96d33f8000-FNMedia async request task=2048
1760795405   ffff8e96d33f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e96d33f8000-FNMedia, config trusted): LOADED
1760795405   ffff8e96d33f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e96d33f8000-FNMedia, config trusted): UNLOADING
1760795430   ffff8e7759669980 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1760795430   ffff8e7759669980 spa.c:6928:spa_tryimport(): spa_tryimport: importing FNMedia
1760795430   ffff8e7759669980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e7759669980-FNMedia, config trusted): LOADING
1760795430   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795430   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795430   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795430   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795430   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795430   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795430   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795431   ffff8e77b198cc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1004642101/1000000000
1760795431   ffff8e77b1989980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1008651411/1000000000
1760795431   ffff8e7759669980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa $import-ffff8e7759669980-FNMedia. txg 16951679
1760795431   ffff8e7759669980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e7759669980-FNMedia, config untrusted): using uberblock with txg=16951679
1760795431   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795431   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795431   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795431   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795431   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795431   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795431   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795432   ffff8e78071f3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1011555901/1000000000
1760795432   ffff8e77b2eb8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1011578203/1000000000
1760795432   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading checkpoint txg
1760795432   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading indirect vdev metadata
1760795432   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Checking feature flags
1760795432   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading special MOS directories
1760795432   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading properties
1760795432   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading AUX vdevs
1760795432   ffff8e96d5fd8000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66dd4eea-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795432   ffff8e7759669980 spa.c:9245:spa_async_request(): spa=$import-ffff8e7759669980-FNMedia async request task=4096
1760795432   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading vdev metadata
1760795433   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading dedup tables
1760795433   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading BRT
1760795433   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying Log Devices
1760795433   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying pool data
1760795433   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Calculating deflated space
1760795433   ffff8e7759669980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Starting import
1760795433   ffff8e7759669980 spa.c:9245:spa_async_request(): spa=$import-ffff8e7759669980-FNMedia async request task=2048
1760795433   ffff8e7759669980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e7759669980-FNMedia, config trusted): LOADED
1760795433   ffff8e7759669980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e7759669980-FNMedia, config trusted): UNLOADING
1760795483   ffff8e77b1bb1980 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1760795483   ffff8e77b1bb1980 spa.c:6928:spa_tryimport(): spa_tryimport: importing FNMedia
1760795483   ffff8e77b1bb1980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e77b1bb1980-FNMedia, config trusted): LOADING
1760795483   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795483   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795483   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795483   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795483   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795483   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795483   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795484   ffff8e7754ec6600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1003075361/1000000000
1760795484   ffff8e7754ec1980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1019063928/1000000000
1760795484   ffff8e77b1bb1980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa $import-ffff8e77b1bb1980-FNMedia. txg 16951679
1760795484   ffff8e77b1bb1980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e77b1bb1980-FNMedia, config untrusted): using uberblock with txg=16951679
1760795484   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795484   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795484   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795484   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795484   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795484   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795484   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795485   ffff8e77a8ab6600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1005526471/1000000000
1760795485   ffff8e7807e54c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1005520979/1000000000
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading checkpoint txg
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading indirect vdev metadata
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Checking feature flags
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading special MOS directories
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading properties
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading AUX vdevs
1760795486   ffff8e96c8018000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66dd4eea-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795486   ffff8e77b1bb1980 spa.c:9245:spa_async_request(): spa=$import-ffff8e77b1bb1980-FNMedia async request task=4096
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading vdev metadata
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading dedup tables
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading BRT
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying Log Devices
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying pool data
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Calculating deflated space
1760795486   ffff8e77b1bb1980 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Starting import
1760795486   ffff8e77b1bb1980 spa.c:9245:spa_async_request(): spa=$import-ffff8e77b1bb1980-FNMedia async request task=2048
1760795486   ffff8e77b1bb1980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e77b1bb1980-FNMedia, config trusted): LOADED
1760795486   ffff8e77b1bb1980 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e77b1bb1980-FNMedia, config trusted): UNLOADING
1760795500   ffff8e77621dcc80 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1760795500   ffff8e77621dcc80 spa.c:6928:spa_tryimport(): spa_tryimport: importing FNMedia
1760795500   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e77621dcc80-FNMedia, config trusted): LOADING
1760795500   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795500   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795500   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795500   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795500   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795500   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795500   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795501   ffff8e96cbe04c80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1006297368/1000000000
1760795501   ffff8e96d0643300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1010320968/1000000000
1760795501   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa $import-ffff8e77621dcc80-FNMedia. txg 16951679
1760795501   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e77621dcc80-FNMedia, config untrusted): using uberblock with txg=16951679
1760795501   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795501   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795501   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795501   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795501   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795501   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795501   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795502   ffff8e775cc9b300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1002631331/1000000000
1760795502   ffff8e77ad38cc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1006689717/1000000000
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading checkpoint txg
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading indirect vdev metadata
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Checking feature flags
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading special MOS directories
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading properties
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading AUX vdevs
1760795502   ffff8e77a72be600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66dd4eea-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795502   ffff8e77621dcc80 spa.c:9245:spa_async_request(): spa=$import-ffff8e77621dcc80-FNMedia async request task=4096
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading vdev metadata
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading dedup tables
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading BRT
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying Log Devices
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Verifying pool data
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Calculating deflated space
1760795502   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Starting import
1760795502   ffff8e77621dcc80 spa.c:9245:spa_async_request(): spa=$import-ffff8e77621dcc80-FNMedia async request task=2048
1760795502   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e77621dcc80-FNMedia, config trusted): LOADED
1760795502   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8e77621dcc80-FNMedia, config trusted): UNLOADING
1760795502   ffff8e77621dcc80 metaslab.c:1687:spa_set_allocator(): spa allocator: dynamic
1760795502   ffff8e77621dcc80 spa.c:6773:spa_import(): spa_import: importing FNMedia
1760795502   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): LOADING
1760795502   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795502   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795502   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795502   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795502   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795502   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795502   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795503   ffff8e775cca9980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1008396834/1000000000
1760795503   ffff8e77b0c19980 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1008420583/1000000000
1760795503   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951679
1760795503   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951679
1760795503   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795503   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795503   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795503   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795503   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795503   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795503   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795504   ffff8e96d4426600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1003679742/1000000000
1760795504   ffff8e96d34b6600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1003677423/1000000000
1760795504   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading checkpoint txg
1760795504   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading indirect vdev metadata
1760795504   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Checking feature flags
1760795505   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading special MOS directories
1760795505   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading properties
1760795505   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading AUX vdevs
1760795505   ffff8e77b19b3300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66dd4eea-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795505   ffff8e77621dcc80 spa.c:9245:spa_async_request(): spa=FNMedia async request task=4096
1760795505   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading vdev metadata
1760795505   ffff8e77621dcc80 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'FNMedia' Loading dedup tables
1760795520   ffff8e77621dcc80 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config trusted): FAILED: ddt_load failed [error=52]
1760795520   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): UNLOADING
1760795520   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): spa_load_retry: rewind, max txg: 16951678
1760795520   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config trusted): LOADING
1760795520   ffff8e96d6306600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795520   ffff8e96d6306600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795520   ffff8e96d6306600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795520   ffff8e96d6306600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795520   ffff8e96d6306600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795520   ffff8e96d6306600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795520   ffff8e96d6306600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795521   ffff8e96d0643300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1001985340/1000000000
1760795521   ffff8e9bce4f0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1009975080/1000000000
1760795521   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951678
1760795521   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951678)
1760795521   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760795521   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951678
1760795521   ffff8e77621dcc80 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760795521   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING
1760795521   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): spa_load_retry: rewind, max txg: 16951677
1760795521   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): LOADING
1760795521   ffff8e96c5f78000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795521   ffff8e96c5f78000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795521   ffff8e96c5f78000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795521   ffff8e96c5f78000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795521   ffff8e96c5f78000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795521   ffff8e96c5f78000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795521   ffff8e96c5f78000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795522   ffff8e7755eee600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1007294649/1000000000
1760795522   ffff8e963f868000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1011299614/1000000000
1760795522   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951677
1760795522   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951677)
1760795522   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760795522   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951677
1760795522   ffff8e77621dcc80 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760795522   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING
1760795522   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): spa_load_retry: rewind, max txg: 16951676
1760795522   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): LOADING
1760795522   ffff8e96cb720000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66ad85b6-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795522   ffff8e96cb720000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/67e1a3fa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795522   ffff8e96cb720000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/685381f3-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795522   ffff8e96cb720000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795522   ffff8e96cb720000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683f34aa-97a6-11ed-b959-ac1f6b483888': probe done, cant_read=0 cant_write=1
1760795522   ffff8e96cb720000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f23f9038-da26-4973-952c-9e839b3e594b': probe done, cant_read=0 cant_write=1
1760795522   ffff8e96cb720000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/efb8e117-7881-48fb-9226-f3d6f8171109': probe done, cant_read=0 cant_write=1
1760795523   ffff8e96d4960000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68089bab-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1012216239/1000000000
1760795523   ffff8e96c8cfb300 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/683c9144-97a6-11ed-b959-ac1f6b483888': open error=2 timeout=1012205656/1000000000
1760795523   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': best uberblock found for spa FNMedia. txg 16951676
1760795523   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': label discarded as txg is too large (16951679 > 16951676)
1760795523   ffff8e77621dcc80 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/68239595-97a6-11ed-b959-ac1f6b483888': failed to read label config
1760795523   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): using uberblock with txg=16951676
1760795523   ffff8e77621dcc80 spa_misc.c:415:spa_load_failed(): spa_load(FNMedia, config untrusted): FAILED: label config unavailable
1760795523   ffff8e77621dcc80 spa_misc.c:429:spa_load_note(): spa_load(FNMedia, config untrusted): UNLOADING

Any thoughts on transaction rollback? The data that was being written at the time was non-critical, I can loose ~1 hour’s worth of transactions as I have a backup for those.

Arwen · October 18, 2025, 2:47pm

The only thing I can think of, is to manually select the TXG:

zpool import -fm -T 16951679 -o readonly=on FNMedia

Or perhaps based on the more recent error messages, use an older TXG:

zpool import -fm -T 16951676 -o readonly=on FNMedia

Maybe the L2ARC / Cache device is also borked, I just don't know enough about this level of trouble shooting, and I don't have an easy test environment. But, this seems odd;

Removing the L2ARC / Cache device should never cause data loss, as it is not the primary source of the data.

chereszabor · October 21, 2025, 12:51pm

Wanted to circle back, with an update;

Currently using the read-only command Arwen suggested below to attempt to import the pool
zpool import -fm -T 16951677 -o readonly=on FNMedia (note the slightly different TXG value)

Initial attempt failed but that was likely due to a web session timeout. After SSHing into truenas and running the command from ssh, current attempt has been running for the last ~12h or so, something is definitely happening as CPU utilization has been hovering ~15% continuously.

The system also shows “pool.dataset.sync_db_keys” and other “sync activity”

One downside is I haven’t been able to actively track current status of the import as any additional zpool command just hangs/waits until the import is complete.

If anyone has any suggestions on how to do real time monitoring, I would appreciate!

In any case, will update the thread when there are any changes/updates.

Thanks again!

chereszabor · October 25, 2025, 1:40am

Quick update and question:

Kicked off a import attempt with the following command

 zpool import -d /dev/disk/by-partuuid -f -FX -T 16951679 -o readonly=on FNMedia

It’s been now running for nearly 3 days, without any outputs outside of dbgmsg reporting the following:

1761340534   ffff9bb51618e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off ae720fb7000
1761340534   ffff9bb51618e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off ae720fa8000
1761340534   ffff9bb51618e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off ae720f9d000
1761340534   ffff9b94b5829980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbcee63000
1761340534   ffff9b92bf5e8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbcef32000
1761340534   ffff9bb51618e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=6 off 8f5987c1000
1761340534   ffff9bb295176600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=6 off 8f5987be000
1761340534   ffff9b932886cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbcee5e000
1761340534   ffff9b92a028b300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbcee16000
1761340534   ffff9bb220de0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbcede3000
1761340534   ffff9b92bf5ee600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbced3e000
1761340534   ffff9b94b582cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbced3e000
1761340534   ffff9b92c48b8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 19acd381000
1761340534   ffff9b94b582e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=6 off adbceda3000
1761340534   ffff9b943d771980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbced2b000
1761340534   ffff9bb1f877cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbced2b000
1761340534   ffff9b94b582e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=6 off adbceee4000
1761340534   ffff9b932886cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbceda9000
1761340534   ffff9bb220de4c80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbceeea000
1761340534   ffff9b943d771980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbcee90000
1761340534   ffff9b932886cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=6 off adbceeb7000
1761340534   ffff9bb201d3cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off adbcee5e000
1761340534   ffff9bb295176600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 8f5987c1000
1761340534   ffff9bb51618cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off adbced92000
1761340534   ffff9bb516189980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbcef06000
1761340534   ffff9b944cd83300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=6 off adbcee8a000
1761340534   ffff9b925f918000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off adbcef15000
1761340534   ffff9bb295171980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=6 off adbcef44000
1761340534   ffff9b94b582cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbceebd000
1761340534   ffff9b92c48b8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbcef4a000
1761340534   ffff9b92a028b300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off adbceed3000
1761340534   ffff9b92c48b9980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off adbceea6000
1761340534   ffff9b92c48b9980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 8f5987b8000
1761340534   ffff9bb1f8779980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbcee27000
1761340534   ffff9bb51618e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbcef1b000
1761340534   ffff9bb1f877e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbceed9000
1761340534   ffff9bb51618e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off adbcef38000
1761340534   ffff9bb51618e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=6 off adbcef2c000
1761340534   ffff9bb51618e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off 8f5987b9000
1761340534   ffff9b943d773300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbced98000
1761340534   ffff9b931cf8b300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off adbcef00000
1761340534   ffff9b944cd80000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off adbcee3d000
1761340534   ffff9b94b5829980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbcee7f000
1761340534   ffff9b94b5829980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 8f5987b5000
1761340534   ffff9b943d774c80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbceeac000
1761340534   ffff9bb1f8778000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbcee58000
1761340534   ffff9b944cd81980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off adbcee52000
1761340534   ffff9b92bf5e8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off adbcee79000
1761340534   ffff9b925f918000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=6 off 19ac5ad3000
1761340534   ffff9bb295176600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off 19ac5ad9000
1761340534   ffff9bb516189980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbcef33000
1761340534   ffff9bb1f8779980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off adbcef60000
1761340534   ffff9bb220de0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbcee43000
1761340534   ffff9bb1f877cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=6 off ae720f96000
1761340534   ffff9b92a028b300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbcef66000
1761340534   ffff9b92bf5e8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off 19ac5b19000
1761340534   ffff9b92c48bb300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off 19ac5aaa000
1761340534   ffff9bb1cbbb8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off 19ac5ac8000
1761340534   ffff9b94b5829980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off 19ac5aef000
1761340534   ffff9bb51618cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off 19ac5b10000
1761340534   ffff9bb51618b300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off 19ac5ac2000
1761340534   ffff9b932886cc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off 19ac5aaa000
1761340534   ffff9b92549bcc80 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off ae720f96000
1761340534   ffff9b94b582e600 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbced29000
1761340534   ffff9bb295171980 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off adbced29000
1761340534   ffff9b943d773300 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off 8f5987b3000
1761340534   ffff9bb1cbbb8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=6 off 8f5987b3000

Again given the fact that there are two know drives that are out of sync this is expected

Any thoughts on how long a process should take given the pool is ~60TB?

chereszabor · October 26, 2025, 9:43pm

So after 5 days, the pool was imported with in a per previous import configuration.

However, I got the following error:

cannot mount '/FNMedia': failed to create mountpoint: Read-only file system
Import was successful, but unable to mount some datasets

Is this an expected outcome, and if so, how should I go about resolving it?

Also, for reference, here is the current status of the pool:

  pool: FNMedia
 state: ONLINE
status: One or more devices could not be used because the label is missing or
	invalid.  Sufficient replicas exist for the pool to continue
	functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
  scan: scrub repaired 0B in 1 days 20:00:01 with 0 errors on Tue Oct  7 04:19:10 2025
expand: expanded raidz2-0 copied 50.4T in 1 days 07:22:20, on Mon Oct  6 15:42:14 2025
config:

	NAME                                      STATE     READ WRITE CKSUM
	FNMedia                                   ONLINE       0     0     0
	  raidz2-0                                ONLINE       0     0     0
	    68239595-97a6-11ed-b959-ac1f6b483888  ONLINE       0     0     0
	    67e1a3fa-97a6-11ed-b959-ac1f6b483888  ONLINE       0     0     0
	    685381f3-97a6-11ed-b959-ac1f6b483888  ONLINE       0     0     0
	    68089bab-97a6-11ed-b959-ac1f6b483888  ONLINE       0     0     1
	    683c9144-97a6-11ed-b959-ac1f6b483888  ONLINE       0     0     0
	    683f34aa-97a6-11ed-b959-ac1f6b483888  ONLINE       0     0     0
	    f23f9038-da26-4973-952c-9e839b3e594b  ONLINE       0     0    10
	    efb8e117-7881-48fb-9226-f3d6f8171109  ONLINE       0     0     0
	logs	
	  66ad85b6-97a6-11ed-b959-ac1f6b483888    ONLINE       0     0     0
	cache
	  66dd4eea-97a6-11ed-b959-ac1f6b483888    UNAVAIL      0     0     0

errors: No known data errors