Bringing back a SUSPENDED pool that doesn't show up with `zpool status`

HoneyBadger · June 6, 2025, 4:09pm

Just reporting back from some back and forth in DM’s - there might be issues even in the earlier transaction groups.

@coolnodje I’d recommend moving the drives to your new server (R730XD) as it will significantly help the cooling of the disks there. You can attempt an import at the earlier txg number but based on the dbgmsg output it is still calling out metadata and data errors. There are tunables that can be set to ignore them but that obviously results in the errors becoming permanent.

As an additional note, please ensure that the storage controller in your R730XD is either a true HBA (PERC HBA330) or has been reflashed/set to behave as one (a crossflashed H330 will work, an H730 in HBA mode is sub-optimal but will work as well) but ensure that it is not a “RAID card” attempting to present virtual disks.

coolnodje · June 6, 2025, 4:21pm

The new server has a SAS9305-16I LSI controller.

I’ll be moving to it, I suspect the old one has hardware issues, I’m not even sure the heat is the source problem. Maybe.

According to the errors, shall I still aim for
zpool import -nfFXT 36292723 subramanya
or shall I use the earlier txg=36292722 that was notified in the log ?

HoneyBadger · June 6, 2025, 4:31pm

You can still aim for 36292723 as the hopeful insert point but since it rewound from there regardless it may require even more drastic levels of recovery.

coolnodje · June 7, 2025, 8:14am

After cleaning up a dust filter which significantly improved the air flow which lowered HDD operating temperature to below 35C, I decided to go for a dry run on the current SilverStone D380 instance:
zpool import -nfFXT 36292723 subramanya

It started at 18:35, everything was looking good hardware and log (/proc/spl/kstat/zfs/dbgmsg) wise.

Here the result this morning, the process stopped at 5:34:

timestamp    message 
1749227306   ffff9155d1f98000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
[boot-pool pool]
1749227306   ffff9155ea29e300 spa_history.c:306:spa_history_log_sync(): txg 1571262 open pool version 5000; software version zfs-2.1.99-2799-gc71d2fb20; uts (none) 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Tue Jan 28 03:14:06 UTC 2025 x86_64
1749227306   ffff9155d1f98000 spa.c:9204:spa_async_request(): spa=boot-pool async request task=32
1749227306   ffff9155ea29e300 spa_history.c:306:spa_history_log_sync(): txg 1571264 import pool version 5000; software version zfs-2.1.99-2799-gc71d2fb20; uts (none) 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Tue Jan 28 03:14:06 UTC 2025 x86_64
1749227306   ffff9155ed724a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 87, smp_length 59232, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11350 ms, loading_time 2 ms, ms_max_size 2146365440, max size error 2146365440, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ecda4a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 106, smp_length 270312, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11350 ms, loading_time 11 ms, ms_max_size 2124718080, max size error 2124718080, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ed724a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 88, smp_length 285864, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11353 ms, loading_time 16 ms, ms_max_size 2139947008, max size error 2139947008, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ed724a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 93, smp_length 49096, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11370 ms, loading_time 2 ms, ms_max_size 2145525760, max size error 2145525760, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ecda4a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 91, smp_length 197624, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11362 ms, loading_time 12 ms, ms_max_size 2128207872, max size error 2128207872, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ea3a98c0 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 90, smp_length 386792, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11353 ms, loading_time 22 ms, ms_max_size 1225740288, max size error 1225740288, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ea3ae300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 92, smp_length 376200, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11362 ms, loading_time 18 ms, ms_max_size 1781149696, max size error 1781149696, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ed724a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571265, spa boot-pool, vdev_id 0, ms_id 94, smp_length 259128, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11372 ms, loading_time 11 ms, ms_max_size 1314934784, max size error 1314934784, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ea3a98c0 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571265, spa boot-pool, vdev_id 0, ms_id 96, smp_length 306088, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11376 ms, loading_time 15 ms, ms_max_size 1138450432, max size error 1138450432, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ecda4a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571265, spa boot-pool, vdev_id 0, ms_id 95, smp_length 411600, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11374 ms, loading_time 19 ms, ms_max_size 1133998080, max size error 1133998080, old_weight 780000000000001, new_weight 780000000000001
1749227308   ffff9155ea29e300 spa_history.c:293:spa_history_log_sync(): command: zpool import -N -f boot-pool
1749227347   ffff9155ea3b0000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749227347   ffff9155ea3b0000 spa.c:6916:spa_tryimport(): spa_tryimport: importing subramanya
1749227347   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ea3b0000-subramanya, config trusted): LOADING
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff9155ea3b0000-subramanya. txg 36292731
1749227347   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ea3b0000-subramanya, config untrusted): using uberblock with txg=36292731
1749227347   ffff9155e5baca40 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749227347   ffff9155d9343180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb75000
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749227348   ffff9155ea3b0000 spa.c:9204:spa_async_request(): spa=$import-ffff9155ea3b0000-subramanya async request task=2048
1749227348   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ea3b0000-subramanya, config trusted): LOADED
1749227348   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ea3b0000-subramanya, config trusted): UNLOADING
1749227348   ffff9155ea3b0000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749227348   ffff9155ea3b0000 spa.c:6761:spa_import(): spa_import: importing subramanya
1749227348   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227348   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292731
1749227348   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292731
1749227348   ffff9155e855b180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=0
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749227349   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): Read 7 log space maps (7 total blocks - blksz = 131072 bytes) in 16 ms
1749227349   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749227349   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749227349   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749227349   ffff9155e75fe300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 0, spa subramanya, vdev_id 0, ms_id 266, smp_length 279784, unflushed_allocs 417792, unflushed_frees 430080, freed 0, defer 0 + 0, unloaded time 54703 ms, loading_time 65 ms, ms_max_size 128653340672, max size error 128653168640, old_weight 900000000000001, new_weight 900000000000001
1749227350   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 1 metadata errors and 2 data errors
1749227350   ffff9155ea3b0000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config trusted): FAILED: spa_load_verify failed [error=5]
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): UNLOADING
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_retry: rewind, max txg: 36292730
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292730
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292730)
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292730
1749227350   ffff9155ea3b0000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): spa_load_retry: rewind, max txg: 36292729
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): LOADING
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292729
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292729)
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292729
1749227350   ffff9155ea3b0000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): spa_load_retry: rewind, max txg: 36292728
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): LOADING
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292728
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292728)
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292728
1749227350   ffff9155ea3b0000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749227350   ffff9155ea2f0000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
[backup pool]
[FAST pool]
1749227550   ffff915701310000 spa.c:6916:spa_tryimport(): spa_tryimport: importing subramanya
1749227550   ffff915701310000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff915701310000-subramanya, config trusted): LOADING
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227550   ffff915701310000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff915701310000-subramanya. txg 36292731
1749227550   ffff915701310000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff915701310000-subramanya, config untrusted): using uberblock with txg=36292731
1749227550   ffff9156706a98c0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749227550   ffff915701a7ca40 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb75000
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749227550   ffff915701310000 spa.c:9204:spa_async_request(): spa=$import-ffff915701310000-subramanya async request task=2048
1749227550   ffff915701310000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff915701310000-subramanya, config trusted): LOADED
1749227550   ffff915701310000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff915701310000-subramanya, config trusted): UNLOADING
1749227691   ffff9155ee2f8000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749227691   ffff9155ee2f8000 spa.c:6913:spa_tryimport(): spa_tryimport: importing subramanya, max_txg=36292723
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ee2f8000-subramanya, config trusted): LOADING
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff9155ee2f8000-subramanya. txg 36292723
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292723)
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config. Trying again without txg restrictions.
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ee2f8000-subramanya, config untrusted): using uberblock with txg=36292723
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749227691   ffff9155ee2f8000 spa.c:9204:spa_async_request(): spa=$import-ffff9155ee2f8000-subramanya async request task=2048
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ee2f8000-subramanya, config trusted): LOADED
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ee2f8000-subramanya, config trusted): UNLOADING
1749227691   ffff9155ee2f8000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749227691   ffff9155ee2f8000 spa.c:6761:spa_import(): spa_import: importing subramanya
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292723
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292723)
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config. Trying again without txg restrictions.
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292723
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=0
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749227692   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): Read 6 log space maps (6 total blocks - blksz = 131072 bytes) in 7 ms
1749227692   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749227692   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749227692   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749227692   ffff9155ea28e300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 0, spa subramanya, vdev_id 0, ms_id 266, smp_length 279704, unflushed_allocs 516096, unflushed_frees 663552, freed 0, defer 0 + 0, unloaded time 397401 ms, loading_time 187 ms, ms_max_size 128664842240, max size error 128664645632, old_weight 900000000000001, new_weight 900000000000001
1749227693   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749227693   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
  (spa_load_verify_data=1, spa_load_verify_metadata=1)
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 44, weight 840000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 1237761, loaded 619794 ms ago, max_size 17179643904
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 52, weight 880000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 0, loaded 619480 ms ago, max_size 17179869184
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 45, weight 880000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 0, loaded 619480 ms ago, max_size 17179869184
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 53, weight 880000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 0, loaded 619480 ms ago, max_size 17179869184
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 46, weight 880000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 0, loaded 619480 ms ago, max_size 17179869184
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 47, weight 880000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 0, loaded 619480 ms ago, max_size 17179869184
1749249416   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915626712ac0 repairing c=2 devidx=5 offset=54055f96000
1749249416   ffff9155d093b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91565392a600 repairing c=2 devidx=5 offset=54055f97000
1749249416   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915644891300 repairing c=2 devidx=5 offset=54055f98000
1749249416   ffff9156733de300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91598952f200 repairing c=2 devidx=5 offset=54055f8f000
1749249416   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157165fd580 repairing c=2 devidx=5 offset=54055f8e000
1749249416   ffff9157002c0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91564f474740 repairing c=2 devidx=5 offset=54055f94000
1749249416   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91594997df00 repairing c=2 devidx=5 offset=54055f8d000
1749249416   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91564fb16d40 repairing c=2 devidx=5 offset=54055f8b000
1749249418   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157d2ace880 repairing c=3 devidx=5 offset=535d51a6000
1749249419   ffff9156733de300 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0dc000
1749249419   ffff9156733de300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91594558f6c0 repairing c=1 devidx=5 offset=53c4f0dc000
1749249419   ffff9155e300b180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0df000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915648111300 repairing c=1 devidx=5 offset=53c4f0df000
1749249419   ffff9156733de300 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0da000
1749249419   ffff9156733de300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156277097c0 repairing c=1 devidx=5 offset=53c4f0da000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0de000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156cdf804c0 repairing c=1 devidx=5 offset=53c4f0de000
1749249419   ffff9155e300b180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0db000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915627708980 repairing c=1 devidx=5 offset=53c4f0db000
1749249419   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915903e9d0c0 repairing c=4 devidx=5 offset=53c4f0e0000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0d9000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915841f71c80 repairing c=1 devidx=5 offset=53c4f0d9000
1749249419   ffff9157002c0000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0cb000
1749249419   ffff9157002c0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91590a4df200 repairing c=1 devidx=5 offset=53c4f0cb000
1749249419   ffff9156733de300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91583e94df00 repairing c=4 devidx=5 offset=53c4f0d3000
1749249419   ffff9157002c0000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0c8000
1749249419   ffff9157002c0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156481d3440 repairing c=1 devidx=5 offset=53c4f0c8000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915627bd63c0 repairing c=4 devidx=5 offset=53c4f0d8000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9158be5eb900 repairing c=4 devidx=5 offset=53c4f0d6000
1749249419   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9155e4ba2ac0 repairing c=4 devidx=5 offset=53c4f0d5000
1749249419   ffff9157002c0000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0b8000
1749249419   ffff9157002c0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157d1a76d40 repairing c=1 devidx=5 offset=53c4f0b8000
1749249419   ffff9156733de300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915901b83900 repairing c=4 devidx=5 offset=53c4f0c2000
1749249419   ffff9157002c0000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0ae000
1749249419   ffff9157002c0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156cdf85f00 repairing c=1 devidx=5 offset=53c4f0ae000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91584073e880 repairing c=4 devidx=5 offset=53c4f0d2000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157d3762f80 repairing c=4 devidx=5 offset=53c4f0d4000
1749249419   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915652e2a140 repairing c=4 devidx=5 offset=53c4f0d1000
1749249419   ffff9155e300b180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0cf000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915626d4bdc0 repairing c=1 devidx=5 offset=53c4f0cf000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915626d49c80 repairing c=4 devidx=5 offset=53c4f0d0000
1749249419   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156d0f94c00 repairing c=4 devidx=5 offset=53c4f0ce000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157128a50c0 repairing c=4 devidx=5 offset=53c4f0cd000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0ca000
1749249419   ffff9156706b3180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0c9000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156538d4c00 repairing c=1 devidx=5 offset=53c4f0ca000
1749249419   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156481d0000 repairing c=1 devidx=5 offset=53c4f0c9000
1749249419   ffff9155e300b180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0c7000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9158424b3dc0 repairing c=1 devidx=5 offset=53c4f0c7000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0c6000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157f892f6c0 repairing c=1 devidx=5 offset=53c4f0c6000
[…]
1749267336   ffff9155db026300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156482bf6c0 repairing c=1 devidx=0 offset=1357eff2000
1749267336   ffff9155db026300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156482bf6c0 repairing c=2 devidx=1 offset=1357eff2000
1749267336   ffff9155d2d48000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915714ed2600 repairing c=1 devidx=0 offset=1357c56d000
1749267336   ffff9155d2d48000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915714ed2600 repairing c=2 devidx=1 offset=1357c56d000
1749267359   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 8 metadata errors and 17937 data errors
1749267359   ffff9155ee2f8000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config trusted): FAILED: spa_load_verify failed [error=6]
1749267359   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): UNLOADING
1749267363   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_retry: rewind, max txg: 36292722
1749267363   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749267363   ffff9155e8338000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749267363   ffff9155e8338000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749267363   ffff9155e8338000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749267364   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': open error=2 timeout=1003438410/1000000000
1749267364   ffff9155e6704a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': open error=2 timeout=1007585577/1000000000
1749267364   ffff9155e6700000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': open error=2 timeout=1015360540/1000000000
1749267364   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): vdev tree has 1 missing top-level vdevs.
1749267364   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
1749267364   ffff9155ee2f8000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: unable to open vdev tree [error=2]
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 15005074635607672362, path: N/A, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 1843460257378462166, path: N/A, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 12560554049367260037, path: /dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 3504886088678984596, path: /dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 1026282999581038360, path: /dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4, healthy
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 4494896413352636629, path: /dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7, healthy
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 18426049172646806820, path: /dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 10710542068254203083, path: /dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405, healthy
1749267364   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING

I understand from the last lines that some harware issues made the process stop, because it couldn’t access some of the pool’s vdevs :

1749267364   ffff9155ee2f8000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: unable to open vdev tree [error=2]
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 15005074635607672362, path: N/A, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 1843460257378462166, path: N/A, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 12560554049367260037, path: /dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 3504886088678984596, path: /dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 1026282999581038360, path: /dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4, healthy
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 4494896413352636629, path: /dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7, healthy
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 18426049172646806820, path: /dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 10710542068254203083, path: /dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405, healthy
1749267364   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING

Looking at kernel logs ( I discovered the versatile journalctl -o short-precise -k -b all and recommand it over dmesg -T):

Jun 06 18:56:27.668615 freenas kernel:  </TASK>
Jun 06 18:56:27.668714 freenas kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
Jun 07 02:26:38.828970 freenas kernel: perf: interrupt took too long (2536 > 2500), lowering kernel.perf_event_max_sample_rate to 78750
Jun 07 05:31:57.512769 freenas kernel: ata5.00: exception Emask 0x50 SAct 0x2600000 SErr 0x4090800 action 0xe frozen
Jun 07 05:31:57.513284 freenas kernel: ata5.00: irq_stat 0x00400040, connection status changed
Jun 07 05:31:57.513421 freenas kernel: ata5: SError: { HostInt PHYRdyChg 10B8B DevExch }
Jun 07 05:31:57.513548 freenas kernel: ata5.00: failed command: READ FPDMA QUEUED
Jun 07 05:31:57.518675 freenas kernel: ata5.00: cmd 60/00:a8:58:25:de/08:00:ab:01:00/40 tag 21 ncq dma 1048576 in
                                                res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x50 (ATA bus error)
Jun 07 05:31:57.518870 freenas kernel: ata5.00: status: { DRDY }
Jun 07 05:31:57.526984 freenas kernel: ata5.00: failed command: READ FPDMA QUEUED
Jun 07 05:31:57.527141 freenas kernel: ata5.00: cmd 60/10:b0:e0:37:de/00:00:ab:01:00/40 tag 22 ncq dma 8192 in
                                                res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x50 (ATA bus error)
Jun 07 05:31:57.527307 freenas kernel: ata5.00: status: { DRDY }
Jun 07 05:31:57.533832 freenas kernel: ata5.00: failed command: READ FPDMA QUEUED
Jun 07 05:31:57.533962 freenas kernel: ata5.00: cmd 60/80:c8:58:2d:de/00:00:ab:01:00/40 tag 25 ncq dma 65536 in
                                                res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x50 (ATA bus error)
Jun 07 05:31:57.534060 freenas kernel: ata5.00: status: { DRDY }
Jun 07 05:31:57.538968 freenas kernel: ata5: hard resetting link

One can clearly see some issues suddenly coming up at ~5:31, which roughly matches the last log from dbgmsg (Timestamp 1749267364) where it says it can’t access vdevs and stops.

Then the kernel log is full of these or the likes of:

zio pool=subramanya vdev=/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8 error=5 type=1 offset=1673710948352 size=4096 flags=1573248
Jun 07 05:35:22.957531 freenas kernel: I/O error, dev sdb, sector 2576 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

up until ~7:33 even though nothing runs on the HDD anymore, and then the log becomes quiet up until now.

The drives temperature and I/O graphs from TN Reporting are clean and consistent from the beginning of the operations 'till 5:34: all disk I/O identical, temperature stable.
Though there are some holes in temperature reporting on some drives, but matching the hardware issues, so it makes sense. But reported temp is never above 35C.

I’ll run the same zpool import -nfFXT 36292723 subramanya on the replacement box now.

neofusion · June 7, 2025, 10:17am

It’s concerning that it lists 3 devices as “can’t open”…

coolnodje · June 7, 2025, 10:18am

OK, on the new system Dell R730Xd+LSI SAS9305-16I, even though everything seem to be running smoothly with the import, I seem to be plagued with this bug: LSI 9305 errors filling log | TrueNAS Community

This gives me continuous kernel log and prevents the UI Storage Dashboard or Disks pages from being responsive.

I’m getting a flood of these:

Jun 07 12:14:13.152058 truenas kernel: mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)
Jun 07 12:14:13.152352 truenas kernel: mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)
Jun 07 12:14:13.152421 truenas kernel: mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)
Jun 07 12:14:13.156110 truenas kernel: sd 1:0:1:0: device_block, handle(0x001b)
Jun 07 12:14:13.162850 truenas kernel: sd 1:0:5:0: device_block, handle(0x001f)
Jun 07 12:14:13.168231 truenas kernel: sd 1:0:10:0: device_block, handle(0x0024)
Jun 07 12:14:14.151301 truenas kernel: mpt3sas_cm0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
Jun 07 12:14:14.151416 truenas kernel: mpt3sas_cm0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
Jun 07 12:14:14.152990 truenas kernel: mpt3sas_cm0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
Jun 07 12:14:14.401899 truenas kernel: sd 1:0:8:0: device_block, handle(0x0022)
Jun 07 12:14:14.407977 truenas kernel: sd 1:0:1:0: device_unblock and setting to running, handle(0x001b)
Jun 07 12:14:14.408700 truenas kernel: sd 1:0:5:0: device_unblock and setting to running, handle(0x001f)
Jun 07 12:14:14.409372 truenas kernel: sd 1:0:10:0: device_unblock and setting to running, handle(0x0024)
Jun 07 12:14:15.903098 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.903217 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.903249 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.903275 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.904845 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.905078 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.905200 truenas kernel: sd 1:0:6:0: device_block, handle(0x0020)
Jun 07 12:14:15.912005 truenas kernel: sd 1:0:8:0: device_unblock and setting to running, handle(0x0022)
Jun 07 12:14:16.903150 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:16.903300 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:16.903338 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:16.908047 truenas kernel: sd 1:0:6:0: device_unblock and setting to running, handle(0x0020)
Jun 07 12:14:17.162632 truenas kernel: sd 1:0:8:0: Power-on or device reset occurred
Jun 07 12:14:17.163172 truenas kernel: sd 1:0:3:0: Power-on or device reset occurred

(all yellow = warning, I think)
and earlier on some of:

Jun 07 12:07:28.150756 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:07:28.150782 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:07:28.150816 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:07:28.152138 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:07:28.152208 truenas kernel: sd 1:0:3:0: [sdd] tag#3923 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
Jun 07 12:07:28.152726 truenas kernel: sd 1:0:3:0: [sdd] tag#3923 CDB: Read(16) 88 00 00 00 00 05 74 ff ff 08 00 00 00 78 00 00
Jun 07 12:07:28.153131 truenas kernel: I/O error, dev sdd, sector 23437770504 op 0x0:(READ) flags 0x80700 phys_seg 15 prio class 2
Jun 07 12:07:28.153167 truenas kernel: sd 1:0:3:0: [sdd] tag#3920 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
Jun 07 12:07:28.153586 truenas kernel: sd 1:0:3:0: [sdd] tag#3920 CDB: Read(16) 88 00 00 00 00 05 74 ff ff 88 00 00 00 38 00 00
Jun 07 12:07:28.153984 truenas kernel: I/O error, dev sdd, sector 23437770632 op 0x0:(READ) flags 0x80700 phys_seg 7 prio class 2
Jun 07 12:07:28.154028 truenas kernel: sd 1:0:3:0: [sdd] tag#3919 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
Jun 07 12:07:28.154408 truenas kernel: sd 1:0:3:0: [sdd] tag#3919 CDB: Read(16) 88 00 00 00 00 05 74 ff ff c8 00 00 00 28 00 00
Jun 07 12:07:28.154784 truenas kernel: I/O error, dev sdd, sector 23437770696 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
Jun 07 12:07:28.154816 truenas kernel: sd 1:0:3:0: [sdd] tag#3918 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
Jun 07 12:07:28.155191 truenas kernel: sd 1:0:3:0: [sdd] tag#3918 CDB: Read(16) 88 00 00 00 00 05 74 ff fe 78 00 00 00 80 00 00
Jun 07 12:07:28.155556 truenas kernel: I/O error, dev sdd, sector 23437770360 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2
Jun 07 12:07:28.155602 truenas kernel: sd 1:0:3:0: [sdd] tag#3917 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
Jun 07 12:07:28.156028 truenas kernel: sd 1:0:3:0: [sdd] tag#3917 CDB: Read(16) 88 00 00 00 00 05 74 ff fe 00 00 00 00 70 00 00
Jun 07 12:07:28.156416 truenas kernel: I/O error, dev sdd, sector 23437770240 op 0x0:(READ) flags 0x80700 phys_seg 14 prio class 2
Jun 07 12:07:28.405131 truenas kernel: sd 1:0:3:0: Power-on or device reset occurred
Jun 07 12:07:28.405629 truenas kernel: sd 1:0:10:0: Power-on or device reset occurred

with I/O error in Red

It’s very annoying but it seems ok.

coolnodje · June 9, 2025, 9:07am

It’s been 48h almost for this zpool import -nfFXT 36292723 subramanya command.

I have no idea whether this is expected with a 6wZ2 of 12TB disks.

I still have this flood of message like these:

Jun 09 11:00:33.801527 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.804442 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.804544 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.807526 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.812808 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.813039 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.813248 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.813302 truenas kernel: sd 1:0:3:0: [sdd] tag#5 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
Jun 09 11:00:33.814093 truenas kernel: sd 1:0:3:0: [sdd] tag#5 CDB: Read(16) 88 00 00 00 00 00 8a 29 5c 28 00 00 08 00 00 00
Jun 09 11:00:33.814778 truenas kernel: I/O error, dev sdd, sector 2317966376 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2
Jun 09 11:00:33.814861 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.814899 truenas kernel: zio pool=subramanya vdev=/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405 error=5 type=1 offset=1186797735936 size=1048576 flags=1074267280
Jun 09 11:00:33.814953 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.814990 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.815032 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.815066 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.815107 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.815698 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.816423 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.817220 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.817933 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 09 11:00:33.819488 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)

but things look ok otherwise, especially the /proc/spl/kstat/zfs/dbgmsg which keep outputing things like:

1749459793   ffff939db8fe0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff939af71ad0c0 repairing c=4 devidx=2 offset=3f3f560000
1749459793   ffff939850fd3280 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff939b0568aac0 repairing c=4 devidx=2 offset=3f3f550000
1749459793   ffff9399785b4bc0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff939d1924c280 repairing c=4 devidx=2 offset=3f3f590000
1749459793   ffff939c64e46500 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff939c68e95a40 repairing c=4 devidx=2 offset=3f3f5d0000
1749459793   ffff939991f11940 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff93988b00e3c0 repairing c=4 devidx=2 offset=3f3f558000
1749459793   ffff939892b04bc0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff939c67d94c00 repairing c=4 devidx=2 offset=3f3f570000
1749459793   ffff939992b59940 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff93998a81aac0 repairing c=4 devidx=2 offset=3f3f540000
1749459793   ffff939978521940 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff93989257a140 repairing c=4 devidx=2 offset=3f3f588000
1749459793   ffff9399785b4bc0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff939a4f2984c0 repairing c=4 devidx=2 offset=3f3f548000
1749459803   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749459808   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749459818   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749459820   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749459828   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749459835   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749459835   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749459847   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749459855   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749459857   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749459863   ffff939978619940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0

which I believe are expected.

Is this time frame expected or am I having an issue related to these kernel messages?

coolnodje · June 9, 2025, 7:21pm

I stopped the operation, thought it was taking too long, and the read throughput mean being below 5MiB/s on all 6 drives I thought something was up.
And the controller logs were just too much to feel good about it.

I found a way to fix the flood of mpt3sas_cm0 errors.
I had to move the LSI SAS9305-16I controller to another PCI port, boot again, found even more errors, and move it back to where it was and then errors disappeared…
I actually did solve the same issue once with just the same moving/removing from the PCI port, booting, then move back/add again. That was back in December when I was putting the box together. I just forgot. But such an odd issue.

I restarted the process zpool import -nfFXT 36292723 subramanya, everything seems normal. No kernel errors, expect formiddlewared still hanging on listing Storage and Disk content.
But the mean read throughput doesn’t seem much higher though.

Hopefully it will return in less than 48h, otherwise I might just as well run it without -n.

coolnodje · June 10, 2025, 9:36am

The command hasn’t yet returned but the process seem to be over, the 6 drives I/O are entirely flat since Starting import log (maybe because of -n dry run)

Stared at 20:56:14 (1749495374)
Logs ended at 11:09:00 (1749546540) with:

1749546540   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 0 metadata errors and 3 data errors
1749546540   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749546540   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749546540   ffff8da081966500 spa.c:9209:spa_async_request(): spa=subramanya async request task=2048
1749546540   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADED

There were much error reporting for the 2 first hours, then the last error reporting log was at 23:03:23 (1749503003) and nothing for the following 12hours and finally the Starting import log.

timestamp    message 
1749495374   ffff8da081966500 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1749495374   ffff8da081966500 spa.c:6913:spa_tryimport(): spa_tryimport: importing subramanya, max_txg=36292723
1749495374   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8da081966500-subramanya, config trusted): LOADING
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749495374   ffff8da081966500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff8da081966500-subramanya. txg 36292723
1749495374   ffff8da081966500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292723)
1749495374   ffff8da081966500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config. Trying again without txg restrictions.
1749495374   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8da081966500-subramanya, config untrusted): using uberblock with txg=36292723
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749495374   ffff8d9edae89940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749495374   ffff8da081966500 spa.c:9209:spa_async_request(): spa=$import-ffff8da081966500-subramanya async request task=2048
1749495374   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8da081966500-subramanya, config trusted): LOADED
1749495374   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8da081966500-subramanya, config trusted): UNLOADING
1749495374   ffff8da081966500 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1749495374   ffff8da081966500 spa.c:6761:spa_import(): spa_import: importing subramanya
1749495374   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749495374   ffff8da081966500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292723
1749495374   ffff8da081966500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292723)
1749495374   ffff8da081966500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config. Trying again without txg restrictions.
1749495374   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292723
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=0
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749495374   ffff8da50b57e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749495374   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): Read 6 log space maps (6 total blocks - blksz = 131072 bytes) in 14 ms
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749495374   ffff8da4332e4bc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 0, spa subramanya, vdev_id 0, ms_id 266, smp_length 279704, unflushed_allocs 516096, unflushed_frees 663552, freed 0, defer 0 + 0, unloaded time 3031892 ms, loading_time 24 ms, ms_max_size 128664842240, max size error 128664645632, old_weight 900000000000001, new_weight 900000000000001
1749495375   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749495375   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
  (spa_load_verify_data=1, spa_load_verify_metadata=1)
1749495965   ffff8da4329f6500 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=1 off 2fc8270b000
1749495965   ffff8da1811a4bc0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=1 off 2fc82703000
1749495965   ffff8da1811a4bc0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da521389c80 repairing c=0 devidx=1 offset=2fc82703000
1749495965   ffff8da4329f6500 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8d9fa51f04c0 repairing c=0 devidx=1 offset=2fc8270b000
1749495965   ffff8da50b573280 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=1 off 2fc826f3000
1749495965   ffff8da50b571940 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=1 off 2fc826db000
1749495965   ffff8da50b573280 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da27c205a40 repairing c=0 devidx=1 offset=2fc826f3000
1749495965   ffff8da50b571940 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da180f3af80 repairing c=0 devidx=1 offset=2fc826db000
1749495965   ffff8da50ad99940 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=1 off 2fc826eb000
1749495965   ffff8d9ed0ca9940 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=1 off 2fc826fb000
1749495965   ffff8da50ad99940 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8d9fa62bdf00 repairing c=0 devidx=1 offset=2fc826eb000
1749495965   ffff8da1811a3280 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=1 off 2fc826e3000
1749495965   ffff8d9ed0ca9940 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da27c2076c0 repairing c=0 devidx=1 offset=2fc826fb000
1749495965   ffff8da1811a3280 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da515750000 repairing c=0 devidx=1 offset=2fc826e3000

[...]

1749503003   ffff8d9edbc43280 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=5 off 57fb6a4000
1749503003   ffff8d9edbc43280 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da1a8a40000 repairing c=0 devidx=5 offset=57fb6a4000
1749503003   ffff8da50ad99940 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804d0c000
1749503003   ffff8da50ad99940 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8d9f26b8f200 repairing c=1 devidx=3 offset=5804d0c000
1749503003   ffff8d9f99400000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804d1c000
1749503003   ffff8d9f99400000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8d9fa5f50980 repairing c=1 devidx=3 offset=5804d1c000
1749503003   ffff8da35e1c8000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804cc4000
1749503003   ffff8da35e1c8000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8d9f17a59c80 repairing c=1 devidx=3 offset=5804cc4000
1749503003   ffff8da50b573280 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804d34000
1749503003   ffff8d9f99f58000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804d04000
1749503003   ffff8da50b573280 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da28e582f80 repairing c=1 devidx=3 offset=5804d34000
1749503003   ffff8d9f99f58000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da443fef200 repairing c=1 devidx=3 offset=5804d04000
1749503003   ffff8da50ad9e500 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804ce4000
1749503003   ffff8da50b983280 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804d14000
1749503003   ffff8d9f9e7d8000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804d2c000
1749503003   ffff8da50b571940 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804d24000
1749503003   ffff8da50ad9e500 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da430312f80 repairing c=1 devidx=3 offset=5804ce4000
1749503003   ffff8d9f9e7d8000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da28e585a40 repairing c=1 devidx=3 offset=5804d2c000
1749503003   ffff8da50b983280 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8d9f26b8ed40 repairing c=1 devidx=3 offset=5804d14000
1749503003   ffff8d9ed0bf8000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804cdc000
1749503003   ffff8da50b571940 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8d9fa5f57200 repairing c=1 devidx=3 offset=5804d24000
1749503003   ffff8d9f99f59940 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804cfc000
1749503003   ffff8d9ed0bf8000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da433161c80 repairing c=1 devidx=3 offset=5804cdc000
1749503003   ffff8d9f99f59940 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da443fe84c0 repairing c=1 devidx=3 offset=5804cfc000
1749503003   ffff8da50ad99940 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804cd4000
1749503003   ffff8da50ad99940 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da430313dc0 repairing c=1 devidx=3 offset=5804cd4000
1749503003   ffff8da35e1c8000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804cbc000
1749503003   ffff8da35e1c8000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da28c486880 repairing c=1 devidx=3 offset=5804cbc000
1749503003   ffff8da50b573280 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804cf4000
1749503003   ffff8d9edbc43280 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804cec000
1749503003   ffff8d9f99400000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804cb4000
1749503003   ffff8da50b573280 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da43317d580 repairing c=1 devidx=3 offset=5804cf4000
1749503003   ffff8d9f99400000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8d9f17a58980 repairing c=1 devidx=3 offset=5804cb4000
1749503003   ffff8d9edbc43280 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da4331797c0 repairing c=1 devidx=3 offset=5804cec000
1749503003   ffff8da50ad9e500 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804ccc000
1749503003   ffff8d9f99f58000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=3 off 5804cac000
1749503003   ffff8da50ad9e500 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da433162f80 repairing c=1 devidx=3 offset=5804ccc000
1749503003   ffff8d9f99f58000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da52601b900 repairing c=1 devidx=3 offset=5804cac000
1749503003   ffff8d9edbc43280 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=5 off 5804d4c000
1749503003   ffff8d9edbc43280 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff8da19bdc1300 repairing c=0 devidx=5 offset=5804d4c000
1749506405   ffff8da081841940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 57340, spa boot-pool, vdev_id 0, ms_id 50, smp_length 94304, unflushed_allocs 0, unflushed_frees 16384, freed 0, defer 16384 + 0, unloaded time 14062766 ms, loading_time 2 ms, ms_max_size 1073283072, max size error 1073164288, old_weight 780000000000001, new_weight 780000000000001
1749506835   ffff8da331c21940 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 57424, spa boot-pool, vdev_id 0, ms_id 21, weight 740000000000001, selected txg 57306 (604166 ms ago), alloc_txg 0, loaded 14486443 ms ago, max_size 1034633216
1749506902   ffff8da3654fb280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 57437, spa boot-pool, vdev_id 0, ms_id 95, smp_length 79880, unflushed_allocs 0, unflushed_frees 102400, freed 0, defer 0 + 49152, unloaded time 14559395 ms, loading_time 2 ms, ms_max_size 1067896832, max size error 1067847680, old_weight 780000000000001, new_weight 780000000000001
1749507501   ffff8da331c21940 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 57554, spa boot-pool, vdev_id 0, ms_id 20, weight 740000000000001, selected txg 57436 (604157 ms ago), alloc_txg 0, loaded 15152039 ms ago, max_size 896659456
1749546540   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 0 metadata errors and 3 data errors
1749546540   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749546540   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749546540   ffff8da081966500 spa.c:9209:spa_async_request(): spa=subramanya async request task=2048
1749546540   ffff8da081966500 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADED

coolnodje · June 10, 2025, 4:36pm

Still no additional log or I/O activity after 8 more hours.
Is there anything else to expect before zpool import -nfFXT 36292723 subramanya returns ?

HoneyBadger · June 10, 2025, 7:37pm

You should see a final line of “Finished importing” when the pool is in - and since you’re not seeing it, and combined with the earlier errors in dmesg indicating I/O errors I worry that it’s not able to complete.

1749051628   ffff94fe6a059900 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'ssdpool' Finished importing

coolnodje · June 10, 2025, 8:05pm

There wasn’t any kernel error message this time. Everything is fine on this side.

cat /proc/spl/kstat/zfs/dbgmsg |grep spa_import_progress_set_notes_impl
shows:

1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749495374   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749495375   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749546540   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749546540   ffff8da081966500 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import

These are all from the same launch starting at

20:56:14 (1749495374)
A first Starting import is logged then another one again at 11:09:00 the next day (1749546540) and since then I/O is flat, nothing is happening.

There must another issue.

Shall I stop this process and try to run the import without -n ? Or try again with it until it finishes properly?

coolnodje · June 10, 2025, 8:33pm

The zpool import process is in D state, along with 4 other middlewared processes.

It seems connected to listing Storage and Disk when accessing UI Dashboard and UI Storage.

I guess reboot is the only option and start again.

Is it expected that listing Disks and Storage on the UI hangs when zpool import is running?

It does that consistently on my instance.

coolnodje · June 11, 2025, 12:37pm

I rebooted and started again the process.
Everything was smooth, and logs report less and less .
Can it be that the dry run process fixes errors on the go?

1749587979   ffff8c4ae3440000 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1749587979   ffff8c4ae3440000 spa.c:6916:spa_tryimport(): spa_tryimport: importing subramanya
1749587979   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ae3440000-subramanya, config trusted): LOADING
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff8c4ae3440000-subramanya. txg 36292731
1749587979   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ae3440000-subramanya, config untrusted): using uberblock with txg=36292731
1749587979   ffff8c4b8bae0000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4b8bae1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749587979   ffff8c4b8bae6500 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb75000
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749587979   ffff8c4ae3440000 spa.c:9209:spa_async_request(): spa=$import-ffff8c4ae3440000-subramanya async request task=2048
1749587979   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ae3440000-subramanya, config trusted): LOADED
1749587979   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ae3440000-subramanya, config trusted): UNLOADING
1749587979   ffff8c4ae3440000 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1749587979   ffff8c4ae3440000 spa.c:6761:spa_import(): spa_import: importing subramanya
1749587979   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749587979   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292731
1749587979   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292731
1749587979   ffff8c49937e9940 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0
1749587979   ffff8c49937ee500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=0
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749587979   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749587980   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749587980   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749587980   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749587980   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749587981   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): Read 7 log space maps (7 total blocks - blksz = 131072 bytes) in 21 ms
1749587981   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749587981   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749587981   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749587981   ffff8c4ae3768000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 0, spa subramanya, vdev_id 0, ms_id 266, smp_length 279784, unflushed_allocs 417792, unflushed_frees 430080, freed 0, defer 0 + 0, unloaded time 37346 ms, loading_time 74 ms, ms_max_size 128653340672, max size error 128653168640, old_weight 900000000000001, new_weight 900000000000001
1749587982   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 1 metadata errors and 2 data errors
1749587982   ffff8c4ae3440000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config trusted): FAILED: spa_load_verify failed [error=5]
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): UNLOADING
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_retry: rewind, max txg: 36292730
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749587982   ffff8c4c31c66500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4c31c66500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4c31c66500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4c31c66500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4c31c66500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4c31c66500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292730
1749587982   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292730)
1749587982   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292730
1749587982   ffff8c4ae3440000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): spa_load_retry: rewind, max txg: 36292729
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): LOADING
1749587982   ffff8c4c33679940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4c33679940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4c33679940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4c33679940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4c33679940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4c33679940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292729
1749587982   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292729)
1749587982   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292729
1749587982   ffff8c4ae3440000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): spa_load_retry: rewind, max txg: 36292728
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): LOADING
1749587982   ffff8c4ce7229940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce7229940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce7229940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce7229940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce7229940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce7229940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292728
1749587982   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292728)
1749587982   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292728
1749587982   ffff8c4ae3440000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749587982   ffff8c4ae3440000 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1749587982   ffff8c4ae3440000 spa.c:6916:spa_tryimport(): spa_tryimport: importing twowaymirrored
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ae3440000-twowaymirrored, config trusted): LOADING
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/09ecd083-a36b-4bae-b6cd-ce58aa847f1c': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4c6c7b1a-8a6d-4258-bd74-bf136de81a76': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f811fbee-c88d-4ab6-a720-3f020e1d24ea': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d285472a-77b3-4451-9406-cd790f675b7f': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/c68a7ecc-e637-4b21-bdd8-b1980224a4c1': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6b896543-5dcb-44d9-841f-f4cbe4c96a7b': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6b896543-5dcb-44d9-841f-f4cbe4c96a7b': best uberblock found for spa $import-ffff8c4ae3440000-twowaymirrored. txg 64472
1749587982   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ae3440000-twowaymirrored, config untrusted): using uberblock with txg=64472
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d285472a-77b3-4451-9406-cd790f675b7f': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/09ecd083-a36b-4bae-b6cd-ce58aa847f1c': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4c6c7b1a-8a6d-4258-bd74-bf136de81a76': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/c68a7ecc-e637-4b21-bdd8-b1980224a4c1': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6b896543-5dcb-44d9-841f-f4cbe4c96a7b': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ce722e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f811fbee-c88d-4ab6-a720-3f020e1d24ea': probe done, cant_read=0 cant_write=1
1749587982   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading checkpoint txg
1749587982   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading indirect vdev metadata
1749587982   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Checking feature flags
1749587982   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading special MOS directories
1749587982   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading properties
1749587982   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading AUX vdevs
1749587982   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading vdev metadata
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading dedup tables
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading BRT
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Verifying Log Devices
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Verifying pool data
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Calculating deflated space
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Starting import
1749587983   ffff8c4ae3440000 spa.c:9209:spa_async_request(): spa=$import-ffff8c4ae3440000-twowaymirrored async request task=2048
1749587983   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ae3440000-twowaymirrored, config trusted): LOADED
1749587983   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ae3440000-twowaymirrored, config trusted): UNLOADING
1749587983   ffff8c4ae3440000 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1749587983   ffff8c4ae3440000 spa.c:6761:spa_import(): spa_import: importing twowaymirrored
1749587983   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(twowaymirrored, config trusted): LOADING
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f811fbee-c88d-4ab6-a720-3f020e1d24ea': probe done, cant_read=0 cant_write=1
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/c68a7ecc-e637-4b21-bdd8-b1980224a4c1': probe done, cant_read=0 cant_write=1
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d285472a-77b3-4451-9406-cd790f675b7f': probe done, cant_read=0 cant_write=1
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/09ecd083-a36b-4bae-b6cd-ce58aa847f1c': probe done, cant_read=0 cant_write=1
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6b896543-5dcb-44d9-841f-f4cbe4c96a7b': probe done, cant_read=0 cant_write=1
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4c6c7b1a-8a6d-4258-bd74-bf136de81a76': probe done, cant_read=0 cant_write=1
1749587983   ffff8c4ae3440000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6b896543-5dcb-44d9-841f-f4cbe4c96a7b': best uberblock found for spa twowaymirrored. txg 64472
1749587983   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(twowaymirrored, config untrusted): using uberblock with txg=64472
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/09ecd083-a36b-4bae-b6cd-ce58aa847f1c': probe done, cant_read=0 cant_write=0
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/d285472a-77b3-4451-9406-cd790f675b7f': probe done, cant_read=0 cant_write=0
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/f811fbee-c88d-4ab6-a720-3f020e1d24ea': probe done, cant_read=0 cant_write=0
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/c68a7ecc-e637-4b21-bdd8-b1980224a4c1': probe done, cant_read=0 cant_write=0
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4c6c7b1a-8a6d-4258-bd74-bf136de81a76': probe done, cant_read=0 cant_write=0
1749587983   ffff8c4998894bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/6b896543-5dcb-44d9-841f-f4cbe4c96a7b': probe done, cant_read=0 cant_write=0
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading checkpoint txg
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading indirect vdev metadata
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Checking feature flags
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading special MOS directories
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading properties
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading AUX vdevs
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading vdev metadata
1749587983   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(twowaymirrored, config trusted): Read 23 log space maps (23 total blocks - blksz = 131072 bytes) in 25 ms
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading dedup tables
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Loading BRT
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Verifying Log Devices
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Verifying pool data
1749587983   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(twowaymirrored, config trusted): spa_load_verify found 0 metadata errors and 2 data errors
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Calculating deflated space
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Starting import
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Claiming ZIL blocks
1749587983   ffff8c4ae3440000 mmp.c:239:mmp_thread_start(): MMP thread started pool 'twowaymirrored' gethrtime 38963695059
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Syncing ZIL claims
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Updating configs
1749587983   ffff8c4ae3440000 spa.c:9209:spa_async_request(): spa=twowaymirrored async request task=1
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Starting resilvers
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Restarting device removals
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Cleaning up inconsistent objsets
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Cleaning up temporary userrefs
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Restarting initialize
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Restarting TRIM
1749587983   ffff8c4ae3440000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'twowaymirrored' Finished importing
1749587983   ffff8c4ae3440000 spa.c:9209:spa_async_request(): spa=twowaymirrored async request task=2048
1749587983   ffff8c4ae3440000 spa_misc.c:429:spa_load_note(): spa_load(twowaymirrored, config trusted): LOADED
1749587983   ffff8c4c31b2e500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 64474, spa twowaymirrored, vdev_id 2, ms_id 69, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 38976 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749587983   ffff8c4c31b2e500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 64474, spa twowaymirrored, vdev_id 1, ms_id 68, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 38976 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
[...]
metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 2, ms_id 80, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 1, ms_id 80, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 0, ms_id 80, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 2, ms_id 81, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 1, ms_id 66, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 0, ms_id 67, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 2, ms_id 73, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 1, ms_id 73, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 0, ms_id 73, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 2, ms_id 74, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 1, ms_id 74, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 0, ms_id 74, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 2, ms_id 75, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 1, ms_id 75, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 0, ms_id 75, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 2, ms_id 76, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 1, ms_id 76, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 0, ms_id 76, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 2, ms_id 77, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 1, ms_id 77, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 0, ms_id 77, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 2, ms_id 78, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 1, ms_id 78, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 0, ms_id 78, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 2, ms_id 79, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 1, ms_id 79, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588603   ffff8c4ce62c0000 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 64598, spa twowaymirrored, vdev_id 0, ms_id 79, weight 880000000000001, selected txg 64480 (604320 ms ago), alloc_txg 0, loaded 620110 ms ago, max_size 17179869184
1749588653   ffff8c4ce4164bc0 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1749588653   ffff8c4ce4164bc0 spa.c:6916:spa_tryimport(): spa_tryimport: importing subramanya
1749588653   ffff8c4ce4164bc0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ce4164bc0-subramanya, config trusted): LOADING
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ce4164bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff8c4ce4164bc0-subramanya. txg 36292731
1749588653   ffff8c4ce4164bc0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ce4164bc0-subramanya, config untrusted): using uberblock with txg=36292731
1749588653   ffff8c4afc090000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ae2584bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749588653   ffff8c4cef9dcbc0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb75000
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749588653   ffff8c4ce4164bc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749588653   ffff8c4ce4164bc0 spa.c:9209:spa_async_request(): spa=$import-ffff8c4ce4164bc0-subramanya async request task=2048
1749588653   ffff8c4ce4164bc0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ce4164bc0-subramanya, config trusted): LOADED
1749588653   ffff8c4ce4164bc0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c4ce4164bc0-subramanya, config trusted): UNLOADING
1749588684   ffff8c499867b280 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1749588684   ffff8c499867b280 spa.c:6916:spa_tryimport(): spa_tryimport: importing subramanya
1749588684   ffff8c499867b280 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c499867b280-subramanya, config trusted): LOADING
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749588684   ffff8c499867b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff8c499867b280-subramanya. txg 36292731
1749588684   ffff8c499867b280 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c499867b280-subramanya, config untrusted): using uberblock with txg=36292731
1749588684   ffff8c4c32b21940 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749588684   ffff8c495178cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749588684   ffff8c4c32b24bc0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb75000
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749588684   ffff8c499867b280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749588684   ffff8c499867b280 spa.c:9209:spa_async_request(): spa=$import-ffff8c499867b280-subramanya async request task=2048
1749588684   ffff8c499867b280 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c499867b280-subramanya, config trusted): LOADED
1749588684   ffff8c499867b280 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c499867b280-subramanya, config trusted): UNLOADING
1749588981   ffff8c49511b9940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1749588981   ffff8c49511b9940 spa.c:6913:spa_tryimport(): spa_tryimport: importing subramanya, max_txg=36292723
1749588981   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c49511b9940-subramanya, config trusted): LOADING
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749588981   ffff8c49511b9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff8c49511b9940-subramanya. txg 36292723
1749588981   ffff8c49511b9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292723)
1749588981   ffff8c49511b9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config. Trying again without txg restrictions.
1749588981   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c49511b9940-subramanya, config untrusted): using uberblock with txg=36292723
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4ced9acbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749588981   ffff8c49511b9940 spa.c:9209:spa_async_request(): spa=$import-ffff8c49511b9940-subramanya async request task=2048
1749588981   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c49511b9940-subramanya, config trusted): LOADED
1749588981   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8c49511b9940-subramanya, config trusted): UNLOADING
1749588981   ffff8c49511b9940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1749588981   ffff8c49511b9940 spa.c:6761:spa_import(): spa_import: importing subramanya
1749588981   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749588981   ffff8c49511b9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292723
1749588981   ffff8c49511b9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292723)
1749588981   ffff8c49511b9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config. Trying again without txg restrictions.
1749588981   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292723
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0
1749588981   ffff8c4aff5fb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=0
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749588981   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749588982   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): Read 6 log space maps (6 total blocks - blksz = 131072 bytes) in 10 ms
1749588982   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749588982   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749588982   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749588982   ffff8c49502d9940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 0, spa subramanya, vdev_id 0, ms_id 266, smp_length 279704, unflushed_allocs 516096, unflushed_frees 663552, freed 0, defer 0 + 0, unloaded time 1038314 ms, loading_time 11 ms, ms_max_size 128664842240, max size error 128664645632, old_weight 900000000000001, new_weight 900000000000001
1749588982   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749588982   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
  (spa_load_verify_data=1, spa_load_verify_metadata=1)
1749640145   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 0 metadata errors and 3 data errors
1749640145   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749640145   ffff8c49511b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749640145   ffff8c49511b9940 spa.c:9209:spa_async_request(): spa=subramanya async request task=2048
1749640145   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADED

alas, just like last time, the import process doesn’t return, even though the IOs are flat on the devices.
The fact that it is in D state doesn’t seem to matter, zpool import seem to spend a lot of time in D state.

PK1048 · June 11, 2025, 12:58pm

No. The dryrun process very specifically makes no changes.

I suspect one or more of your disks has damage due to running at excessive temperature and that is causing two things:

An I/O block which is hanging the import
Causing the disk to return unreliable data (which explains why you get different results)

Getting this pool cleaned up enough to import may be a lost cause due to the above.

HoneyBadger · June 11, 2025, 2:08pm

Here’s where you’re getting stuck at from my understanding:

1749588982   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
  (spa_load_verify_data=1, spa_load_verify_metadata=1)
1749640145   ffff8c49511b9940 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 0 metadata errors and 3 data errors

Because we’re at the point of 0 metadata errors and 3 data errors, this may be the point at which the tunables to remove the safety checks on import could be toggled - and you see if it comes online in read-only mode, then determine if zpool status -v will give you a list of the damaged files. If you’re willing to accept the impact, then you export the pool, re-import r/w and commit it.

Warning: Beyond this point are tunables that can permanently commit errors to a pool as a last resort. Do not enable without full understanding of the consequences.

To disable the error checking, get to a root prompt on TrueNAS (sudo -s) and then issue

echo 0 > /sys/module/zfs/parameters/spa_load_verify_metadata
echo 0 > /sys/module/zfs/parameters/spa_load_verify_data

Once this is done, attempt the pool import with

zpool import -fFXT 36292723 -R /mnt -o readonly=on subramanya

If this works, attempt a zpool status -v and see if it can identify the known-damaged files that were listed before.

zpool export can unmount the pool.

To import read-write and commit, remove the -o readonly=on parameter.

coolnodje · June 11, 2025, 4:02pm

zpool import -fFXT 36292723 -R /mnt -o readonly=on subramanya returned immediately.

zpool status -v output is

pool: subramanya                                       
state: ONLINE                                          
status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.                                    
action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'.                                            
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: resilvered 2.91M in 00:00:01 with 0 errors on Wed Jun  4 03:22:02 2025
config:

        NAME                                      STATE
    READ WRITE CKSUM
        subramanya                                ONLINE
       0     0     0
          raidz2-0                                ONLINE
       0     0     0
            69e33d70-2e29-4440-bafd-0f183720274b  ONLINE
       0     0     0
            90aea466-1527-4960-ad48-98bc5f0ffd21  ONLINE
       0     0     0
            ddb020b8-2290-495d-8343-089f94d31ad4  ONLINE
       0     0     0
            4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7  ONLINE
       0     0     0
            e6d7bead-7310-43d8-8236-9aba92acd3c8  ONLINE
       0     0     0
            33870927-1683-4d16-a597-7efee3e7b405  ONLINE
       0     0     1

errors: No known data errors

I don’t know what to make out this.
I don’t see any know-damaged file.
Is that a good or bad sign?

winnielinnie · June 11, 2025, 4:03pm

winnielinnie · June 11, 2025, 4:05pm

The next step would be to reboot and run a full scrub on the pool.

Make sure no tunables are still applied.

EDIT: Maybe not. Since the pool only successfully imported as “readonly”, @HoneyBadger might advise a different next step before doing a full scrub. (e.g, importing the pool normally, without the “readonly” option.)

coolnodje · June 11, 2025, 4:06pm

I wish I was doing
You caught me on a mobile with multiple language keyboard enabled…