Bringing back a SUSPENDED pool that doesn't show up with `zpool status`

TrueNAS General
61

Just reporting back from some back and forth in DM’s - there might be issues even in the earlier transaction groups.

@coolnodje I’d recommend moving the drives to your new server (R730XD) as it will significantly help the cooling of the disks there. You can attempt an import at the earlier txg number but based on the dbgmsg output it is still calling out metadata and data errors. There are tunables that can be set to ignore them but that obviously results in the errors becoming permanent.

As an additional note, please ensure that the storage controller in your R730XD is either a true HBA (PERC HBA330) or has been reflashed/set to behave as one (a crossflashed H330 will work, an H730 in HBA mode is sub-optimal but will work as well) but ensure that it is not a “RAID card” attempting to present virtual disks.

2 Likes
62

The new server has a SAS9305-16I LSI controller.

I’ll be moving to it, I suspect the old one has hardware issues, I’m not even sure the heat is the source problem. Maybe.

According to the errors, shall I still aim for
zpool import -nfFXT 36292723 subramanya
or shall I use the earlier txg=36292722 that was notified in the log ?

63

You can still aim for 36292723 as the hopeful insert point but since it rewound from there regardless it may require even more drastic levels of recovery.

64

After cleaning up a dust filter which significantly improved the air flow which lowered HDD operating temperature to below 35C, I decided to go for a dry run on the current SilverStone D380 instance:
zpool import -nfFXT 36292723 subramanya

It started at 18:35, everything was looking good hardware and log (/proc/spl/kstat/zfs/dbgmsg) wise.

Here the result this morning, the process stopped at 5:34:

timestamp    message 
1749227306   ffff9155d1f98000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
[boot-pool pool]
1749227306   ffff9155ea29e300 spa_history.c:306:spa_history_log_sync(): txg 1571262 open pool version 5000; software version zfs-2.1.99-2799-gc71d2fb20; uts (none) 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Tue Jan 28 03:14:06 UTC 2025 x86_64
1749227306   ffff9155d1f98000 spa.c:9204:spa_async_request(): spa=boot-pool async request task=32
1749227306   ffff9155ea29e300 spa_history.c:306:spa_history_log_sync(): txg 1571264 import pool version 5000; software version zfs-2.1.99-2799-gc71d2fb20; uts (none) 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Tue Jan 28 03:14:06 UTC 2025 x86_64
1749227306   ffff9155ed724a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 87, smp_length 59232, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11350 ms, loading_time 2 ms, ms_max_size 2146365440, max size error 2146365440, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ecda4a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 106, smp_length 270312, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11350 ms, loading_time 11 ms, ms_max_size 2124718080, max size error 2124718080, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ed724a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 88, smp_length 285864, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11353 ms, loading_time 16 ms, ms_max_size 2139947008, max size error 2139947008, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ed724a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 93, smp_length 49096, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11370 ms, loading_time 2 ms, ms_max_size 2145525760, max size error 2145525760, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ecda4a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 91, smp_length 197624, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11362 ms, loading_time 12 ms, ms_max_size 2128207872, max size error 2128207872, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ea3a98c0 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 90, smp_length 386792, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11353 ms, loading_time 22 ms, ms_max_size 1225740288, max size error 1225740288, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ea3ae300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571264, spa boot-pool, vdev_id 0, ms_id 92, smp_length 376200, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11362 ms, loading_time 18 ms, ms_max_size 1781149696, max size error 1781149696, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ed724a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571265, spa boot-pool, vdev_id 0, ms_id 94, smp_length 259128, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11372 ms, loading_time 11 ms, ms_max_size 1314934784, max size error 1314934784, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ea3a98c0 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571265, spa boot-pool, vdev_id 0, ms_id 96, smp_length 306088, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11376 ms, loading_time 15 ms, ms_max_size 1138450432, max size error 1138450432, old_weight 780000000000001, new_weight 780000000000001
1749227306   ffff9155ecda4a40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1571265, spa boot-pool, vdev_id 0, ms_id 95, smp_length 411600, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11374 ms, loading_time 19 ms, ms_max_size 1133998080, max size error 1133998080, old_weight 780000000000001, new_weight 780000000000001
1749227308   ffff9155ea29e300 spa_history.c:293:spa_history_log_sync(): command: zpool import -N -f boot-pool
1749227347   ffff9155ea3b0000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749227347   ffff9155ea3b0000 spa.c:6916:spa_tryimport(): spa_tryimport: importing subramanya
1749227347   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ea3b0000-subramanya, config trusted): LOADING
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff9155ea3b0000-subramanya. txg 36292731
1749227347   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ea3b0000-subramanya, config untrusted): using uberblock with txg=36292731
1749227347   ffff9155e5baca40 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ef054a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749227347   ffff9155d9343180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb75000
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749227347   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749227348   ffff9155ea3b0000 spa.c:9204:spa_async_request(): spa=$import-ffff9155ea3b0000-subramanya async request task=2048
1749227348   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ea3b0000-subramanya, config trusted): LOADED
1749227348   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ea3b0000-subramanya, config trusted): UNLOADING
1749227348   ffff9155ea3b0000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749227348   ffff9155ea3b0000 spa.c:6761:spa_import(): spa_import: importing subramanya
1749227348   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227348   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292731
1749227348   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292731
1749227348   ffff9155e855b180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=0
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749227348   ffff9155e9b6b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749227348   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749227349   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): Read 7 log space maps (7 total blocks - blksz = 131072 bytes) in 16 ms
1749227349   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749227349   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749227349   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749227349   ffff9155e75fe300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 0, spa subramanya, vdev_id 0, ms_id 266, smp_length 279784, unflushed_allocs 417792, unflushed_frees 430080, freed 0, defer 0 + 0, unloaded time 54703 ms, loading_time 65 ms, ms_max_size 128653340672, max size error 128653168640, old_weight 900000000000001, new_weight 900000000000001
1749227350   ffff9155ea3b0000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 1 metadata errors and 2 data errors
1749227350   ffff9155ea3b0000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config trusted): FAILED: spa_load_verify failed [error=5]
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): UNLOADING
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_retry: rewind, max txg: 36292730
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227350   ffff915670688000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292730
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292730)
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292730
1749227350   ffff9155ea3b0000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): spa_load_retry: rewind, max txg: 36292729
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): LOADING
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227350   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292729
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292729)
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292729
1749227350   ffff9155ea3b0000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): spa_load_retry: rewind, max txg: 36292728
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): LOADING
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea7298c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292728
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292728)
1749227350   ffff9155ea3b0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292728
1749227350   ffff9155ea3b0000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749227350   ffff9155ea3b0000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749227350   ffff9155ea2f0000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
[backup pool]
[FAST pool]
1749227550   ffff915701310000 spa.c:6916:spa_tryimport(): spa_tryimport: importing subramanya
1749227550   ffff915701310000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff915701310000-subramanya, config trusted): LOADING
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227550   ffff915701310000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff915701310000-subramanya. txg 36292731
1749227550   ffff915701310000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff915701310000-subramanya, config untrusted): using uberblock with txg=36292731
1749227550   ffff9156706a98c0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227550   ffff9155e65e4a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749227550   ffff915701a7ca40 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb75000
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749227550   ffff915701310000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749227550   ffff915701310000 spa.c:9204:spa_async_request(): spa=$import-ffff915701310000-subramanya async request task=2048
1749227550   ffff915701310000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff915701310000-subramanya, config trusted): LOADED
1749227550   ffff915701310000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff915701310000-subramanya, config trusted): UNLOADING
1749227691   ffff9155ee2f8000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749227691   ffff9155ee2f8000 spa.c:6913:spa_tryimport(): spa_tryimport: importing subramanya, max_txg=36292723
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ee2f8000-subramanya, config trusted): LOADING
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff9155ee2f8000-subramanya. txg 36292723
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292723)
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config. Trying again without txg restrictions.
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ee2f8000-subramanya, config untrusted): using uberblock with txg=36292723
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ea3a98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749227691   ffff9155ee2f8000 spa.c:9204:spa_async_request(): spa=$import-ffff9155ee2f8000-subramanya async request task=2048
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ee2f8000-subramanya, config trusted): LOADED
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9155ee2f8000-subramanya, config trusted): UNLOADING
1749227691   ffff9155ee2f8000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749227691   ffff9155ee2f8000 spa.c:6761:spa_import(): spa_import: importing subramanya
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292723
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292723)
1749227691   ffff9155ee2f8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config. Trying again without txg restrictions.
1749227691   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292723
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=0
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749227691   ffff9156733dca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749227691   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749227692   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): Read 6 log space maps (6 total blocks - blksz = 131072 bytes) in 7 ms
1749227692   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749227692   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749227692   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749227692   ffff9155ea28e300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 0, spa subramanya, vdev_id 0, ms_id 266, smp_length 279704, unflushed_allocs 516096, unflushed_frees 663552, freed 0, defer 0 + 0, unloaded time 397401 ms, loading_time 187 ms, ms_max_size 128664842240, max size error 128664645632, old_weight 900000000000001, new_weight 900000000000001
1749227693   ffff9155ee2f8000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749227693   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
  (spa_load_verify_data=1, spa_load_verify_metadata=1)
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 44, weight 840000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 1237761, loaded 619794 ms ago, max_size 17179643904
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 52, weight 880000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 0, loaded 619480 ms ago, max_size 17179869184
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 45, weight 880000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 0, loaded 619480 ms ago, max_size 17179869184
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 53, weight 880000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 0, loaded 619480 ms ago, max_size 17179869184
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 46, weight 880000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 0, loaded 619480 ms ago, max_size 17179869184
1749227971   ffff9155d3cbca40 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237881, spa backup, vdev_id 0, ms_id 47, weight 880000000000001, selected txg 1237763 (603843 ms ago), alloc_txg 0, loaded 619480 ms ago, max_size 17179869184
1749249416   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915626712ac0 repairing c=2 devidx=5 offset=54055f96000
1749249416   ffff9155d093b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91565392a600 repairing c=2 devidx=5 offset=54055f97000
1749249416   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915644891300 repairing c=2 devidx=5 offset=54055f98000
1749249416   ffff9156733de300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91598952f200 repairing c=2 devidx=5 offset=54055f8f000
1749249416   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157165fd580 repairing c=2 devidx=5 offset=54055f8e000
1749249416   ffff9157002c0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91564f474740 repairing c=2 devidx=5 offset=54055f94000
1749249416   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91594997df00 repairing c=2 devidx=5 offset=54055f8d000
1749249416   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91564fb16d40 repairing c=2 devidx=5 offset=54055f8b000
1749249418   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157d2ace880 repairing c=3 devidx=5 offset=535d51a6000
1749249419   ffff9156733de300 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0dc000
1749249419   ffff9156733de300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91594558f6c0 repairing c=1 devidx=5 offset=53c4f0dc000
1749249419   ffff9155e300b180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0df000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915648111300 repairing c=1 devidx=5 offset=53c4f0df000
1749249419   ffff9156733de300 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0da000
1749249419   ffff9156733de300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156277097c0 repairing c=1 devidx=5 offset=53c4f0da000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0de000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156cdf804c0 repairing c=1 devidx=5 offset=53c4f0de000
1749249419   ffff9155e300b180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0db000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915627708980 repairing c=1 devidx=5 offset=53c4f0db000
1749249419   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915903e9d0c0 repairing c=4 devidx=5 offset=53c4f0e0000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0d9000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915841f71c80 repairing c=1 devidx=5 offset=53c4f0d9000
1749249419   ffff9157002c0000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0cb000
1749249419   ffff9157002c0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91590a4df200 repairing c=1 devidx=5 offset=53c4f0cb000
1749249419   ffff9156733de300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91583e94df00 repairing c=4 devidx=5 offset=53c4f0d3000
1749249419   ffff9157002c0000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0c8000
1749249419   ffff9157002c0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156481d3440 repairing c=1 devidx=5 offset=53c4f0c8000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915627bd63c0 repairing c=4 devidx=5 offset=53c4f0d8000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9158be5eb900 repairing c=4 devidx=5 offset=53c4f0d6000
1749249419   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9155e4ba2ac0 repairing c=4 devidx=5 offset=53c4f0d5000
1749249419   ffff9157002c0000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0b8000
1749249419   ffff9157002c0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157d1a76d40 repairing c=1 devidx=5 offset=53c4f0b8000
1749249419   ffff9156733de300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915901b83900 repairing c=4 devidx=5 offset=53c4f0c2000
1749249419   ffff9157002c0000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0ae000
1749249419   ffff9157002c0000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156cdf85f00 repairing c=1 devidx=5 offset=53c4f0ae000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff91584073e880 repairing c=4 devidx=5 offset=53c4f0d2000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157d3762f80 repairing c=4 devidx=5 offset=53c4f0d4000
1749249419   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915652e2a140 repairing c=4 devidx=5 offset=53c4f0d1000
1749249419   ffff9155e300b180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0cf000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915626d4bdc0 repairing c=1 devidx=5 offset=53c4f0cf000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915626d49c80 repairing c=4 devidx=5 offset=53c4f0d0000
1749249419   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156d0f94c00 repairing c=4 devidx=5 offset=53c4f0ce000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157128a50c0 repairing c=4 devidx=5 offset=53c4f0cd000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0ca000
1749249419   ffff9156706b3180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0c9000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156538d4c00 repairing c=1 devidx=5 offset=53c4f0ca000
1749249419   ffff9156706b3180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156481d0000 repairing c=1 devidx=5 offset=53c4f0c9000
1749249419   ffff9155e300b180 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0c7000
1749249419   ffff9155e300b180 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9158424b3dc0 repairing c=1 devidx=5 offset=53c4f0c7000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=1 devidx=5 off 53c4f0c6000
1749249419   ffff9155d0a898c0 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9157f892f6c0 repairing c=1 devidx=5 offset=53c4f0c6000
[…]
1749267336   ffff9155db026300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156482bf6c0 repairing c=1 devidx=0 offset=1357eff2000
1749267336   ffff9155db026300 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff9156482bf6c0 repairing c=2 devidx=1 offset=1357eff2000
1749267336   ffff9155d2d48000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915714ed2600 repairing c=1 devidx=0 offset=1357c56d000
1749267336   ffff9155d2d48000 vdev_raidz.c:2780:vdev_raidz_io_done_verified(): zio=ffff915714ed2600 repairing c=2 devidx=1 offset=1357c56d000
1749267359   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 8 metadata errors and 17937 data errors
1749267359   ffff9155ee2f8000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config trusted): FAILED: spa_load_verify failed [error=6]
1749267359   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): UNLOADING
1749267363   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_retry: rewind, max txg: 36292722
1749267363   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749267363   ffff9155e8338000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749267363   ffff9155e8338000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749267363   ffff9155e8338000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749267364   ffff9156706a8000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': open error=2 timeout=1003438410/1000000000
1749267364   ffff9155e6704a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': open error=2 timeout=1007585577/1000000000
1749267364   ffff9155e6700000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': open error=2 timeout=1015360540/1000000000
1749267364   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): vdev tree has 1 missing top-level vdevs.
1749267364   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
1749267364   ffff9155ee2f8000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: unable to open vdev tree [error=2]
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 15005074635607672362, path: N/A, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 1843460257378462166, path: N/A, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 12560554049367260037, path: /dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 3504886088678984596, path: /dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 1026282999581038360, path: /dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4, healthy
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 4494896413352636629, path: /dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7, healthy
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 18426049172646806820, path: /dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 10710542068254203083, path: /dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405, healthy
1749267364   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING

I understand from the last lines that some harware issues made the process stop, because it couldn’t access some of the pool’s vdevs :

1749267364   ffff9155ee2f8000 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: unable to open vdev tree [error=2]
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 15005074635607672362, path: N/A, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 1843460257378462166, path: N/A, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 12560554049367260037, path: /dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 3504886088678984596, path: /dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 1026282999581038360, path: /dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4, healthy
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 4494896413352636629, path: /dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7, healthy
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 18426049172646806820, path: /dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8, can't open
1749267364   ffff9155ee2f8000 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 10710542068254203083, path: /dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405, healthy
1749267364   ffff9155ee2f8000 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING

Looking at kernel logs ( I discovered the versatile journalctl -o short-precise -k -b all and recommand it over dmesg -T):

Jun 06 18:56:27.668615 freenas kernel:  </TASK>
Jun 06 18:56:27.668714 freenas kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
Jun 07 02:26:38.828970 freenas kernel: perf: interrupt took too long (2536 > 2500), lowering kernel.perf_event_max_sample_rate to 78750
Jun 07 05:31:57.512769 freenas kernel: ata5.00: exception Emask 0x50 SAct 0x2600000 SErr 0x4090800 action 0xe frozen
Jun 07 05:31:57.513284 freenas kernel: ata5.00: irq_stat 0x00400040, connection status changed
Jun 07 05:31:57.513421 freenas kernel: ata5: SError: { HostInt PHYRdyChg 10B8B DevExch }
Jun 07 05:31:57.513548 freenas kernel: ata5.00: failed command: READ FPDMA QUEUED
Jun 07 05:31:57.518675 freenas kernel: ata5.00: cmd 60/00:a8:58:25:de/08:00:ab:01:00/40 tag 21 ncq dma 1048576 in
                                                res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x50 (ATA bus error)
Jun 07 05:31:57.518870 freenas kernel: ata5.00: status: { DRDY }
Jun 07 05:31:57.526984 freenas kernel: ata5.00: failed command: READ FPDMA QUEUED
Jun 07 05:31:57.527141 freenas kernel: ata5.00: cmd 60/10:b0:e0:37:de/00:00:ab:01:00/40 tag 22 ncq dma 8192 in
                                                res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x50 (ATA bus error)
Jun 07 05:31:57.527307 freenas kernel: ata5.00: status: { DRDY }
Jun 07 05:31:57.533832 freenas kernel: ata5.00: failed command: READ FPDMA QUEUED
Jun 07 05:31:57.533962 freenas kernel: ata5.00: cmd 60/80:c8:58:2d:de/00:00:ab:01:00/40 tag 25 ncq dma 65536 in
                                                res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x50 (ATA bus error)
Jun 07 05:31:57.534060 freenas kernel: ata5.00: status: { DRDY }
Jun 07 05:31:57.538968 freenas kernel: ata5: hard resetting link

One can clearly see some issues suddenly coming up at ~5:31, which roughly matches the last log from dbgmsg (Timestamp 1749267364) where it says it can’t access vdevs and stops.

Then the kernel log is full of these or the likes of:

zio pool=subramanya vdev=/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8 error=5 type=1 offset=1673710948352 size=4096 flags=1573248
Jun 07 05:35:22.957531 freenas kernel: I/O error, dev sdb, sector 2576 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

up until ~7:33 even though nothing runs on the HDD anymore, and then the log becomes quiet up until now.

The drives temperature and I/O graphs from TN Reporting are clean and consistent from the beginning of the operations 'till 5:34: all disk I/O identical, temperature stable.
Though there are some holes in temperature reporting on some drives, but matching the hardware issues, so it makes sense. But reported temp is never above 35C.

I’ll run the same zpool import -nfFXT 36292723 subramanya on the replacement box now.

1 Like
65

It’s concerning that it lists 3 devices as “can’t open”…

66

OK, on the new system Dell R730Xd+LSI SAS9305-16I, even though everything seem to be running smoothly with the import, I seem to be plagued with this bug: LSI 9305 errors filling log | TrueNAS Community

This gives me continuous kernel log and prevents the UI Storage Dashboard or Disks pages from being responsive.

I’m getting a flood of these:

Jun 07 12:14:13.152058 truenas kernel: mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)
Jun 07 12:14:13.152352 truenas kernel: mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)
Jun 07 12:14:13.152421 truenas kernel: mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)
Jun 07 12:14:13.156110 truenas kernel: sd 1:0:1:0: device_block, handle(0x001b)
Jun 07 12:14:13.162850 truenas kernel: sd 1:0:5:0: device_block, handle(0x001f)
Jun 07 12:14:13.168231 truenas kernel: sd 1:0:10:0: device_block, handle(0x0024)
Jun 07 12:14:14.151301 truenas kernel: mpt3sas_cm0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
Jun 07 12:14:14.151416 truenas kernel: mpt3sas_cm0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
Jun 07 12:14:14.152990 truenas kernel: mpt3sas_cm0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
Jun 07 12:14:14.401899 truenas kernel: sd 1:0:8:0: device_block, handle(0x0022)
Jun 07 12:14:14.407977 truenas kernel: sd 1:0:1:0: device_unblock and setting to running, handle(0x001b)
Jun 07 12:14:14.408700 truenas kernel: sd 1:0:5:0: device_unblock and setting to running, handle(0x001f)
Jun 07 12:14:14.409372 truenas kernel: sd 1:0:10:0: device_unblock and setting to running, handle(0x0024)
Jun 07 12:14:15.903098 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.903217 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.903249 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.903275 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.904845 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.905078 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:15.905200 truenas kernel: sd 1:0:6:0: device_block, handle(0x0020)
Jun 07 12:14:15.912005 truenas kernel: sd 1:0:8:0: device_unblock and setting to running, handle(0x0022)
Jun 07 12:14:16.903150 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:16.903300 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:16.903338 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:14:16.908047 truenas kernel: sd 1:0:6:0: device_unblock and setting to running, handle(0x0020)
Jun 07 12:14:17.162632 truenas kernel: sd 1:0:8:0: Power-on or device reset occurred
Jun 07 12:14:17.163172 truenas kernel: sd 1:0:3:0: Power-on or device reset occurred

(all yellow = warning, I think)
and earlier on some of:

Jun 07 12:07:28.150756 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:07:28.150782 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:07:28.150816 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:07:28.152138 truenas kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Jun 07 12:07:28.152208 truenas kernel: sd 1:0:3:0: [sdd] tag#3923 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
Jun 07 12:07:28.152726 truenas kernel: sd 1:0:3:0: [sdd] tag#3923 CDB: Read(16) 88 00 00 00 00 05 74 ff ff 08 00 00 00 78 00 00
Jun 07 12:07:28.153131 truenas kernel: I/O error, dev sdd, sector 23437770504 op 0x0:(READ) flags 0x80700 phys_seg 15 prio class 2
Jun 07 12:07:28.153167 truenas kernel: sd 1:0:3:0: [sdd] tag#3920 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
Jun 07 12:07:28.153586 truenas kernel: sd 1:0:3:0: [sdd] tag#3920 CDB: Read(16) 88 00 00 00 00 05 74 ff ff 88 00 00 00 38 00 00
Jun 07 12:07:28.153984 truenas kernel: I/O error, dev sdd, sector 23437770632 op 0x0:(READ) flags 0x80700 phys_seg 7 prio class 2
Jun 07 12:07:28.154028 truenas kernel: sd 1:0:3:0: [sdd] tag#3919 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
Jun 07 12:07:28.154408 truenas kernel: sd 1:0:3:0: [sdd] tag#3919 CDB: Read(16) 88 00 00 00 00 05 74 ff ff c8 00 00 00 28 00 00
Jun 07 12:07:28.154784 truenas kernel: I/O error, dev sdd, sector 23437770696 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
Jun 07 12:07:28.154816 truenas kernel: sd 1:0:3:0: [sdd] tag#3918 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
Jun 07 12:07:28.155191 truenas kernel: sd 1:0:3:0: [sdd] tag#3918 CDB: Read(16) 88 00 00 00 00 05 74 ff fe 78 00 00 00 80 00 00
Jun 07 12:07:28.155556 truenas kernel: I/O error, dev sdd, sector 23437770360 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2
Jun 07 12:07:28.155602 truenas kernel: sd 1:0:3:0: [sdd] tag#3917 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
Jun 07 12:07:28.156028 truenas kernel: sd 1:0:3:0: [sdd] tag#3917 CDB: Read(16) 88 00 00 00 00 05 74 ff fe 00 00 00 00 70 00 00
Jun 07 12:07:28.156416 truenas kernel: I/O error, dev sdd, sector 23437770240 op 0x0:(READ) flags 0x80700 phys_seg 14 prio class 2
Jun 07 12:07:28.405131 truenas kernel: sd 1:0:3:0: Power-on or device reset occurred
Jun 07 12:07:28.405629 truenas kernel: sd 1:0:10:0: Power-on or device reset occurred

with I/O error in Red

It’s very annoying but it seems ok.