That line in the zdb -l output identifies the labels it was able to read. I believe zdb -l will show the contents of more than one label if they do not match, but I have not run into that (mis-matched labels, they have always either been there, or not or corrupt in the same way).
FYI Jira
1 Like
So Iāve been tagged in here to look at this case of weird mismatching txg values and labels that just seem to have gone silly.
From a shell, in your home directory on the boot drive, can you do a dd if=/dev/sda of=sda.bin count=32 bs=1M - after that, youāll need to SCP/SFTP into your system and retrieve those files. Then Iāll DM you a place to upload them to.
Basically, I want to see if somethingās amiss with your partition table and/or the magic ZFS labels on the drive or partition itself. That chunk of 32MB off each drive should contain both label 0 and 1.
@coolnodje In addition to gathering the 32M chunks - I donāt believe the zpool history can target an exported pool, so a written one will have to suffice - can you describe the version of TrueNAS you were on when the pool was created, and any upgrades that have happened since? I get the impression that itās fairly new.
@HoneyBadger Let me know the place you want me to upload the files to.
You want a 32Mb chunk from each drive, from sda to sdf correct?
This pool is actually fairly old. It was my initial pool back in, probably, 2012 I think, starting with Freenas 9.3 or so. It evolved all along with each version, and grew from 6x3TB ā 6x4TB and finally 6x12TB
Any pointer on pool labels documentation? I still donāt get how it works really.
1 Like
And just for giggles, letās get the contents of /proc/spl/kstat/zfs/dbgmsg immediately following the zpool import subramanya command.
Correct.
If youāre able, at which point did you migrate from CORE to SCALE/CE - and was there a capacity expansion after the migration?
You want a paste here or a file along with the chunks?
AFAICT It seems to basically choke on the txg: label discarded as txg is too large (36292731 > 36292730)
The CORE to SCALE migration happened august 2024.
The capacity expansion from 6TB->12TB happened after it, in December or January 2025 I think
A paste here will be fine, it should be fairly short. Just mostly wanting to see where itās starting the txg walk and how far it goes back before hurking.
Iām assuming you went from CORE 13 ā SCALE 24.04 at that point, and the online expand from 6T->12T happened under 24.10 - sounds correct?
I do not have any, I do not know if @HoneyBadger does.
Essentially ZFS labels are written to the start and end of each logical disk (disk or partition) used by ZFS. There are 4 copies for redundancy, 2 at the start and 2 at the end. Each label has enough information to reassemble the zpool it is a component of, including identifying where it fits in the zpool as well as all the other components in the zpool. The zpool and all itās components are identified via unique identifiers so that duplicate names do not confuse it (if you want to import a zpool and there are duplicate zpool names you have to import by unique identifier).
So there are sections for the zpool (including things like most recent transaction group (txg), so you know if a given component needs to be resilvered and starting from when).
And sections for the component vdevs, with a section for this component as well as for all the others. Including most recent TXG so you can fit together the timeline of what went offline when (to a certain degree).
Iām assuming you went from CORE 13 ā SCALE 24.04 at that point, and the online expand from 6T->12T happened under 24.10 - sounds correct?
You assume right 
Though Iām not sure I remember when 24.10 was out and I upgraded. But you know the release dates better than me, and I tend to upgrade quickly.
timestamp message
1749153521 ffff9ad2d0078000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749153521 ffff9ad2d0078000 spa.c:6916:spa_tryimport(): spa_tryimport: importing boot-pool
1749153521 ffff9ad2d0078000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2d0078000-boot-pool, config trusted): LOADING
1749153521 ffff9ad2d0c34a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdh3': probe done, cant_read=0 cant_write=1
1749153521 ffff9ad2d0078000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdh3': best uberblock found for spa $import-ffff9ad2d0078000-boot-pool. txg 1557561
1749153521 ffff9ad2d0078000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2d0078000-boot-pool, config untrusted): using uberblock with txg=1557561
1749153521 ffff9ad2d0c34a40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdh3': probe done, cant_read=0 cant_write=1
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading checkpoint txg
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading indirect vdev metadata
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Checking feature flags
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading special MOS directories
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading properties
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading AUX vdevs
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading vdev metadata
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading dedup tables
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading BRT
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Verifying Log Devices
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Verifying pool data
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Calculating deflated space
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Starting import
1749153521 ffff9ad2d0078000 spa.c:9204:spa_async_request(): spa=$import-ffff9ad2d0078000-boot-pool async request task=2048
1749153521 ffff9ad2d0078000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2d0078000-boot-pool, config trusted): LOADED
1749153521 ffff9ad2d0078000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2d0078000-boot-pool, config trusted): UNLOADING
1749153521 ffff9ad2d0078000 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749153521 ffff9ad2d0078000 spa.c:6761:spa_import(): spa_import: importing boot-pool
1749153521 ffff9ad2d0078000 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): LOADING
1749153521 ffff9ad2ea228000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdh3': probe done, cant_read=0 cant_write=1
1749153521 ffff9ad2d0078000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdh3': best uberblock found for spa boot-pool. txg 1557561
1749153521 ffff9ad2d0078000 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config untrusted): using uberblock with txg=1557561
1749153521 ffff9ad2ea228000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdh3': probe done, cant_read=0 cant_write=0
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading checkpoint txg
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading indirect vdev metadata
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Checking feature flags
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading special MOS directories
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading properties
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading AUX vdevs
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading vdev metadata
1749153521 ffff9ad2d0078000 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): Read 6 log space maps (6 total blocks - blksz = 131072 bytes) in 1 ms
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading dedup tables
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading BRT
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Verifying Log Devices
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Verifying pool data
1749153521 ffff9ad2d0078000 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): spa_load_verify found 0 metadata errors and 1 data errors
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Calculating deflated space
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Starting import
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Claiming ZIL blocks
1749153521 ffff9ad2d0078000 mmp.c:239:mmp_thread_start(): MMP thread started pool 'boot-pool' gethrtime 11237580669
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Syncing ZIL claims
1749153521 ffff9ad2ed40ca40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557562, spa boot-pool, vdev_id 0, ms_id 21, smp_length 521992, unflushed_allocs 86016, unflushed_frees 266240, freed 0, defer 0 + 0, unloaded time 11237 ms, loading_time 20 ms, ms_max_size 1429790720, max size error 1429696512, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ed40ca40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557562, spa boot-pool, vdev_id 0, ms_id 25, smp_length 238240, unflushed_allocs 192512, unflushed_frees 200704, freed 0, defer 0 + 0, unloaded time 11258 ms, loading_time 10 ms, ms_max_size 1196363776, max size error 1196281856, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ed463180 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557562, spa boot-pool, vdev_id 0, ms_id 81, smp_length 30272, unflushed_allocs 405504, unflushed_frees 135168, freed 0, defer 0 + 0, unloaded time 11268 ms, loading_time 3 ms, ms_max_size 1582977024, max size error 1582911488, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ed463180 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557562, spa boot-pool, vdev_id 0, ms_id 85, smp_length 13240, unflushed_allocs 192512, unflushed_frees 126976, freed 0, defer 0 + 0, unloaded time 11271 ms, loading_time 0 ms, ms_max_size 1521889280, max size error 1521811456, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ed40ca40 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557562, spa boot-pool, vdev_id 0, ms_id 86, smp_length 300584, unflushed_allocs 135168, unflushed_frees 135168, freed 0, defer 0 + 0, unloaded time 11273 ms, loading_time 12 ms, ms_max_size 1359093760, max size error 1359052800, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Updating configs
1749153521 ffff9ad2d0078000 spa.c:9204:spa_async_request(): spa=boot-pool async request task=1
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Starting resilvers
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Restarting device removals
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Cleaning up inconsistent objsets
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Cleaning up temporary userrefs
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Restarting initialize
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Restarting TRIM
1749153521 ffff9ad2d0078000 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Finished importing
1749153521 ffff9ad2d0078000 spa.c:9204:spa_async_request(): spa=boot-pool async request task=2048
1749153521 ffff9ad2d0078000 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): LOADED
1749153521 ffff9ad2ea343180 spa_history.c:306:spa_history_log_sync(): txg 1557563 open pool version 5000; software version zfs-2.1.99-2799-gc71d2fb20; uts (none) 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Tue Jan 28 03:14:06 UTC 2025 x86_64
1749153521 ffff9ad2d0078000 spa.c:9204:spa_async_request(): spa=boot-pool async request task=32
1749153521 ffff9ad2ea343180 spa_history.c:306:spa_history_log_sync(): txg 1557565 import pool version 5000; software version zfs-2.1.99-2799-gc71d2fb20; uts (none) 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Tue Jan 28 03:14:06 UTC 2025 x86_64
1749153521 ffff9ad2ed426300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557565, spa boot-pool, vdev_id 0, ms_id 87, smp_length 59232, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11342 ms, loading_time 2 ms, ms_max_size 2146365440, max size error 2146365440, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ee296300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557565, spa boot-pool, vdev_id 0, ms_id 82, smp_length 46512, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11342 ms, loading_time 3 ms, ms_max_size 1684971520, max size error 1684971520, old_weight 740000000000001, new_weight 780000000000001
1749153521 ffff9ad2ee296300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557565, spa boot-pool, vdev_id 0, ms_id 91, smp_length 197624, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11345 ms, loading_time 17 ms, ms_max_size 2128207872, max size error 2128207872, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ed426300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557565, spa boot-pool, vdev_id 0, ms_id 88, smp_length 285864, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11344 ms, loading_time 18 ms, ms_max_size 2139947008, max size error 2139947008, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ee296300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557565, spa boot-pool, vdev_id 0, ms_id 93, smp_length 49096, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11362 ms, loading_time 1 ms, ms_max_size 2145525760, max size error 2145521664, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ee293180 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557565, spa boot-pool, vdev_id 0, ms_id 90, smp_length 386792, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11345 ms, loading_time 22 ms, ms_max_size 1225740288, max size error 1225740288, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ee2918c0 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557565, spa boot-pool, vdev_id 0, ms_id 92, smp_length 376200, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11345 ms, loading_time 23 ms, ms_max_size 1781149696, max size error 1781149696, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ed426300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557565, spa boot-pool, vdev_id 0, ms_id 94, smp_length 259128, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11363 ms, loading_time 13 ms, ms_max_size 1314934784, max size error 1314934784, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ee293180 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557566, spa boot-pool, vdev_id 0, ms_id 96, smp_length 306088, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11367 ms, loading_time 14 ms, ms_max_size 1138450432, max size error 1138450432, old_weight 780000000000001, new_weight 780000000000001
1749153521 ffff9ad2ee296300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1557566, spa boot-pool, vdev_id 0, ms_id 95, smp_length 411600, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 11364 ms, loading_time 20 ms, ms_max_size 1133998080, max size error 1133998080, old_weight 780000000000001, new_weight 780000000000001
1749153523 ffff9ad2ea343180 spa_history.c:293:spa_history_log_sync(): command: zpool import -N -f boot-pool
1749153562 ffff9ad2e6f13180 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749153562 ffff9ad2e6f13180 spa.c:6916:spa_tryimport(): spa_tryimport: importing subramanya
1749153562 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2e6f13180-subramanya, config trusted): LOADING
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff9ad2e6f13180-subramanya. txg 36292731
1749153562 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2e6f13180-subramanya, config untrusted): using uberblock with txg=36292731
1749153562 ffff9ad2ea220000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eee93180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749153562 ffff9ad2eaf08000 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb75000
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749153562 ffff9ad2e6f13180 spa.c:9204:spa_async_request(): spa=$import-ffff9ad2e6f13180-subramanya async request task=2048
1749153562 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2e6f13180-subramanya, config trusted): LOADED
1749153562 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2e6f13180-subramanya, config trusted): UNLOADING
1749153562 ffff9ad2e6f13180 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749153562 ffff9ad2e6f13180 spa.c:6761:spa_import(): spa_import: importing subramanya
1749153562 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749153562 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292731
1749153562 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292731
1749153562 ffff9ad2e7784a40 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749153564 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): Read 7 log space maps (7 total blocks - blksz = 131072 bytes) in 21 ms
1749153564 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749153564 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749153564 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749153564 ffff9ad2ea358000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 0, spa subramanya, vdev_id 0, ms_id 266, smp_length 279784, unflushed_allocs 417792, unflushed_frees 430080, freed 0, defer 0 + 0, unloaded time 54285 ms, loading_time 58 ms, ms_max_size 128653340672, max size error 128653168640, old_weight 900000000000001, new_weight 900000000000001
1749153565 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 1 metadata errors and 2 data errors
1749153565 ffff9ad2e6f13180 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config trusted): FAILED: spa_load_verify failed [error=5]
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): UNLOADING
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_retry: rewind, max txg: 36292730
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749153565 ffff9ad2f720ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f720ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f720ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f720ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f720ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f720ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292730
1749153565 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292730)
1749153565 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292730
1749153565 ffff9ad2e6f13180 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): spa_load_retry: rewind, max txg: 36292729
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): LOADING
1749153565 ffff9ad2eb11e300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2eb11e300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2eb11e300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2eb11e300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2eb11e300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2eb11e300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292729
1749153565 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292729)
1749153565 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292729
1749153565 ffff9ad2e6f13180 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): spa_load_retry: rewind, max txg: 36292728
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): LOADING
1749153565 ffff9ad2f7218000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f7218000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f7218000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f7218000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f7218000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f7218000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292728
1749153565 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292728)
1749153565 ffff9ad2e6f13180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292728
1749153565 ffff9ad2e6f13180 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749153565 ffff9ad2f49d18c0 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749153565 ffff9ad2f49d18c0 spa.c:6916:spa_tryimport(): spa_tryimport: importing backup
1749153565 ffff9ad2f49d18c0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2f49d18c0-backup, config trusted): LOADING
1749153565 ffff9ad2f49d0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ead50f4e-eb70-419d-a31b-22821b2324a7': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f49d18c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ead50f4e-eb70-419d-a31b-22821b2324a7': best uberblock found for spa $import-ffff9ad2f49d18c0-backup. txg 1237749
1749153565 ffff9ad2f49d18c0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2f49d18c0-backup, config untrusted): using uberblock with txg=1237749
1749153565 ffff9ad2f49d0000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ead50f4e-eb70-419d-a31b-22821b2324a7': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading checkpoint txg
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading indirect vdev metadata
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Checking feature flags
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading special MOS directories
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading properties
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading AUX vdevs
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading vdev metadata
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading dedup tables
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading BRT
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Verifying Log Devices
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Verifying pool data
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Calculating deflated space
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Starting import
1749153565 ffff9ad2f49d18c0 spa.c:9204:spa_async_request(): spa=$import-ffff9ad2f49d18c0-backup async request task=2048
1749153565 ffff9ad2f49d18c0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2f49d18c0-backup, config trusted): LOADED
1749153565 ffff9ad2f49d18c0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2f49d18c0-backup, config trusted): UNLOADING
1749153565 ffff9ad2f49d18c0 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749153565 ffff9ad2f49d18c0 spa.c:6761:spa_import(): spa_import: importing backup
1749153565 ffff9ad2f49d18c0 spa_misc.c:429:spa_load_note(): spa_load(backup, config trusted): LOADING
1749153565 ffff9ad2f71c6300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ead50f4e-eb70-419d-a31b-22821b2324a7': probe done, cant_read=0 cant_write=1
1749153565 ffff9ad2f49d18c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ead50f4e-eb70-419d-a31b-22821b2324a7': best uberblock found for spa backup. txg 1237749
1749153565 ffff9ad2f49d18c0 spa_misc.c:429:spa_load_note(): spa_load(backup, config untrusted): using uberblock with txg=1237749
1749153565 ffff9ad2f71c6300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ead50f4e-eb70-419d-a31b-22821b2324a7': probe done, cant_read=0 cant_write=0
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading checkpoint txg
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading indirect vdev metadata
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Checking feature flags
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading special MOS directories
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading properties
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading AUX vdevs
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading vdev metadata
1749153565 ffff9ad2f49d18c0 spa_misc.c:429:spa_load_note(): spa_load(backup, config trusted): Read 16 log space maps (16 total blocks - blksz = 131072 bytes) in 65 ms
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading dedup tables
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Loading BRT
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Verifying Log Devices
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Verifying pool data
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Calculating deflated space
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Starting import
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Claiming ZIL blocks
1749153565 ffff9ad2f49d18c0 mmp.c:239:mmp_thread_start(): MMP thread started pool 'backup' gethrtime 55933137323
1749153565 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Syncing ZIL claims
1749153565 ffff9ad2f71c0000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237750, spa backup, vdev_id 0, ms_id 35, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 55933 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153565 ffff9ad2f71c0000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237750, spa backup, vdev_id 0, ms_id 36, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 55933 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153565 ffff9ad2f71c0000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237750, spa backup, vdev_id 0, ms_id 37, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 55933 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153565 ffff9ad2f71c0000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237750, spa backup, vdev_id 0, ms_id 38, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 55933 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153565 ffff9ad2f71c0000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237750, spa backup, vdev_id 0, ms_id 39, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 55933 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Updating configs
1749153566 ffff9ad2f49d18c0 spa.c:9204:spa_async_request(): spa=backup async request task=1
1749153566 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Starting resilvers
1749153566 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Restarting device removals
1749153566 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Cleaning up inconsistent objsets
1749153566 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Cleaning up temporary userrefs
1749153566 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Restarting initialize
1749153566 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Restarting TRIM
1749153566 ffff9ad2f49d18c0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'backup' Finished importing
1749153566 ffff9ad2f49d18c0 spa.c:9204:spa_async_request(): spa=backup async request task=2048
1749153566 ffff9ad2f49d18c0 spa_misc.c:429:spa_load_note(): spa_load(backup, config trusted): LOADED
1749153566 ffff9ad2f72098c0 spa_history.c:306:spa_history_log_sync(): txg 1237751 open pool version 5000; software version zfs-2.1.99-2799-gc71d2fb20; uts freenas 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Tue Jan 28 03:14:06 UTC 2025 x86_64
1749153566 ffff9ad2e06f18c0 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237751, spa backup, vdev_id 0, ms_id 40, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 56245 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2e2530000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237751, spa backup, vdev_id 0, ms_id 41, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 56245 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2e2530000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237751, spa backup, vdev_id 0, ms_id 42, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 56245 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2e2530000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237751, spa backup, vdev_id 0, ms_id 43, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 56245 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2e2530000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237751, spa backup, vdev_id 0, ms_id 44, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 56245 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2e2530000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237751, spa backup, vdev_id 0, ms_id 45, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 56245 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2e2530000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237751, spa backup, vdev_id 0, ms_id 46, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 56245 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2e2530000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237751, spa backup, vdev_id 0, ms_id 47, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 56245 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2e2530000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237751, spa backup, vdev_id 0, ms_id 48, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 56245 ms, loading_time 0 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2e06f18c0 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 1237752, spa backup, vdev_id 0, ms_id 8, smp_length 48, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 56245 ms, loading_time 28 ms, ms_max_size 17179869184, max size error 17179869184, old_weight 880000000000001, new_weight 880000000000001
1749153566 ffff9ad2f49d18c0 spa.c:9204:spa_async_request(): spa=backup async request task=32
1749153566 ffff9ad2f72098c0 spa_history.c:306:spa_history_log_sync(): txg 1237753 import pool version 5000; software version zfs-2.1.99-2799-gc71d2fb20; uts freenas 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Tue Jan 28 03:14:06 UTC 2025 x86_64
1749153571 ffff9ad2f72098c0 spa_history.c:293:spa_history_log_sync(): command: zpool import 14824704259760706208 -R /mnt -m -N -f -o cachefile=/data/zfs/zpool.cache
1749153625 ffff9ad2ea343180 metaslab.c:3711:metaslab_condense(): condensing: txg 1557587, msp[21] ffff9ad920288000, vdev id 0, spa boot-pool, smp size 525040, segments 468, forcing condense=FALSE
1749154187 ffff9ad2f72098c0 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237878, spa backup, vdev_id 0, ms_id 40, weight 880000000000001, selected txg 1237757 (604754 ms ago), alloc_txg 0, loaded 620412 ms ago, max_size 17179869184
1749154187 ffff9ad2f72098c0 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237878, spa backup, vdev_id 0, ms_id 48, weight 880000000000001, selected txg 1237757 (604754 ms ago), alloc_txg 0, loaded 620412 ms ago, max_size 17179869184
1749154187 ffff9ad2f72098c0 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237878, spa backup, vdev_id 0, ms_id 41, weight 880000000000001, selected txg 1237757 (604754 ms ago), alloc_txg 0, loaded 620412 ms ago, max_size 17179869184
1749154187 ffff9ad2f72098c0 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237878, spa backup, vdev_id 0, ms_id 42, weight 880000000000001, selected txg 1237757 (604754 ms ago), alloc_txg 0, loaded 620412 ms ago, max_size 17179869184
1749154187 ffff9ad2f72098c0 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237878, spa backup, vdev_id 0, ms_id 39, weight 840000000000001, selected txg 1237757 (604754 ms ago), alloc_txg 1237755, loaded 620723 ms ago, max_size 17179652096
1749154187 ffff9ad2f72098c0 metaslab.c:2603:metaslab_unload(): metaslab_unload: txg 1237878, spa backup, vdev_id 0, ms_id 47, weight 880000000000001, selected txg 1237757 (604754 ms ago), alloc_txg 0, loaded 620412 ms ago, max_size 17179869184
1749154393 ffff9ad2d038b180 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749154393 ffff9ad2d038b180 spa.c:6916:spa_tryimport(): spa_tryimport: importing subramanya
1749154393 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2d038b180-subramanya, config trusted): LOADING
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa $import-ffff9ad2d038b180-subramanya. txg 36292731
1749154393 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2d038b180-subramanya, config untrusted): using uberblock with txg=36292731
1749154393 ffff9ad2e9c46300 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2f54018c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749154393 ffff9ad2e9c46300 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb75000
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Calculating deflated space
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Starting import
1749154393 ffff9ad2d038b180 spa.c:9204:spa_async_request(): spa=$import-ffff9ad2d038b180-subramanya async request task=2048
1749154393 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2d038b180-subramanya, config trusted): LOADED
1749154393 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9ad2d038b180-subramanya, config trusted): UNLOADING
1749154393 ffff9ad2d038b180 metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic
1749154393 ffff9ad2d038b180 spa.c:6761:spa_import(): spa_import: importing subramanya
1749154393 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749154393 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292731
1749154393 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292731
1749154393 ffff9ad3c187ca40 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749154393 ffff9ad3c187b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=0
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749154393 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749154395 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): Read 7 log space maps (7 total blocks - blksz = 131072 bytes) in 17 ms
1749154395 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749154395 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749154395 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749154395 ffff9ad2ee296300 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 0, spa subramanya, vdev_id 0, ms_id 266, smp_length 279784, unflushed_allocs 417792, unflushed_frees 430080, freed 0, defer 0 + 0, unloaded time 884760 ms, loading_time 87 ms, ms_max_size 128653340672, max size error 128653168640, old_weight 900000000000001, new_weight 900000000000001
1749154395 ffff9ad2d038b180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749154395 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 1 metadata errors and 2 data errors
1749154395 ffff9ad2d038b180 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config trusted): FAILED: spa_load_verify failed [error=5]
1749154395 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): UNLOADING
1749154395 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_retry: rewind, max txg: 36292730
1749154395 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): LOADING
1749154395 ffff9ad3c14f98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749154395 ffff9ad3c14f98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749154395 ffff9ad3c14f98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749154395 ffff9ad3c14f98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749154395 ffff9ad3c14f98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749154395 ffff9ad3c14f98c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292730
1749154396 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292730)
1749154396 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749154396 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292730
1749154396 ffff9ad2d038b180 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749154396 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749154396 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): spa_load_retry: rewind, max txg: 36292729
1749154396 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): LOADING
1749154396 ffff9ad3c187ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad3c187ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad3c187ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad3c187ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad3c187ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad3c187ca40 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292729
1749154396 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292729)
1749154396 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749154396 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292729
1749154396 ffff9ad2d038b180 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749154396 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
1749154396 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): spa_load_retry: rewind, max txg: 36292728
1749154396 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): LOADING
1749154396 ffff9ad33c2e6300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad33c2e6300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad33c2e6300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad33c2e6300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad33c2e6300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad33c2e6300 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=1
1749154396 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': best uberblock found for spa subramanya. txg 36292728
1749154396 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': label discarded as txg is too large (36292731 > 36292728)
1749154396 ffff9ad2d038b180 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': failed to read label config
1749154396 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292728
1749154396 ffff9ad2d038b180 spa_misc.c:415:spa_load_failed(): spa_load(subramanya, config untrusted): FAILED: label config unavailable
1749154396 ffff9ad2d038b180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): UNLOADING
A place to upload the 32Mb chunks?
Thanks!
So basically the label is the entire content output by zdb -l.
The content starting with
------------------------------------
LABEL 0
------------------------------------
So itās the metadata written on dicks part of a zpool.
Makes sense, and children represent the disks.
Was there a field identifying which of the four labels it found (0 1 2 3)? If the labels all match it does not repeat itself 4 times, but it does tell you which labels it found.
Also now I make sense of your sentence !
And I suppose LABEL 0 1 2 3 indicates the type of pool, isnāt it ?
DM sent - check your private messages.
LABEL 0 1 2 3 is the redundancy - all four labels are (should be) identical copies, in case one is damaged you can import off of one of the other three.
So looking at a snippet of the debug messages:
1749153562 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config untrusted): using uberblock with txg=36292731
1749153562 ffff9ad2e7784a40 vdev_raidz.c:2694:raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/33870927-1683-4d16-a597-7efee3e7b405': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ddb020b8-2290-495d-8343-089f94d31ad4': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/69e33d70-2e29-4440-bafd-0f183720274b': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/90aea466-1527-4960-ad48-98bc5f0ffd21': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e6d7bead-7310-43d8-8236-9aba92acd3c8': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2eaf098c0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7': probe done, cant_read=0 cant_write=0
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading checkpoint txg
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading indirect vdev metadata
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Checking feature flags
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading special MOS directories
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading properties
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading AUX vdevs
1749153562 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading vdev metadata
1749153564 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): Read 7 log space maps (7 total blocks - blksz = 131072 bytes) in 21 ms
1749153564 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading dedup tables
1749153564 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Loading BRT
1749153564 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying Log Devices
1749153564 ffff9ad2ea358000 metaslab.c:2490:metaslab_load_impl(): metaslab_load: txg 0, spa subramanya, vdev_id 0, ms_id 266, smp_length 279784, unflushed_allocs 417792, unflushed_frees 430080, freed 0, defer 0 + 0, unloaded time 54285 ms, loading_time 58 ms, ms_max_size 128653340672, max size error 128653168640, old_weight 900000000000001, new_weight 900000000000001
1749153565 ffff9ad2e6f13180 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'subramanya' Verifying pool data
1749153565 ffff9ad2e6f13180 spa_misc.c:429:spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 1 metadata errors and 2 data errors
The party starts at txg=36292731 and pretty promptly hits an error with
raidz_parity_verify(): found error on col=0 devidx=3 off 5405bb6f000
and then down at the end we have
spa_load_note(): spa_load(subramanya, config trusted): spa_load_verify found 1 metadata errors and 2 data errors
With two members of the RAIDZ2 being ābehindā from their offline/removed behavior and not having that txg=36292731 present on them, they canāt be used to reconstruct and correct that metadata error. So itās erroring out.
Forcing a pool rewind back to txg=36292727 should be enough, but you could also go back to txg=36292723 to ensure full redundancy - but it is āfurther back in timeā so to speak.
What this doesnāt necessarily identify though is the reason those two drives dropped off in the first place - but heat may have been a contributing factor here as your drives have non-zero values logged for Time in Over-Temperature and the DS380 is known to have poor ventilation as you mentioned.
Community user @briancmoses has helpfully logged some information about 3D-printing some ductwork and fan changes in his own DS380 that may be helpful here:
2 Likes
Iām not sure I get that: arenāt 4 drives on a 6xRAIDZ2 pool enough to at least ārunā? then why not enough to import?
Good catch on Time in Over-Temperature I didnāt spot it.
330 -D- Time in Over-Temperature ā is 330 seconds? hours?
But itās actually on SDA, which isnāt in error. SDE and SDF were the ones at fault.
Itās actually hard to believe heat is the source of the issue, or at least not heat on the disk themselves.
Looking at the sequence of the events, itās still very hard to explain IMHO.
Am I correct into thinking that whichever disk failure should result in a FAULTED disk?
REMOVED if for disconnection as PK1048 was saying.
Here the REMOVED and FAULTED donāt really make sense.
Disks are not so close with the MB and CPU in a DS380. But maybe the SOC to which disks are attached started to heat up and produce errors.
I created cooling duct years ago, though with simple stiff cardboard, after this post from @briancmoses. And it was good with WD Red.
They never got hotter than 40C even in summer and were below 30C in winter.
But these WD Gold are running much hotter.
The plan is to move them to a Dell R730XD, and away with power consumptionā¦
ā4 drives in a 6wZ2ā is enough to run if everything else is fine. If one drive returns bad data (eg: sda) then you then have ā3 drives in a 6wZ2ā which isnāt enough - and because drives sde and sdf donāt have any knowledge of txg=36292731 they canāt provide the redundancy blocks for that transaction group. And it looks like we still see that error even going back to earlier groups.
Iām trying to dig that up from WDās spec sheets, but itās not being forthcoming. I certainly hope itās seconds but I have my suspicions that it isnāt, given that most of the other time-durations (eg: power-on time, SMART timestamps) are recorded in running hours ā¦ 
You can filter it down with the smartctl command on the other drives and grepping the output for Over-Temperature to see what the values are.
High temperatures do bad things to drives. They do worse things to drive controllers since those tend to just fall off the PCIe bus and not recover, but definitely is not good for the sustained operation of your drive especially if the over-temp time is in hours as Iām fearing.
If the disk isnāt permanently faulted but rather stalled or throttled itself to avoid persistent over-temp conditions - that might manifest to ZFS as the drive being non-responsive on the SATA bus, at which point it was flagged as REMOVED - if it cooled off and recovered, it likely resilvered, tried to catch up - but if it fell too far behind, repeated the error, or stayed offline too long, it might catch the FAULTED state instead.
2 Likes
Such airflow guides are indispensible, and should ship with the caseā¦ but are still not sufficient to ensure good cooling, even with a pair of high-pressure fans on the side.
WD Gold are rebranded HGST Ultrastar. Maybe HGST has better documentation?
I think these are two different lines of products. Theyāre both for sale on the website. Anyway, I couldnāt find anything.
ChatGPT says heās sure itās HOURS !
maybe, but then itās SDA only reporting non zero Time in Over-Temperature. It seats at the top of the tower. Will def. put it down where ventilation is better until I finally move to another box.
HoneyBadger proposed to import with rewinding to txg=36292723.
using zpool import -nfFXT 36292723 subramanya.
Iām doing the dryrun with -n and will apply today if all goes well.
As an aside: does anyone think itās a bad idea to power down a TrueNAS instance every day for the night?
Iām confident they are the very same drives with two different stickers. WD Gold even have the 3.3V power disable behaviour of HGST drives, contrary to all other colours of WD.
I would not, however, trust ChatGPT or any other AI. LLMs are basically programmed to output something statistically plausible with only a more or less pronounced bias towards what the users wants to hear. āTruthā is not on the menu, even IF the training set was thoroughly wetted (cough, cough!).
Powering down once a day should not be an issue. But inactive time is also the time slot which could be used for maintenance, SMART tests and scrubs.
3 Likes
I lost my session running zpool import -nfFXT 36292723 subramanya as it takes very long and didnāt think of using tmux.
Incidentally, I ran zpool import -nfFXT 36292723 and the result is very different than with the previoulsy ran zpool import:
pool: FAST
id: 15095281718178502118
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
FAST ONLINE
nvme0n1p2 ONLINE
pool: subramanya
id: 15005074635607672362
state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:
subramanya DEGRADED
raidz2-0 DEGRADED
69e33d70-2e29-4440-bafd-0f183720274b UNAVAIL
90aea466-1527-4960-ad48-98bc5f0ffd21 ONLINE
ddb020b8-2290-495d-8343-089f94d31ad4 ONLINE
4e570c79-6ff8-4ab4-94e3-0fc804c1a3a7 ONLINE
e6d7bead-7310-43d8-8236-9aba92acd3c8 ONLINE
33870927-1683-4d16-a597-7efee3e7b405 ONLINE
Is this because of the command parameters that does a deeper check on the status?
Or is it because of the previous command zpool import -nfFXT 36292723 subramanya I ran? Is it building up information even in dryrun mode?
The UNAVAIL status on a disk is worrying me.
Now in the middle of another zpool import -nfFXT 36292723 subramanya I canāt get any information from the UI or even zpool import. The command seem to block access to disks
That line identifies at which of the four possible locations the ZFS label was found. In this case, at all four locations. I have had failures which included the line
labels = 2 3
Where the labels at the start of the component were missing.
You only need one good label to import the data from that specific component. More ZFS redundancy.
By the way, I am using the term component instead of disk or device since it can be almost any valid block device.
1 Like