Gone for the weekend, returned to pool missing, possible drive failure, no idea what to do

You funny man! That was a luxury item. And to be honest, while they may have existed, they certainly were not in my home. Tape, Glue, Bailing Wire, and String. Pick your poison. And fiberglass was also a luxury item however we actually had that around.

I just finished the cooling fan shroud 3d print. I’ll put it on another day after I know stuff is okay again.

So, I’m not touching anything but it doesn’t appear that I have the pools imported yet. Assuming that there is still more to do?

Also, I would love thoughts on backing up. Should I have a second backup truenas server? What do you all do for backups?

Hold for @HoneyBadger or @Protopia to advise.

Otherwise backup is generally a 2nd system, preferably in a different location if at all possible. Or at the very least dump the photos onto an external HDD; won’t protect against bitrot, but fairly cost effective & better than nothing.

1 Like

So - in the first instance you ran zpool import -nF -R /mnt Valkyrie and it seemed to suggest that it should import OK, and then you ran zpool import -F -R /mnt Valkyrie and it appeared actually to have imported.

Again when you ran zpool import -nF -R /mnt Valkyrie it seemed to suggest that it should import, and then you ran zpool import -F -R /mnt Valkyrie and it failed to import and said one or more devices were unavailable.

If you can put your hardware back to the state it was in in the first attempt, maybe it will still import.

If you have already done this and it doesn’t now import, then the situation appears to have got significantly worse.

It is my belief that with changing hardware and reboots we need to re-establish whether the pool can be imported or not with “-F” and if it can then we need to import it and see what the situation is.

Yes. Oddly enough your dbgmsg log there does not seem to show actual errors after the import attempt at using uberblock with txg=3083521 which is where I expect it’s going in.

I would suggest that you reboot your system, which will ensure the pool isn’t mounted - check the webUI and zpool status -v from the command line to be sure - and then run:

zpool import -fF -R /mnt Valkyrie -o readonly=on

Grab the output of the last 200 lines of dbgmsg as before.

If the import returns without errors, then your data should be visible - but be read-only - on the /mnt/Valkyrie path. If that’s the case, check that any recently added files seem to be OK. Exporting the pool (zpool export Valkyrie) and then re-running the command without -o readonly=on will commit those changes.

If the import returns errors, we have more work to do.

root@bigtruenas[~]# zpool status -v        
  pool: boot-pool
 state: ONLINE
status: One or more features are enabled on the pool despite not being
        requested by the 'compatibility' property.
action: Consider setting 'compatibility' to an appropriate value, or
        adding needed features to the relevant file in
        /etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d.
  scan: scrub repaired 0B in 00:00:19 with 0 errors on Wed Jun 25 03:45:20 2025
config:

        NAME         STATE     READ WRITE CKSUM
        boot-pool    ONLINE       0     0     0
          nvme0n1p2  ONLINE       0     0     0

errors: No known data errors
root@bigtruenas[~]# 

root@bigtruenas[~]# zpool import -fF -R /mnt Valkyrie -o readonly=on
cannot import 'Valkyrie': one or more devices is currently unavailable
root@bigtruenas[~]#                                                 

root@bigtruenas[~]# tail -n 200 /proc/spl/kstat/zfs/dbgmsg          
1751040183   ffff8bc511550000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Verifying Log Devices
1751040183   ffff8bc511550000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Verifying pool data
1751040183   ffff8bc54056a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b5c30075000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b60e8536000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f162000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b61565ca000
1751040183   ffff8bc5420a8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b659f1b0000
1751040183   ffff8bc54056a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b659f1b3000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f1b4000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b659f1a7000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f1b1000
1751040183   ffff8bc54056a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b659f1ad000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b659f18c000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f1a2000
1751040183   ffff8bc522622040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f1ae000
1751040183   ffff8bc522620000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b659f1aa000
1751040183   ffff8bc54056a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f1a8000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b659f1a1000
1751040183   ffff8bc540fdc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f1ab000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f1a5000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b659f19e000
1751040183   ffff8bc522620000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b659f1a4000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f19f000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f19c000
1751040183   ffff8bc522620000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b659f19b000
1751040183   ffff8bc5420a8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b621dfe8000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5c3007f000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5c30076000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5c3007e000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5c3007b000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5c30079000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5c3007c000
1751040183   ffff8bc522622040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5c30078000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60e8539000
1751040183   ffff8bc540fdc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60e853c000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60e853b000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60e8530000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60e8532000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60e8538000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60e8536000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60e8533000
1751040183   ffff8bc522622040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60e852f000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b6476f10000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b6476f0d000
1751040183   ffff8bc522620000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b65345cc000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b60e853b000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b60e8538000
1751040183   ffff8bc522622040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b60e8539000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b60e8535000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b60e8532000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b60e852f000
1751040183   ffff8bc522620000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b60e8533000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b60e8530000
1751040183   ffff8bc540fdc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b60e852d000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b60e8536000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b621e007000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b621e004000
1751040183   ffff8bc54056a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f1b1000
1751040183   ffff8bc54056a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f1aa000
1751040183   ffff8bc54056a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f1a1000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f1b3000
1751040183   ffff8bc540fdc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f1a2000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f1b0000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f1ae000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f1ad000
1751040183   ffff8bc540fdc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f190000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f1a7000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f1a8000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f1a5000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f19c000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f19b000
1751040183   ffff8bc522622040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f19f000
1751040183   ffff8bc522622040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f196000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f18d000
1751040183   ffff8bc5420a8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f18c000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b659f18a000
1751040183   ffff8bc54056a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b621e009000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b621e007000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b621e006000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b5e94478000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b5e94477000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b5e94475000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b5e94471000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b5e9446f000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b5e94472000
1751040183   ffff8bc522620000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b5e94474000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b6476f16000
1751040183   ffff8bc5420a8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b6476f15000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b6476f13000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b6476f0c000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b6476f0a000
1751040183   ffff8bc522622040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b6476f09000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b6476f00000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b6200019000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b621e006000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b63a56d5000
1751040183   ffff8bc5420a8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b5c3007e000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b5c3007b000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b5c30079000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b5c3007c000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b5c30078000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b5c30076000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b63a56d2000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5e94477000
1751040183   ffff8bc522620000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5e94471000
1751040183   ffff8bc5420aa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e94472000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e94475000
1751040183   ffff8bc540fdc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e9446f000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5e94474000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e94478000
1751040183   ffff8bc5420a8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b621dfe8000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60e8535000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b6476f0f000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b65345ce000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b65345cb000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f18f000
1751040183   ffff8bc540fdc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f1a4000
1751040183   ffff8bc5420a8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b659f19e000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6476f16000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6476f15000
1751040183   ffff8bc522620000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6476f13000
1751040183   ffff8bc522622040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6476f10000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6476f0d000
1751040183   ffff8bc54056a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6476f0c000
1751040183   ffff8bc522620000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6476f09000
1751040183   ffff8bc540fdc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6476f0a000
1751040183   ffff8bc522622040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6476f0f000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f15a000
1751040183   ffff8bc5420a8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f190000
1751040183   ffff8bc5a29a0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659f18d000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6533381000
1751040183   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b65345cf000
1751040183   ffff8bc540fdc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b65345cc000
1751040183   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b65345ce000
1751040183   ffff8bc540fdc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b65345cb000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b5e9442d000
1751040183   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b5c30066000
1751040183   ffff8bc540568000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60e8482000
1751040183   ffff8bc522620000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b6534542000
1751040183   ffff8bc540fd8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b63a564b000
1751040184   ffff8bc523c5c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=9 off b60e8481000
1751040184   ffff8bc5420b0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b61346dd000
1751040184   ffff8bc540fda040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=9 off b61565c5000
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): spa_load_verify found 1 metadata errors and 41 data errors
1751040184   ffff8bc511550000 spa_misc.c:415:spa_load_failed(): spa_load(Valkyrie, config trusted): FAILED: spa_load_verify failed [error=5]
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): UNLOADING
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): spa_load_retry: rewind, max txg: 3083520
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): LOADING
1751040184   ffff8bc51f600000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f600000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdi2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f600000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sde2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f600000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f600000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f600000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f600000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdk2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f600000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdf2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f600000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdg2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f600000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdc2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc511550000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': best uberblock found for spa Valkyrie. txg 3083520
1751040184   ffff8bc511550000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': label discarded as txg is too large (3083521 > 3083520)
1751040184   ffff8bc511550000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': failed to read label config
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): using uberblock with txg=3083520
1751040184   ffff8bc511550000 spa_misc.c:415:spa_load_failed(): spa_load(Valkyrie, config untrusted): FAILED: label config unavailable
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): UNLOADING
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): spa_load_retry: rewind, max txg: 3083519
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): LOADING
1751040184   ffff8bc51f66a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sde2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f66a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f66a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f66a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f66a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdi2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f66a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdg2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f66a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdk2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f66a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdc2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f66a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdf2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc51f66a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc511550000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': best uberblock found for spa Valkyrie. txg 3083519
1751040184   ffff8bc511550000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': label discarded as txg is too large (3083521 > 3083519)
1751040184   ffff8bc511550000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': failed to read label config
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): using uberblock with txg=3083519
1751040184   ffff8bc511550000 spa_misc.c:415:spa_load_failed(): spa_load(Valkyrie, config untrusted): FAILED: label config unavailable
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): UNLOADING
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): spa_load_retry: rewind, max txg: 3083518
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): LOADING
1751040184   ffff8bc5189dc080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdi2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc5189dc080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc5189dc080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdg2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc5189dc080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc5189dc080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdk2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc5189dc080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc5189dc080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdc2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc5189dc080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sde2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc5189dc080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdf2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc5189dc080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1751040184   ffff8bc511550000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': best uberblock found for spa Valkyrie. txg 3083518
1751040184   ffff8bc511550000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': label discarded as txg is too large (3083521 > 3083518)
1751040184   ffff8bc511550000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdh2': failed to read label config
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): using uberblock with txg=3083518
1751040184   ffff8bc511550000 spa_misc.c:415:spa_load_failed(): spa_load(Valkyrie, config untrusted): FAILED: label config unavailable
1751040184   ffff8bc511550000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): UNLOADING

Okay, so it is starting at 3083521 as expected and attempts a light rewind.

We can try again with zpool import -fFX -R /mnt Valkyrie -o readonly=on to tell it to take a more aggressive rollback posture, or jump straight to 3083514 if that drive is connected with zpool import -fFXT 3083514 -R /mnt Valkyrie -o readonly=on

So I’m not sure which to do. Is one better than the other, or should I just try the first suggestion and then the second? I’m feeling a little skittish.

They are both leveraging the same level of aggressiveness; -T just tells it to not bother starting any higher up, and to begin with a group no higher than 3083514.

I assume you have the “failing” drive still connected?

Once this happens, try to pull another set of dbgmsg logs for me please.

Thank you for explaining. Very much appreciated. Yes, the failing drive is still connected. So I ran that and I hear the hard drives accessing a bit and then silence. and this is still what I’m seeing:

I walked away and came back to the login screen so I don’t know what happened.

Here’s the dbgmsg log.

Welcome to TrueNAS
Last login: Fri Jun 27 17:47:54 MDT 2025 on pts/1
root@bigtruenas[~]# tail -n 200 /proc/spl/kstat/zfs/dbgmsg                    
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f3ab000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f3c3000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f3a8000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f3a5000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f3a2000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f39c000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f3b4000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f3b1000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f396000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f387000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f393000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f38a000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f39f000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f378000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f399000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f384000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f375000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f390000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f381000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f37e000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f5f37b000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d74ce000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d74b9000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b60d74df000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d7523000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d751c000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d751f000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d7517000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d751d000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d7522000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d7514000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d7513000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d750d000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d7519000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d7516000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d7510000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d7511000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d7505000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d7501000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d7508000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d7502000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d750e000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d74ff000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d74fb000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d74fe000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d74f6000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d74f8000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d74f3000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d74f2000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60d74fc000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60d74f5000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d754c000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7543000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7546000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d753a000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7534000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7531000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7537000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7597000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7594000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d764b000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7642000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7645000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d763f000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7633000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7639000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d762d000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7636000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7630000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d761b000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d764e000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7618000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7624000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75fa000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7615000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7621000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d761e000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d760c000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75f1000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75ee000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75eb000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7606000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d760f000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7603000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75fd000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75f4000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75d3000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d7600000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75e5000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75e2000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75d0000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75ca000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75dc000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75d6000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75c4000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75cd000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75c7000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75c1000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75b8000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75be000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75af000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75ac000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75bb000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b60d75b5000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b61341de000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b61341ea000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b61341e7000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614db44000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b614db46000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614db3b000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614db41000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b614db40000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614db38000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614db35000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b614db43000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614db3e000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b614db3d000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b614db3a000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b614db22000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614db20000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b614db34000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b614db37000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614db2f000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b614db2e000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614db23000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b614db1f000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbc7000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbc4000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbc1000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbbb000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbb8000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbbe000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dba9000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbb2000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbb5000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db9d000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db9a000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbac000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbca000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dba3000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dbaf000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db8e000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db8b000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dba6000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db7f000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614dba0000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db97000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db7c000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db94000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db91000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db82000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db79000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db88000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db85000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db6d000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db70000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db6a000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db76000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db52000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db64000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db73000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db5b000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db55000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b614db5e000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6215e88000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b6215e9a000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b6215e97000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b639d0d4000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b639d0d1000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b64667c7000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b64667c3000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b6466826000
1751066820   ffff8bc5d583a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b6466829000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b646682c000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b6466823000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b6466820000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b6466817000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b646681d000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b646681a000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b646680e000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b6466811000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12b6000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12b0000
1751066820   ffff8bc5445d8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12ad000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12aa000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab128f000
1751066820   ffff8bc510f1a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab129b000
1751066820   ffff8bc5cada4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab1295000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab1292000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab1289000
1751066820   ffff8bc5a059a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab1286000
1751066820   ffff8bc51b1cc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12da000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12d4000
1751066820   ffff8bc5d583c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12d1000
1751066820   ffff8bc51cdf4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12ce000
1751066820   ffff8bc5182c0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12c8000
1751066820   ffff8bc51b1c8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12c5000
1751066820   ffff8bc5d5838000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12cb000
1751066820   ffff8bc5182c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off 64ab12c2000
1751066831   ffff8bc51b1cc080 spa.c:9245:spa_async_request(): spa=Valkyrie async request task=2
root@bigtruenas[~]#

Also when I look at the Dashboard, nothing loads, just looks like it’s busy and not loading anything.

@HoneyBadger

I went away from the server for a while. When I came back all I saw was this screen.

I took a look at the actual connected monitor screen and this is what I see

Not sure what to do now. Do I just wait or is something not working?

The console looks like the system rebooted at some point during the import, and it’s now hung up and blocking progress because of a disk timeout. We may need to disconnect the “failing” drive if it’s actually so far gone that we’re throwing ATA errors because of it.

Your dbgmsg is also full of data errors, which is a challenge because if both the failed/failing drives are pulled, there’s no parity to rebuild from. Do you have a third new drive you were planning to replace either of these two with?

I have two new drives that I was planning to use and another spare on the way. If it rebooted, then it was when I was away and it did it on its own. I didn’t restart it. Those two new drives are connected to the system but I haven’t done anything else with them.

Should I shutdown, disconnect the failed drive and restart? Or what’s the best next step?

Are you doing these import attempts using the TrueNAS shell? It kind of looks like it from the screenshots you posted.

If so, you need to use tmux since the import will take longer than the GUI session will last. Using ssh instead of the GUI shell is also an option.

Yes, I didn’t know not to do that. It never occurred to me to not use that. Using SSH now instead.

I shut down the machine, removed the failing drive, and restarted. I ran zpool status -v

mike@bigtruenas:~$ sudo zpool status -v
[sudo] password for mike:
  pool: boot-pool
 state: ONLINE
status: One or more features are enabled on the pool despite not being
        requested by the 'compatibility' property.
action: Consider setting 'compatibility' to an appropriate value, or
        adding needed features to the relevant file in
        /etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d.
  scan: scrub repaired 0B in 00:00:19 with 0 errors on Wed Jun 25 03:45:20 2025
config:

        NAME         STATE     READ WRITE CKSUM
        boot-pool    ONLINE       0     0     0
          nvme0n1p2  ONLINE       0     0     0

errors: No known data errors

Then I ran: zpool import -fFX -R /mnt Valkyrie -o readonly=on and I hear the drives doing things and no command prompt yet, so it’s doing something it seems.

1 Like

I did run in a TMUX and that worked or at least it didn’t just hang like the last time. This is what I got finally.

mike@bigtruenas:~$ sudo zpool import -fFX -R /mnt Valkyrie -o readonly=on
cannot import 'Valkyrie': one or more devices is currently unavailable
mike@bigtruenas:~$ sudo zpool status -v
[sudo] password for mike:
  pool: boot-pool
 state: ONLINE
status: One or more features are enabled on the pool despite not being
        requested by the 'compatibility' property.
action: Consider setting 'compatibility' to an appropriate value, or
        adding needed features to the relevant file in
        /etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d.
  scan: scrub repaired 0B in 00:00:19 with 0 errors on Wed Jun 25 03:45:20 2025
config:

        NAME         STATE     READ WRITE CKSUM
        boot-pool    ONLINE       0     0     0
          nvme0n1p2  ONLINE       0     0     0

errors: No known data errors
mike@bigtruenas:~$


mike@bigtruenas:~$ sudo tail -n 200 /proc/spl/kstat/zfs/dbgmsg
[sudo] password for mike:
1751136526   ffff96c5dd2e0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b61345cf000
1751136526   ffff96c631b30000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5c2e639000
1751136526   ffff96c5e339a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5c2e638000
1751136526   ffff96c5e1f28000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5e928ed000
1751136526   ffff96c631b30000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e92900000
1751136527   ffff96c5e339a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b6133f9a000
1751136527   ffff96c5dd2e0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6154870000
1751136527   ffff96c5dd2e8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b621c5a5000
1751136528   ffff96c60307c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5f65d5a000
1751136529   ffff96c631b30000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b6147a20000
1751136529   ffff96c5dd2e8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b63a3a79000
1751136530   ffff96c5e0aa4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5f65d5f000
1751136530   ffff96c5e1f2a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b646c882000
1751136530   ffff96c5e0aa4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b615486d000
1751136531   ffff96c5e0aa0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e8e6b2000
1751136531   ffff96c5e0aa4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60e4734000
1751136531   ffff96c5e1f28000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60e4744000
1751136531   ffff96c5dd2e0000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60e4743000
1751136531   ffff96c658a98000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e92903000
1751136531   ffff96c658a98000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5e92902000
1751136531   ffff96c5e339a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614fdd6000
1751136531   ffff96c631b30000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5c2e63b000
1751136531   ffff96c5e1f2a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5f421d1000
1751136532   ffff96c60307c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5c2e63c000
1751136532   ffff96c610d64080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b63a3a7a000
1751136533   ffff96c5e1f28000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b615486c000
1751136534   ffff96c610d64080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6154869000
1751136534   ffff96c6008d4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5e92905000
1751136535   ffff96c658a98000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5c2e63e000
1751136536   ffff96c60307c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6154872000
1751136536   ffff96c60307c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6154867000
1751136536   ffff96c5e1f28000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b63a3a77000
1751136536   ffff96c631b30000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e92906000
1751136541   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): spa_load_verify found 4 metadata errors and 1033 data errors
1751136541   ffff96c5dc180000 spa_misc.c:415:spa_load_failed(): spa_load(Valkyrie, config trusted): FAILED: spa_load_verify failed [error=5]
1751136541   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): UNLOADING
1751136541   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): spa_load_retry: rewind, max txg: 3083484
1751136541   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): LOADING
1751136541   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1751136541   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdf2': probe done, cant_read=0 cant_write=1
1751136541   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sde2': probe done, cant_read=0 cant_write=1
1751136541   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdg2': probe done, cant_read=0 cant_write=1
1751136541   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1751136541   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1751136541   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1751136541   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1751136541   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdi2': probe done, cant_read=0 cant_write=1
1751136542   ffff96c6946fa040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdc2': open error=2 timeout=1008552130/1000000000
1751136542   ffff96c6329fa040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': vdev_validate: config guid doesn't match label guid
1751136542   ffff96c6329fa040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': CONFIG: guid 7101089285928253376, top_guid 18067475686433984793
1751136542   ffff96c6329fa040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': LABEL: guid 3006882281538507282, top_guid 18067475686433984793, aux_guid 0
1751136542   ffff96c5dc180000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdg2': best uberblock found for spa Valkyrie. txg 3083484
1751136542   ffff96c5dc180000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdg2': label discarded as txg is too large (3083514 > 3083484)
1751136542   ffff96c5dc180000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdg2': failed to read label config. Trying again without txg restrictions.
1751136542   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): using uberblock with txg=3083484
1751136542   ffff96c5dc180000 vdev.c:2534:vdev_update_path(): vdev_copy_path: vdev 16359988943227625139: vdev_path changed from '/dev/sda2' to '/dev/sdi2'
1751136542   ffff96c5dc180000 vdev.c:2534:vdev_update_path(): vdev_copy_path: vdev 10225035552399033554: vdev_path changed from '/dev/sdd2' to '/dev/sdj2'
1751136542   ffff96c5dc180000 vdev.c:2534:vdev_update_path(): vdev_copy_path: vdev 1735524699209308495: vdev_path changed from '/dev/sde2' to '/dev/sdg2'
1751136542   ffff96c5dc180000 vdev.c:2534:vdev_update_path(): vdev_copy_path: vdev 15271541350893828723: vdev_path changed from '/dev/sdc2' to '/dev/sdf2'
1751136542   ffff96c5dc180000 vdev.c:2534:vdev_update_path(): vdev_copy_path: vdev 17617450205283552828: vdev_path changed from '/dev/sdg2' to '/dev/sdd2'
1751136542   ffff96c5dc180000 vdev.c:2534:vdev_update_path(): vdev_copy_path: vdev 3006882281538507282: vdev_path changed from '/dev/sdh2' to '/dev/sdb2'
1751136542   ffff96c5dc180000 vdev.c:2534:vdev_update_path(): vdev_copy_path: vdev 17119780021793886840: vdev_path changed from '/dev/sdj2' to '/dev/sda2'
1751136542   ffff96c5dc180000 vdev.c:2534:vdev_update_path(): vdev_copy_path: vdev 2296139040122006835: vdev_path changed from '/dev/sdf2' to '/dev/sde2'
1751136542   ffff96c5dc180000 vdev.c:2534:vdev_update_path(): vdev_copy_path: vdev 6524818153576139299: vdev_path changed from '/dev/sdi2' to '/dev/sdc2'
1751136542   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1751136542   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdi2': probe done, cant_read=0 cant_write=1
1751136542   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1751136542   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1751136542   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1751136542   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdf2': probe done, cant_read=0 cant_write=1
1751136542   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1751136542   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdg2': probe done, cant_read=0 cant_write=1
1751136542   ffff96c684a3c080 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sde2': probe done, cant_read=0 cant_write=1
1751136543   ffff96c67982a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdc2': open error=2 timeout=1001877144/1000000000
1751136543   ffff96c6942c0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': vdev_validate: config guid doesn't match label guid
1751136543   ffff96c6942c0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': CONFIG: guid 7101089285928253376, top_guid 18067475686433984793
1751136543   ffff96c6942c0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': LABEL: guid 3006882281538507282, top_guid 18067475686433984793, aux_guid 0
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Loading checkpoint txg
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Loading indirect vdev metadata
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Checking feature flags
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Loading special MOS directories
1751136543   ffff96c5dc180000 dsl_scan.c:609:dsl_scan_init(): resilver can't excise DTL_MISSING when finished; restarting on Valkyrie in txg 3083521
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Loading properties
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Loading AUX vdevs
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Loading vdev metadata
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Loading dedup tables
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Loading BRT
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Verifying Log Devices
1751136543   ffff96c5dc180000 spa_misc.c:2376:spa_import_progress_set_notes_impl(): 'Valkyrie' Verifying pool data
1751136544   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
  (spa_load_verify_data=1, spa_load_verify_metadata=1)
1751136544   ffff96c679d14080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6134506000
1751136544   ffff96c658e64080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6152e0c000
1751136545   ffff96c679d14080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b621ded4000
1751136545   ffff96c658e60000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b64cd021000
1751136545   ffff96c632a32040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b6534414000
1751136545   ffff96c658e60000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b659ece0000
1751136545   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b659ec87000
1751136547   ffff96c687b38000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5e942a3000
1751136550   ffff96c679d14080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b659cb7f000
1751136561   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5c2d370000
1751136561   ffff96c632a32040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5f65880000
1751136561   ffff96c6129d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b64e346c000
1751136563   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60e2d5e000
1751136563   ffff96c6787c4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60e236a000
1751136563   ffff96c658e60000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b64e4cb1000
1751136563   ffff96c600daa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6153006000
1751136566   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b621ba2d000
1751136566   ffff96c6129d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b621be8f000
1751136566   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b647185e000
1751136566   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b64e46de000
1751136566   ffff96c5e1f2a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b64e7d77000
1751136566   ffff96c600daa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b64e9973000
1751136567   ffff96c5e1f2a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=2 off b5e68f5a000
1751136567   ffff96c600daa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b608f0ac000
1751136567   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b621c3fe000
1751136567   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b621c563000
1751136567   ffff96c600daa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b64e2ff4000
1751136567   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b64e547e000
1751136567   ffff96c600daa040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b64e589b000
1751136567   ffff96c658e60000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b64e6d23000
1751136569   ffff96c5e1f2a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b608f0ad000
1751136571   ffff96c5d0ff4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b2d65e34000
1751136586   ffff96c6129d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b621c32c000
1751136602   ffff96c65a08c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b6534180000
1751136616   ffff96c5d0fcc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6152e11000
1751136616   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b63a20b2000
1751136616   ffff96c65a08c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b63a20b1000
1751136616   ffff96c67a81a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b64e2539000
1751136616   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b64e2538000
1751136616   ffff96c5e34d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b61346c9000
1751136617   ffff96c65a08c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b64e2694000
1751136618   ffff96c67a81a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b60e732b000
1751136618   ffff96c5e34d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b63a54a1000
1751136618   ffff96c5e34d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5f421bf000
1751136618   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5f65d0e000
1751136618   ffff96c5e34d4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5f65d5d000
1751136618   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b5f65d95000
1751136618   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60e4735000
1751136618   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6154864000
1751136618   ffff96c5f0bb8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b615486a000
1751136618   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b621dee9000
1751136618   ffff96c5e34d4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b653442a000
1751136618   ffff96c67a81a040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b615486f000
1751136619   ffff96c6129d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b64732e9000
1751136619   ffff96c5d0fcc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5c2e639000
1751136619   ffff96c65a08c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5c2e638000
1751136620   ffff96c6129d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b63a3a76000
1751136620   ffff96c5e34d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5e928ed000
1751136620   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e92900000
1751136620   ffff96c5e34d4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b61345cf000
1751136620   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b61345d6000
1751136621   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6154870000
1751136621   ffff96c5d0fcc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b621c5a5000
1751136622   ffff96c6129d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b6133f9a000
1751136622   ffff96c5e34d4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=4 off b6147a20000
1751136622   ffff96c5d0fcc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b63a3a79000
1751136622   ffff96c5e34d4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5f65d5a000
1751136623   ffff96c65a08c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5f65d5f000
1751136623   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=4 off b646c882000
1751136623   ffff96c5d0fcc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e92903000
1751136624   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b60e4744000
1751136624   ffff96c600aea040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5c2e63b000
1751136624   ffff96c6129d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5e92902000
1751136624   ffff96c5e34d4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5f421d1000
1751136624   ffff96c5d0fcc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e8e6b2000
1751136624   ffff96c65a08c080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60e4734000
1751136624   ffff96c5f0bb8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b60e4743000
1751136624   ffff96c5e34d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b614fdd6000
1751136624   ffff96c5d0fcc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b615486d000
1751136625   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b615486c000
1751136625   ffff96c600aec080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b63a3a7a000
1751136625   ffff96c5f0bb8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5c2e63c000
1751136626   ffff96c5f0bb8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6154869000
1751136627   ffff96c5e34d4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5e92905000
1751136628   ffff96c5d0fcc080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b5c2e63e000
1751136628   ffff96c5e34d4080 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=0 devidx=3 off b6154872000
1751136628   ffff96c5e34d2040 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b6154867000
1751136629   ffff96c67a818000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b63a3a77000
1751136629   ffff96c5f0bb8000 vdev_raidz.c:2722:raidz_parity_verify(): found error on col=1 devidx=3 off b5e92906000
1751136633   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): spa_load_verify found 4 metadata errors and 1033 data errors
1751136633   ffff96c5dc180000 spa_misc.c:415:spa_load_failed(): spa_load(Valkyrie, config trusted): FAILED: spa_load_verify failed [error=5]
1751136633   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): UNLOADING
1751136633   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): spa_load_retry: rewind, max txg: 3083483
1751136633   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config trusted): LOADING
1751136633   ffff96c600b48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdj2': probe done, cant_read=0 cant_write=1
1751136633   ffff96c600b48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdd2': probe done, cant_read=0 cant_write=1
1751136633   ffff96c600b48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdg2': probe done, cant_read=0 cant_write=1
1751136633   ffff96c600b48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sda2': probe done, cant_read=0 cant_write=1
1751136633   ffff96c600b48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdf2': probe done, cant_read=0 cant_write=1
1751136633   ffff96c600b48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sde2': probe done, cant_read=0 cant_write=1
1751136633   ffff96c600b48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1751136633   ffff96c600b48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': probe done, cant_read=0 cant_write=1
1751136633   ffff96c600b48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdi2': probe done, cant_read=0 cant_write=1
1751136634   ffff96c631e7a040 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdc2': open error=2 timeout=1010598827/1000000000
1751136634   ffff96c5d1e48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': vdev_validate: config guid doesn't match label guid
1751136634   ffff96c5d1e48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': CONFIG: guid 7101089285928253376, top_guid 18067475686433984793
1751136634   ffff96c5d1e48000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/sdb2': LABEL: guid 3006882281538507282, top_guid 18067475686433984793, aux_guid 0
1751136634   ffff96c5dc180000 spa_misc.c:415:spa_load_failed(): spa_load(Valkyrie, config untrusted): FAILED: no valid uberblock found
1751136634   ffff96c5dc180000 spa_misc.c:429:spa_load_note(): spa_load(Valkyrie, config untrusted): UNLOADING