I was doing all the disk swapping with poweroff so no hot-swapping.
Some progress.
I disabled ix-zfs.service
and rebooted - I wanted to avoid the zpool import
command running. I think think that is what was hanging the subsequent zpool
commands.
Without ix-zfs
I was able to boot quickly and get to a shell and issue zpool
commands. I was also able to recreate the hanging situation by manually running the zpool import pool-01 -R /mnt -m -f -o cachefile=/data/zfs/zpool.cache
which is the command that appears to be started via ix-zfs.service
. It spits out WARNING: Pool 'pool-01' has encountered an unrecoverable I/O failure and has been suspended
and then never returning.
Next I tried
echo 1 > /sys/module/zfs/parameters/zfs_recover
zpool import pool-01 -R /mnt -o readonly=on
And that seems succesfully imports and mount the pool and datasets so now I can at least see some of the data. That’s as far as I have gotten, but it feels a bit like progress.
Thoughts on next steps? I haven’t been able to import
in RW mode and I still haven’t managed to find an actual hardware failure that I can attribute to this.