Pool is suspended and zpool commands hang

I was doing all the disk swapping with poweroff so no hot-swapping. :-1:

Some progress.

I disabled ix-zfs.service and rebooted - I wanted to avoid the zpool import command running. I think think that is what was hanging the subsequent zpool commands.

Without ix-zfs I was able to boot quickly and get to a shell and issue zpool commands. I was also able to recreate the hanging situation by manually running the zpool import pool-01 -R /mnt -m -f -o cachefile=/data/zfs/zpool.cache which is the command that appears to be started via ix-zfs.service. It spits out WARNING: Pool 'pool-01' has encountered an unrecoverable I/O failure and has been suspended and then never returning.

Next I tried

echo 1 > /sys/module/zfs/parameters/zfs_recover
zpool import pool-01 -R /mnt -o readonly=on

And that seems succesfully imports and mount the pool and datasets so now I can at least see some of the data. That’s as far as I have gotten, but it feels a bit like progress.

Thoughts on next steps? I haven’t been able to import in RW mode and I still haven’t managed to find an actual hardware failure that I can attribute to this.