This next part is understandable. TrueNAS uses an alternate mount point under /mnt, which I forgot to give you the option. So this is generally ignorable.
Now comes the tricky part. Because I did not want to force roll back which causes recent data written to be lost, I had you use the read only option. So you can’t modify your pool, like removing the missing Cache drive. But, that should be okay.
You now have some choices:
Manually mount the datasets under /mnt/FNMedia/X, replacing X with the dataset(s) name(s). Then copy the data off somewhere else.
Export the pool and re-import without the read only option. This will likely take 5 days again, sorry. Since you are rolling back transactions, (the “-T” option), recent writes will be permanently lost.
Option 2 would use something like this:
zpool import -fm -T 16951677 -R /mnt FNMedia
Note that the GUI / Middleware CLI won’t know about your manual pool import. Which if it is fully successful if you choose option 2, then you can manually export it from Unix Shell. This will allow you to import via the GUI / Middleware CLI.
So I’m back, I’ve tried to import the pool without readonly 3 times, each time the system consistently restarts after 4.2 days. I tried looking at syslogs and dmsg but nothing useful popped up as far as i could see.
I’ve fallen back to importing as readonly, which worked, however, I can’t seem to access the pool any way I try. I tried the following:
zfs mount FNMedia
cannot mount '/FNMedia': failed to create mountpoint: Read-only file system
Per AI Recommendations
root@truenas[~]# zfs mount -o mountpoint=/tmp/recovery/FNMedia FNMedia
cannot mount '/FNMedia': failed to create mountpoint: Read-only file system
root@truenas[~]# zfs mount -o mountpoint=/tmp/recovery/FNMedia/Documents FNMedia/Documents
cannot mount '/FNMedia/Documents': failed to create mountpoint: Read-only file system
root@truenas[~]# zfs set mountpoint=/tmp/recovery/FNMedia FNMedia
cannot set property for 'FNMedia': pool is read-only
New drives are on the way, but need to get access to the pool before any transfers can happen…
root@truenas[~]# mount -t zfs FNMedia /mnt/FNMedia
filesystem 'FNMedia' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount FNMedia'.
See zfs(8) for more information.
Trying to set mountpoint=legacy doesn’t work
root@truenas[~]# zfs set mountpoint=legacy FNMedia
cannot set property for 'FNMedia': pool is read-only
Pool imports after a hardware failure can be messy when the new system sees the disks but ZFS refuses to assemble them. I usually start by checking zpool import -f and confirming all drives report the same pool GUID. Sometimes stale labels or incomplete controller mappings confuse the import process. Cleaning up the topology and matching the old layout often brings the pool back online without further damage.
I threw the dmesg logs into Caude to understand potential causes for the reset and inability to import without readonly here are the results:
Primary Issue: Kernel Deadlock in ZFS Spa Operations
The dmesg log shows multiple critical processes hanging for over 241 seconds in spa_lookup() operations:
[ 5438.160864] INFO: task python3:5378 blocked for more than 241 seconds.
[ 5438.160709] INFO: task middlewared (wo:10979 blocked for more than 241 seconds.
[ 5438.168574] INFO: task middlewared (wo:11036 blocked for more than 120 seconds.
All blocked processes are stuck in the same call chain:
cv_wait_common+0xef/0x130 [spl]
spa_lookup+0x3e/0xe0 [zfs]
spa_open_common+0x75/0x440 [zfs]
spa_get_stats+0x4e/0x210 [zfs]
Hopefully I can get the pool mounted one way or another, but any thoughts if i should file a bug?
