Unrecoverable I/O Error after updates

TrueNAS General
, ,
1

I’ve searched and tried everything I found to address this problem but I’m still unable to get anywhere. My hardware setup is in my signature.

In short, I had one drive die and sent it out for RMA. During that time, I upgraded TrueNAS to the latest version. All was fine.

One day I saw an update available for Radarr and triggered the update. The update never completed, I left it for a long time (hours). At some point I powered down the server completely through the UI (not using the case button or the terminal, just a standard shutdown).

When powering it back up, there were Chksum errors and I checked what was going on with zpool status -v. There were 3 files, one Radarr log db file (that I deleted) and two also radarr db files that had a path which made no sense (so I didn’t do anything about them). I forget what occurred after this, but at some point after a reboot it was refusing to mount the pool. It was giving a “Unrecoverable I/O error and has been suspended” message. During boot, it was failing to run ix-service.etc and ix-zfs-etc (if I recall correctly). One post here had a process for booting with the pool unattached and to disconnect the pool. I did that and TrueNAS boots perfectly fine.

When trying to import the pool, it just hangs and never imports (the post said it would). I booted a Void Linux based recovery and it won’t import the pool either. I tried an import using the readonly=on switch and that does work. I can browse through the content that way.

Other things I’ve done:

  • Replaced the original eVGA Supernova 600 Bronze with an eVGA Supernova G2 Gold
  • Replaced all the SATA cables using the existing Intel 612 built into the motherboard
  • Installed a know working PCI-E controller in IT mode with SATA breakout cables to take the Intel 612 out of the equation entirely

Nothing allows me to import the pool in read/write mode. I’m trying to see if I can save the data. If reinstalling TrueNAS will solve the problem I’m willing to do that. I believe based on all the steps I’ve taken, it’s not a hardware issue. I’m looking for feedback or other tests I can attempt to determine the root cause and plan my next steps.

2

readonly=on working means the pool is still there. do not reinstall truenas yet, that will not fix a suspended pool.

first thing i’d do is stay readonly and copy the important stuff off to another disk/pool. get a backup before trying anything that writes to it.

while it is still readonly, post zpool status -v, zpool import, and the relevant disk/controller errors from dmesg. avoid a scrub, zpool clear, or import -F / -X until the data is copied. those can make a bad situation worse.

since you already swapped cables, HBA and PSU, a big read-only copy is also a useful test. if that stays clean, deal with getting it writable after the data is safe. reinstalling the OS wont fix the pool metadata.

2 Likes
3

Thanks nite_route. Here’s the output of the readonly import.

cannot mount ‘/STORAGE’: failed to create mountpoint: Read-only file system
Import was successful, but unable to mount some datasets

Here’s the status results.

pool: STORAGE
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: Link to docs removed
scan: scrub canceled on Wed Aug 12 20:41:07 2026
config:

    NAME                                      STATE     READ WRITE CKSUM
    STORAGE                                   DEGRADED     0     0     0
      raidz2-0                                DEGRADED     0     0     0
        0c32bd7b-ed0e-11ee-a68d-0cc47a69304a  ONLINE       0     0     0
        11335415869523404466                  UNAVAIL      0     0     0  was /dev/disk/by-partuuid/eb4610fb-ebd0-11ee-852e-0cc47a69304a
        06ba8809-cc27-45aa-b787-ee4c3ffca855  ONLINE       0     0     0
        562c8577-4829-11ef-aff4-3cecef4a01be  ONLINE       0     0     0
        fb7444be-eca0-11ee-a355-0cc47a69304a  ONLINE       0     0     0
        9c7cd2d9-eafb-11ee-b908-0cc47a69304a  ONLINE       0     0     0
        cbf69f24-d1bf-4176-a977-f806fcbc16db  ONLINE       0     0     0
        104a6c32-cff7-401d-9bd0-f95606b24b12  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:04:16 with 0 errors on Fri Aug 21 03:49:18 2026
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdi3    ONLINE       0     0     0
        sdg3    ONLINE       0     0     0

errors: No known data errors

dmseg shows nothing wrong except errors related to sound devices.

Error: Driver ‘pcspkr’ is already registered, aborting…
snd_hda_intel 0000:06:00.0: Unknown capability 0

Thanks in advance for your help. I don’t mind losing the data really, it’s not important stuff. It’d be nice to keep it but not necessary. And I don’t have another way to store it due to the amount. I mostly want to make sure I can just get the server back up and running again. Ideally I’d get the custom compose I used for one app because it took me a long time to figure out.

4

the unavail disk is still in the way. raidz2 can run missing one, but a rw import will hang/suspend if that ghost disk is still part of the attempt.

readonly trying to mount at /STORAGE fails because / is not writable from that import. use an altroot:

zpool import -o readonly=on -R /mnt/recover STORAGE

then grab the compose/yaml from whatever apps dataset you care about under /mnt/recover. do that first.

after you have the yaml, if the data really doesnt matter:

zpool import -f -N STORAGE
zpool offline STORAGE 11335415869523404466

if it still suspends youre looking at destroy + new pool. reinstalling truenas still wont fix pool metadata.

the permanent errors file list got cut off in your paste. if you want to keep it, post that section from zpool status -v.

1 Like
5

Thank you nite_route. I was able to mount correctly with the command you provided. I tried to get to the apps, but /mnt/.ix-apps is empty. So it seems there’s no way for me to get to that YAML now. I think it’s because when I disconnected the pool in the UI without the pool powered, I also used the UI function to unset the pool used by Apps (I forgot what the option was called, I was trying whatever to get it going). Really not a big deal, I’m sure I’ll figure out what I did and be able to redo it.

I pasted the full content of the status results, the list of impacted files is empty. I tried to upload a screenshot of it but I’m unable to.

I do have the replacement disk now so if there’s something I should do with it to help resolve this, I can do that.

6

I thought this was interesting. I ran some various attempts at “help me diagnose the problem” through claude code and without going over the massive amount of output and result, this is the finding I found most pertinent.

  • The single unnamed “1 data error” is consistent with a damaged space map — pool metadata, which is why it never resolved to a filename. zdb reads all 349 space maps and the log space maps without complaint, so the damage is narrow, but the kernel hits it the moment it tries to flush metaslabs.

It wants to attempt a read-write repair…. oh boy, this should be interesting.

7

I had it do some further analysis and this is what it determined. This could be useful for others so I’m posting for closure, but I’m pretty sure I only have one path forward now.

Root cause, definitively

MOS object 3205 — a SPA space map (metaslab 309) — block 2, at DVA 0:2703494e8000:9000, has an uncorrectable checksum error.

The mechanism, end to end:

1. A read-write import must sync an initial txg → spa_flush_metaslabs.
2. Flushing metaslab 309 requires reading and rewriting block 2 of its space map.
3. That read returns ECKSUM. The vdev is raidz2 already down one member, so the stripe lacks the surviving redundancy to reconstruct it.
4. A metadata write that cannot complete → ZFS suspends the entire pool → txg_sync wedges → the importing process blocks in txg_wait_synced forever.

A read-only import never flushes metaslabs, never touches object 3205, and works perfectly. That is why the previous session’s read-only-only methodology — chosen for safety — was precisely the thing that hid the fault. It concluded “routine degraded-vdev repair”; the pool had in fact been unimportable read-write for 8 days by then.

I also cleared up the old mystery: the “1 data error” is object 17 in radarr/config, and zdb returns errno 2 for it — the file was deleted, which is why ZFS could never name it. Benign, unrelated.

And the real history is worse than Part 1 suggested. ZED logged err=52 checksum errors on Aug 12 during the scrub — which was then cancelled after 5h46m, followed by two zpool clears that reset the counters without repairing anything. Damage accumulated on a pool running with zero redundancy margin until a space map block died on Aug 25.

What this rules out

Replace-and-resilver is impossible — a resilver requires a read-write import, which is the exact operation that suspends the pool. Same for zpool scrub. Rebooting again just reproduces bookmark 0:3205:0:2 a 15th time.

What’s genuinely fine

All 8 disks pass SMART with zero reallocated/pending/uncorrectable/CRC. Zero ATA errors across all 14 failed boots. Nothing write-protected. The ISO re-read to md59138af17575ea754acb03ef8b6ccecdd, identical to last session, counters 0/0/0. No hardware here is faulty.

1 Like