Mirror pool won't import — reproducible `zio_err=0x34` on root objset during log_replay, all standard recovery flags fail (TrueNAS SCALE, ZFS 2.3.4)

TrueNAS General
, ,
1

Hello everyone. I have some issues with my system and with the help of Claude AI, I tried to fix it. but failed. This post was made with AI, claude had all my reports and contexts.

Hoping someone recognizes this signature. I have a 2-disk mirror pool that went offline unexpectedly and now fails to import with every standard recovery method, despite strong evidence the pool and data are structurally intact.

System: ZimaBlade, TrueNAS SCALE 25.10.5 (reverted back to this version after briefly running a newer update — the pool broke shortly after that update, within about a day)

Pool: 2x 4TB WD Red Plus (WDC WD40EFPX) in a single mirror vdev, ashift=12, ZFS pool version 5000 / filesystem version 5

Symptom: Pool shows offline in the UI. zpool import (no args) correctly detects the pool as ONLINE with both mirror members ONLINE. But actually importing it — by name, with -f, -F, -FX, -o readonly=on, in every combination, with zil_replay_disable=1 and zfs_recover=1 set — fails every time with either:

cannot import 'Nirala_Main_Pool': one or more devices is currently unavailable

or, with readonly/-F variants:

cannot import 'Nirala_Main_Pool': insufficient replicas
Destroy and re-create the pool from a backup source.

What we’ve ruled out:

  • Drive health: SMART overall-health PASSED on both disks, 0 reallocated sectors, 0 pending sectors, 0 offline uncorrectable, 0 UDMA CRC errors on both. Clean extended self-test history.

  • Cabling/power: dmesg shows zero ata reset/link/error events during any import attempt.

  • Label/uberblock corruption: zdb -l on both disks shows all 4 labels intact and identical (txg=4145589 at first check, later txg=4147295), matching guid_sum across both disks.

  • Hostid mismatch: found and ruled out as sole cause (pool hostid didn’t match system hostid, likely from an unclean shutdown, but forcing past this alone didn’t resolve the import).

  • Filesystem/read-write mode: confirmed block devices are fully read-write (/sys/block/sdX/ro = 0, blockdev --getro = 0).

The actual reproducible signature, from zpool events -v after every failed attempt:

ereport.fs.zfs.data
  zio_err = 0x34
  zio_objset = 0x0
  zio_object = 0x125
  zio_level = 0x0
  zio_blkid = 0x0
  zio_priority = 0x0 [SYNC_READ]
followed immediately by:
ereport.fs.zfs.log_replay

This exact pair repeats identically across every import attempt, regardless of flags used. zio_err=0x34 (52, ENOMSG) on object 0x125 in objset 0x0 (the MOS/root objset) during log replay, every time.

Most interesting data point: zdb -e -p /dev/disk/by-partuuid Nirala_Main_Pool completes successfully:

spa_misc.c:430:spa_load_note(): spa_load(Nirala_Main_Pool, config trusted): LOADED

So zdb can fully load the pool’s trusted config from disk, which suggests the underlying data and most metadata are intact — the failure appears isolated to processing one specific intent log record during the import path itself.

Suspected trigger: Two days before discovery, I ran zpool import -f once, unsuccessfully, and I’m not fully certain if it was interrupted (system was later found in a bad state, and I subsequently reverted TrueNAS to the prior version). My best guess is this left a partially-written/corrupted ZIL record referencing object 0x125 that log replay can’t process, and which none of the standard bypass mechanisms are routing around.

Question: Is there a way to force ZFS to skip/discard this specific corrupted log record (object 0x125 in the MOS) during import rather than aborting entirely? Is zfs_recover=1 expected to cover this class of MOS-level replay error, or is there a lower-level tool/patch needed here? Happy to provide full zdb -e -bcsvL output or anything else that would help diagnose.

2

Update: ran memtest86+ overnight, 4 full passes, 0 errors. RAM is ruled out. This pushes suspicion toward a possible bug in the TrueNAS version I briefly upgraded to (issue began within ~24 hours of that upgrade, before I reverted back).

3

Ran it via direct SSH with middlewared stopped first, so this should be a clean test:

sudo systemctl stop middlewared
sudo zpool export Nirala_Main_Pool
sudo zpool import -o readonly=on -f Nirala_Main_Pool

Same result as before — insufficient replicas, and zpool events -v still shows the ereport.fs.zfs.data (zio_err=0x34, zio_object=0x125, zio_objset=0x0) immediately followed by ereport.fs.zfs.log_replay. So per your note, this confirms the failure is in spa_ld_verify_logs/spa_check_logs, not downstream in spa_ld_verify_pool_data.

Also ran zdb -e -p /dev/disk/by-partuuid -dddd Nirala_Main_Pool 0 293 as you suggested — it completed cleanly with no error, showing object 293 as a “DSL props” object, dnode maxblkid 0, 100% full, 12K dsize. No checksum failure surfaced in this specific dump.

zdb -e -C output: features_for_read are only hole_birth, embedded_data, and one com.klarasystems feature — nothing exotic. Full MOS config pasted below if useful. [paste config]

Given object 293 dumps clean via zdb but the import path still fails checksum verification on it during spa_check_logs, is the next move to look at the ZIL chain/log records themselves rather than the dataset object?

4

One more thing for completeness: I mentioned trying -f two days before this thread started — I also may have tried -F at that time (before I understood what these flags did), though I’m not certain. So it’s possible a rewind attempt already occurred prior to any of the diagnostics in this thread.

5

You are probably better off restoring from a backup considering you haven’t had any replies to this thread and few of us really know what you are posting, except for ZFS experts.

6

I am having the same conversation on the zfs forum, somebody there is helping me with this issue. I will post a link to that after a solution is reached.
Thank you though. He thinks it is solvable but the exact cause is still pretty mysterious. We checked and ruled out most common causes.