Mirror pool won't import — reproducible `zio_err=0x34` on root objset during log_replay, all standard recovery flags fail (TrueNAS SCALE, ZFS 2.3.4)

TrueNAS General
, ,
1

Hello everyone. I have some issues with my system and with the help of Claude AI, I tried to fix it. but failed. This post was made with AI, claude had all my reports and contexts.

Hoping someone recognizes this signature. I have a 2-disk mirror pool that went offline unexpectedly and now fails to import with every standard recovery method, despite strong evidence the pool and data are structurally intact.

System: ZimaBlade, TrueNAS SCALE 25.10.5 (reverted back to this version after briefly running a newer update — the pool broke shortly after that update, within about a day)

Pool: 2x 4TB WD Red Plus (WDC WD40EFPX) in a single mirror vdev, ashift=12, ZFS pool version 5000 / filesystem version 5

Symptom: Pool shows offline in the UI. zpool import (no args) correctly detects the pool as ONLINE with both mirror members ONLINE. But actually importing it — by name, with -f, -F, -FX, -o readonly=on, in every combination, with zil_replay_disable=1 and zfs_recover=1 set — fails every time with either:

cannot import 'Nirala_Main_Pool': one or more devices is currently unavailable

or, with readonly/-F variants:

cannot import 'Nirala_Main_Pool': insufficient replicas
Destroy and re-create the pool from a backup source.

What we’ve ruled out:

  • Drive health: SMART overall-health PASSED on both disks, 0 reallocated sectors, 0 pending sectors, 0 offline uncorrectable, 0 UDMA CRC errors on both. Clean extended self-test history.

  • Cabling/power: dmesg shows zero ata reset/link/error events during any import attempt.

  • Label/uberblock corruption: zdb -l on both disks shows all 4 labels intact and identical (txg=4145589 at first check, later txg=4147295), matching guid_sum across both disks.

  • Hostid mismatch: found and ruled out as sole cause (pool hostid didn’t match system hostid, likely from an unclean shutdown, but forcing past this alone didn’t resolve the import).

  • Filesystem/read-write mode: confirmed block devices are fully read-write (/sys/block/sdX/ro = 0, blockdev --getro = 0).

The actual reproducible signature, from zpool events -v after every failed attempt:

ereport.fs.zfs.data
  zio_err = 0x34
  zio_objset = 0x0
  zio_object = 0x125
  zio_level = 0x0
  zio_blkid = 0x0
  zio_priority = 0x0 [SYNC_READ]
followed immediately by:
ereport.fs.zfs.log_replay

This exact pair repeats identically across every import attempt, regardless of flags used. zio_err=0x34 (52, ENOMSG) on object 0x125 in objset 0x0 (the MOS/root objset) during log replay, every time.

Most interesting data point: zdb -e -p /dev/disk/by-partuuid Nirala_Main_Pool completes successfully:

spa_misc.c:430:spa_load_note(): spa_load(Nirala_Main_Pool, config trusted): LOADED

So zdb can fully load the pool’s trusted config from disk, which suggests the underlying data and most metadata are intact — the failure appears isolated to processing one specific intent log record during the import path itself.

Suspected trigger: Two days before discovery, I ran zpool import -f once, unsuccessfully, and I’m not fully certain if it was interrupted (system was later found in a bad state, and I subsequently reverted TrueNAS to the prior version). My best guess is this left a partially-written/corrupted ZIL record referencing object 0x125 that log replay can’t process, and which none of the standard bypass mechanisms are routing around.

Question: Is there a way to force ZFS to skip/discard this specific corrupted log record (object 0x125 in the MOS) during import rather than aborting entirely? Is zfs_recover=1 expected to cover this class of MOS-level replay error, or is there a lower-level tool/patch needed here? Happy to provide full zdb -e -bcsvL output or anything else that would help diagnose.

Data confirmed intact via zdb, but zpool import structurally blocked — now stuck on zdb -B 'Permission denied' (TrueNAS SCALE, ZFS 2.3.4)
2

Update: ran memtest86+ overnight, 4 full passes, 0 errors. RAM is ruled out. This pushes suspicion toward a possible bug in the TrueNAS version I briefly upgraded to (issue began within ~24 hours of that upgrade, before I reverted back).

3

Ran it via direct SSH with middlewared stopped first, so this should be a clean test:

sudo systemctl stop middlewared
sudo zpool export Nirala_Main_Pool
sudo zpool import -o readonly=on -f Nirala_Main_Pool

Same result as before — insufficient replicas, and zpool events -v still shows the ereport.fs.zfs.data (zio_err=0x34, zio_object=0x125, zio_objset=0x0) immediately followed by ereport.fs.zfs.log_replay. So per your note, this confirms the failure is in spa_ld_verify_logs/spa_check_logs, not downstream in spa_ld_verify_pool_data.

Also ran zdb -e -p /dev/disk/by-partuuid -dddd Nirala_Main_Pool 0 293 as you suggested — it completed cleanly with no error, showing object 293 as a “DSL props” object, dnode maxblkid 0, 100% full, 12K dsize. No checksum failure surfaced in this specific dump.

zdb -e -C output: features_for_read are only hole_birth, embedded_data, and one com.klarasystems feature — nothing exotic. Full MOS config pasted below if useful. [paste config]

Given object 293 dumps clean via zdb but the import path still fails checksum verification on it during spa_check_logs, is the next move to look at the ZIL chain/log records themselves rather than the dataset object?

4

One more thing for completeness: I mentioned trying -f two days before this thread started — I also may have tried -F at that time (before I understood what these flags did), though I’m not certain. So it’s possible a rewind attempt already occurred prior to any of the diagnostics in this thread.

5

You are probably better off restoring from a backup considering you haven’t had any replies to this thread and few of us really know what you are posting, except for ZFS experts.

6

I am having the same conversation on the zfs forum, somebody there is helping me with this issue. I will post a link to that after a solution is reached.
Thank you though. He thinks it is solvable but the exact cause is still pretty mysterious. We checked and ruled out most common causes.

7

[RESOLVED] 3TB ZFS mirror pool unimportable due to a single corrupted metadata object; full root cause, working recovery method, and an upstream bug found along the way

Wanted to post a full writeup now that we’ve found a working path forward, both to close the loop for anyone following and to document this for the next person who runs into something similar.

The original problem:

2-disk mirror pool (2x4TB WD Red Plus) on a ZimaBlade running TrueNAS SCALE, ZFS 2.3.4. The pool went offline after a version upgrade that I reverted shortly after. zpool import failed for every mode and every flag combination I tried (-f, -F, -X, -o readonly=on, -N, spa_load_verify_metadata=0, and combinations of these), always failing with either “device unavailable” or “insufficient replicas, destroy and re-create the pool from a backup source.” No backup existed.

Ruling out hardware:

SMART on both disks came back completely clean (zero reallocated sectors, zero pending sectors, zero read errors). Disk labels and uberblocks were consistent and matched across both mirror members. A memtest86+ run overnight, 4 full passes, came back with zero errors, ruling out bad RAM. This was never a hardware problem.

Finding the actual root cause:

With a lot of generous, source-code-level help from a contributor on the OpenZFS forum, we traced the failure to spa_load_verify() in spa.c. This function unconditionally calls dmu_objset_find_dp() to walk every dataset’s properties before allowing an import to proceed, regardless of which flags are passed. In my pool, one specific object, a DSL properties ZAP (object 293 in the MOS), was failing checksum verification (ECKSUM). That single failure was enough to abort the entire import, every time, with no way to skip verification of just that one object. -F and -X didn’t help either, since they actually re-enable full data verification rather than bypassing it.

Confirming the data itself was fine:

zdb -e was able to fully load the pool’s trusted configuration, meaning the pool structure and data were intact and readable, just blocked by this one import-time check. A full zdb -bcsvL traversal of the entire pool (took several hours) found only 8 checksum errors out of 29.2 million total block pointers, and every one of them was confined to pool-internal bookkeeping (the same DSL props object, plus an SPA space map) or disposable TrueNAS service datasets (.system/samba4, .system/netdata-*). None of my actual data (Photos, Documents, Archived, etc.) showed any corruption at all.

The extraction attempt, and a real upstream bug:

Since import was structurally blocked, the recommended path was zdb -B (backup stream extraction), which reads a dataset directly without needing a successful import. Every dataset failed identically with dmu_send_obj: Permission denied. Someone on r/truenas traced this to spa_do_crypt_abd() returning EACCES when the in-memory encryption keystore isn’t populated, which it never is under zdb -e, since it reads devices directly rather than going through zfs load-key. Adding the w (raw send) flag correctly routed around that, but exposed a second problem: zdb crashed outright with:

ASSERT at module/zfs/dsl_crypt.c:2477:dsl_crypto_populate_key_nvlist()
VERIFY3U(dp->dp_spa->spa_errata, !=, 0) failed (0 != 0)

This reproduced identically on a completely separate machine running ZFS 2.2.2 (Ubuntu 22.04), ruling out anything TrueNAS or version specific. I’ve filed this as an issue on the OpenZFS GitHub repo (openzfs/zfs), since it looks like a genuine upstream bug.

The actual working fix:

Someone (again on r/truenas) had an LLM trace the exact code path and found the real cause: DS_FIELD_IVSET_GUID is only populated on snapshots, not live datasets, and the assertion was choking on a live dataset that lacked it. The fix was simply to target a snapshot instead of the live dataset:

zdb -e -p /dev/disk/by-partuuid -B pool/<snapshot_objset_id> w > output.zsend

This worked immediately and cleanly. I tested it end to end: extracted a snapshot, received it into a scratch test pool with zfs recv, unlocked it with zfs load-key and my original passphrase, and confirmed every file opened correctly, PDFs with valid page counts, JPEGs with intact EXIF metadata, nothing corrupted.

Where things stand now:

The plan going forward is to extract the most recent snapshot of every dataset this same way, verify each one, then destroy and recreate the pool fresh (since there’s no way to fix the underlying import block in place without kernel patching or manual on-disk editing, both of which are well beyond what’s reasonable to attempt outside a professional recovery service), and restore everything onto the new pool. Given the snapshot schedule was frequent, the data loss should be minimal, hours at most for most datasets.

Huge thanks to the OpenZFS forum contributor who did most of the heavy source-tracing that got this diagnosed correctly in the first place, and to the folks on r/truenas who found both the encryption/keystore issue and the actual IVSET_GUID snapshot workaround. This took about three weeks of careful, methodical troubleshooting, and I would not have gotten here without the community’s help.

GitHub issue for the underlying bug: https://github.com/openzfs/zfs/issues/19018

Happy to answer questions if anyone else runs into an unimportable pool with a similar signature.

2 Likes