TL;DR: Encrypted dataset’s master node (object 1) fails MAC/authentication on both stored copies. All 181 underlying file/dir objects are confirmed intact and 100% full (29.1G), but the broken root blocks all enumeration, path lookups, and zfs send. No snapshots, no backup. Looking for a way to recover the data, or confirmation there isn’t one.
Setup
RAIDZ1, 4 drives, all ONLINE 0 0 0. Encrypted dataset SATA.pool/SMB/General. Corruption discovered on first boot after VM was powered off ~1 month (clean Proxmox shutdown, no hardware/config changes while off).
Error
errors: Permanent errors have been detected in the following files:
SATA.pool/SMB/General:<0x1>
Object <0x1> = ZFS master node (holds ROOT/DELETE_QUEUE pointers), not a user file.
Confirmed via zdb
-
zdb -dd→ all 181 objects present, 100% full, including several multi-GBZFS plain fileentries. Data is not lost at the object level. -
zdb -ddddddd ... 1→ master node’s block pointer shows two DVAs (two on-disk copies):0:1400910000:2000and0:180065e000:2000. -
Scrub:
repaired 0B ... 0 errors, permanent error on object 1 remains → both copies fail identically, so it’s not a simple “one good copy, one bad” scenario. -
zpool events -v→ error class isereport.fs.zfs.authentication(MAC failure), not a plain checksum error:zio_objset = 0xdb zio_object = 0x1 zio_level = 0x0 zio_blkid = 0x0 -
zfs sendfails instantly, before any data transfers (Invalid argument), so no pool-to-pool copy possible. -
lson the dataset root returns almost nothing (just.DS_Store, created after the fact), and direct path lookups (.../General/knownfoldername) fail identically whether the path is real or made up — name resolution can’t get past the broken root at all.
My read on it
Both DVAs failing authentication identically feels less like “corrupted bytes” and more like the block being authenticated against the wrong crypto context — possibly from an interrupted/torn write to object 1 during that final shutdown or first boot. If so, the pre-corruption version of object 1 might still exist on disk from an earlier txg, just unreferenced now.
Questions
-
Anyone dealt with
ereport.fs.zfs.authenticationon a master node specifically? Known recovery path? -
Is a txg rewind (
zpool import -T/ rewind-related recovery) viable here, given both DVAs fail the same way? Any gotchas specific to encrypted pools? -
Any lower-level way to extract file data directly from the known-intact objects (confirmed via
zdb -dd) without going through the broken root?
Not fishing for “restore from backup” — no snapshots/backup, lesson learned. Happy to provide more zdb output on request.
Env: TrueNAS (smbd 4.22.6-truenas), RAIDZ1 4-wide, encrypted dataset, no snapshots.
Note: this write-up was compiled by an AI assistant while walking through diagnostics with me (the pool owner) step by step. All commands/output above are from my actual system — happy to run anything else that’d help diagnose.