Hi everyone,
I would appreciate some advice on two deleted file objects that remain in the ZFS delete queue and continue to appear as permanent errors.
I no longer need either file and am happy to lose their contents. My goal is to remove these two objects while preserving the remaining data in the dataset.
Pool and dataset names below have been anonymized.
System and storage
- TrueNAS version: 25.10.7
- Pool:
nas2 - Layout: five-disk RAIDZ1, plus one cache device
- Affected dataset:
nas2/data - Mountpoint:
/mnt/nas2/data - Dataset usage: 16.1 TiB; logical usage: 17.6 TiB
- Available space: 35.8 TiB
- No child datasets
- No current snapshots found for this dataset or its descendants
- Both the pool and dataset have
readonly=off - Dataset
quotaandrefquotaare bothnone
I do not currently have an independent backup. I understand that this limits the safe recovery options.
Persistent errors
The pool reports ONLINE, but zpool status -v "nas2" continues to show:
errors: Permanent errors have been detected in the following files:
nas2/data:<0x3812e>
nas2/data:<0x3c486>
At the latest check, all five data disks were ONLINE, with READ=0, WRITE=0, and CKSUM=436 on each. The cache device showed zero errors. I have not established the original cause of the corruption.
What I found with zdb
Both objects are reported as ZFS plain file, with a link count of zero:
| Object ID | Decimal ID | File size in bytes | Links |
|---|---|---|---|
0x3812e |
229678 | 1,180,555,312 | 0 |
0x3c486 |
246918 | 28,124,510,642 | 0 |
The dataset master node identifies object 33 as DELETE_QUEUE. Inspecting it still shows exactly these two entries:
microzap: 13312 bytes, 2 entries
3c486 = 246918
3812e = 229678
The dataset’s object-set ID is 4491 (0x118b).
Error events
An error-only scrub completed immediately but reported:
scrubbed 3 error blocks in 0 days 00:00:00
The same two objects remained listed afterward.
The captured events include:
zio_err = 0x34
zio_objset = 0x118b
zio_object = 0x3812e
zio_level = 0x1
zio_blkid = 0x6
Another captured entry shows:
zio_object = 0x3c486
zio_level = 0x1
zio_blkid = 0x8e
These are excerpts, not a complete listing of all three reported error blocks. The zio_level=1 entries appear to indicate damaged indirect metadata blocks.
Steps already taken
-
Rebooted TrueNAS; the same objects remained.
-
Checked for open, deleted files with:
lsof -nP +aL1 "/mnt/nas2/data"This returned no output.
-
Confirmed
zfs_unlink_suspend_progressis0. -
Confirmed
feature@head_errlogisactive. -
Ran:
zpool scrub -e -w "nas2"Both permanent errors and both delete-queue entries remained.
-
Deleted the former parent directory tree of object
0x3812e. A subsequent check still showed both objects in the delete queue and in the permanent-error list.
I also found OpenZFS discussion #18783, which describes similar persistent errors for deleted objects with links=0, although I do not know whether the underlying cause is the same.
Questions
- Is this the expected outcome when unreadable indirect blocks prevent an unlinked object from being freed, or could it be a known cleanup issue?
- Is there a supported, narrowly scoped way to discard these two objects and complete cleanup while preserving the rest of the dataset?
- Are there relevant fixes or additional read-only diagnostics I should check? I can provide the exact
zfs versionoutput and further logs.
Thank you for any guidance. I would particularly appreciate advice on the appropriate next step before attempting low-level metadata changes.