"Broken" folder - how to recover or delete?

TrueNAS General
1

I have a folder on my RAIDZ1 that just seems to be broken - I know it sounds dumb but I don’t know how to describe it better.

root@truenas1:/mnt/hdd_raidz1/misc# cd ISOs/
root@truenas1:/mnt/hdd_raidz1/misc/ISOs# ls -al
total 0
root@truenas1:/mnt/hdd_raidz1/misc/ISOs# cd ..
root@truenas1:/mnt/hdd_raidz1/misc# rm -rf ISOs/
rm: cannot remove 'ISOs/': Directory not empty
root@truenas1:/mnt/hdd_raidz1/misc# ls -al /mnt/hdd_raidz1/misc/.zfs/snapshot/auto-2025-12-09_11-00/ISOs/
total 38969249
drwxrwxr-x  2 colin        media                17 Oct 18 11:11 .
drwxrws--- 14 media        media                20 Nov 17 14:34 ..
-rwxrwx---  1        64055        64055 5294741504 Aug  6  2022 19045.1865.Vibranium-X64-DE-XXL-SUPER-MULTI.ISO
-rw-rw-r--  1 libvirt-qemu libvirt-qemu 7902445568 Jul 24 07:00 26100.4770.Germanium-X64-DE-XXL-SUPER-MULTI.ISO
-rw-r--r--  1        64055        64055 3291686912 Mar  5  2024 HBCD_PE_x64_1.0.8.iso
-rwxrw-r--  1        64055 media         700448768 Jan 26  2022 KNOPPIX_V9.1CD-2021-01-25-DE.iso
-rwxrw-r--  1 colin        colin             16384 Oct 13 10:30 Thumbs.db
-rw-rw-r--  1 colin        colin         112386048 Sep 30 08:08 VMware-tools-windows-13.0.5-24915695.iso
-rw-rw-r--  1        64055        64055 5026682880 Jun 25  2023 WinPE11_x64_2023.06.25.iso
-rw-r--r--  1        64055        64055 6058381312 Sep 19  2024 WindowsServer2025_DE_Eval_x64.iso
-rw-rw-r--  1 colin        colin        7258769408 Jun 24 08:46 de-de_windows_server_2025_updated_june_2025_x64_dvd_e743555f.iso
-rw-rw-r--  1 colin        colin         589299712 May 28  2025 gparted-live-1.7.0-1-amd64.iso
-rw-rw-r--  1        64055        64055 4022595584 Jun 15  2024 tiny11_24H2_ProWorkstation.iso
-rwxrw-r--  1 libvirt-qemu libvirt-qemu 2773874688 Sep 20  2024 ubuntu-24.04.1-live-server-amd64.iso
-rw-rw-r--  1        64055        64055  724434944 Dec 28  2024 virtio-win-0.1.266.iso
-rw-rw-r--  1        64055        64055  726501376 Apr 17  2025 virtio-win-0.1.271.iso
-rw-rw-r--  1 libvirt-qemu libvirt-qemu  789645312 Sep 15 17:52 virtio-win-0.1.285.iso
root@truenas1:/mnt/hdd_raidz1/misc# cp /mnt/hdd_raidz1/misc/.zfs/snapshot/auto-2025-12-09_11-00/ISOs/*iso ./ISOs/
cp: cannot create regular file './ISOs/HBCD_PE_x64_1.0.8.iso': No such file or directory
cp: cannot create regular file './ISOs/KNOPPIX_V9.1CD-2021-01-25-DE.iso': No such file or directory
cp: cannot create regular file './ISOs/VMware-tools-windows-13.0.5-24915695.iso': No such file or directory
cp: cannot create regular file './ISOs/WinPE11_x64_2023.06.25.iso': No such file or directory
cp: cannot create regular file './ISOs/WindowsServer2025_DE_Eval_x64.iso': No such file or directory
cp: cannot create regular file './ISOs/de-de_windows_server_2025_updated_june_2025_x64_dvd_e743555f.iso': No such file or directory
cp: cannot create regular file './ISOs/gparted-live-1.7.0-1-amd64.iso': No such file or directory
cp: cannot create regular file './ISOs/tiny11_24H2_ProWorkstation.iso': No such file or directory
cp: cannot create regular file './ISOs/ubuntu-24.04.1-live-server-amd64.iso': No such file or directory
cp: cannot create regular file './ISOs/virtio-win-0.1.266.iso': No such file or directory
cp: cannot create regular file './ISOs/virtio-win-0.1.271.iso': No such file or directory
cp: cannot create regular file './ISOs/virtio-win-0.1.285.iso': No such file or directory
root@truenas1:/mnt/hdd_raidz1/misc#

The files seem to be there but also … aren’t. I can’t delete the folder, rolling back to the snapshot shown doesn’t change the behaviour, I cannot create new files (that weren’t there before),

Any ideas how to get this folder working again or at least deleted?

2

The files also still show up in the new snapshot from after my post
checksums for the files in the snapshot check out
deactivating the corresponding smb share changes nothing.

Creating a linked clone from the snapshot that was created just a few minutes ago works.
Or rather the folder therein works. I could promote that dataset, delete the old one and rename the clone but there has to be a better way than that…

3

I’ve now created a clone from a snapshot, prmoted that and renamed the datasets from misc and misc-new to misc-old and misc. The folder works again so if anyone ever has a similar problem - thats a workaround but I’d still like to understand how this happened.
I’ll keep the old dataet lying around for a while so if anyone has an idea and would like me to run some tests its still there…

4

I have no clue. But if you want further diagnosis you can list directory contents using zdb. Which may or may not provider further clues.

  1. Grab the Inode number using stat:
$ stat test
  File: test
  Size: 3               Blocks: 1          IO Block: 3072   directory
Device: 0,64    Inode: 529724      Links: 3
  1. use zdb -ddddd <dataset> <inode>
$ sudo zdb -ddddd tank/unix_homes 529738
Dataset tank/unix_homes [ZPL], ID 307, cr_txg 9998510, 803G, 181775 objects, rootbp DVA[0]=<0:5cad0a86000:2000> DVA[1]=<0:1cb719a000:2000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P 
...
                completions = 528294 (type: Directory)
                .zsh_history = 561332 (type: Regular File)
                .local = 491533 (type: Directory)
...
5

I honestly don’t know what to take from the zdb output but here you go :sweat_smile:

Dataset hdd_raidz1/misc-old [ZPL], ID 56038, cr_txg 44736, 84.6G, 2229 objects, rootbp DVA[0]=<0:2aa40d9c8000:2000> DVA[1]=<0:59d3ffe4000:2000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=13632955L/13632955P fill=2229 cksum=0000000e68b80159:00002724b0e4df2f:0039642c746598ed:3c1f426e131676e0

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
      5593    2    32K    16K    32K     512    32K  100.00  ZFS directory
                                               264   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
        dnode maxblkid: 1
        path    /ISOs
        uid     3002
        gid     3001
        atime   Fri Sep 27 13:53:39 2024
        mtime   Sat Oct 18 11:11:39 2025
        ctime   Sat Oct 18 11:11:39 2025
        crtime  Fri Sep 27 13:53:39 2024
        gen     6166479
        mode    40775
        size    17
        parent  34
        links   2
        pflags  840800000144
        projid  0
        SA xattrs: 80 bytes, 1 entries

                user.DOSATTRIB = \000\000\005\000\005\000\000\000\021\000\000\000\020\000\000\000\073\371\216\344\323\020\333\001
        Fat ZAP stats:
                Pointer table:
                        1024 elements
                        zt_blk: 0
                        zt_numblks: 0
                        zt_shift: 10
                        zt_blks_copied: 0
                        zt_nextblk: 0
                ZAP entries: 15
                Leaf blocks: 1
                Total blocks: 2
                zap_block_type: 0x8000000000000001
                zap_magic: 0x2f52ab2ab
                zap_salt: 0xea85d4133
                Leafs with 2^n pointers:
                          9:      1 *
                Blocks with n*5 entries:
                          3:      1 *
                Blocks n/10 full:
                          2:      1 *
                Entries with n chunks:
                          3:      1 *
                          4:     10 **********
                          5:      3 ***
                          6:      1 *
                Buckets with n entries:
                          0:    497 ****************************************
                          1:     15 **

                Thumbs.db = 734 (type: Regular File)
                virtio-win-0.1.271.iso = 256 (type: Regular File)
                ubuntu-24.04.1-live-server-amd64.iso = 5860 (type: Regular File)
                VMware-tools-windows-13.0.5-24915695.iso = 1158 (type: Regular File)
                virtio-win-0.1.285.iso = 728 (type: Regular File)
                virtio-win-0.1.266.iso = 6953 (type: Regular File)
                HBCD_PE_x64_1.0.8.iso = 644 (type: Regular File)
                WindowsServer2025_DE_Eval_x64.iso = 912 (type: Regular File)
                de-de_windows_server_2025_updated_june_2025_x64_dvd_e743555f.iso = 1408 (type: Regular File)
                26100.4770.Germanium-X64-DE-XXL-SUPER-MULTI.ISO = 10 (type: Regular File)
                19045.1865.Vibranium-X64-DE-XXL-SUPER-MULTI.ISO = 183 (type: Regular File)
                KNOPPIX_V9.1CD-2021-01-25-DE.iso = 5622 (type: Regular File)
                gparted-live-1.7.0-1-amd64.iso = 769 (type: Regular File)
                WinPE11_x64_2023.06.25.iso = 8194 (type: Regular File)
                tiny11_24H2_ProWorkstation.iso = 6765 (type: Regular File)
Indirect blocks:
               0 L1  0:f2a6a91c000:2000 8000L/1000P F=2 B=12742953/12742953 cksum=0000002ab80b3301:0000a3c7b179c0e7:013a5dcf7c59db05:92cbb675a22d55da
               0  L0 0:f2a6a918000:2000 4000L/1000P F=1 B=12742953/12742953 cksum=000000130d185f83:00004ad14e72fb0f:009310be3253c8c7:c0ebce7cd3bd8106
            4000  L0 0:f2a6a91a000:2000 4000L/1000P F=1 B=12742953/12742953 cksum=000000b24a339a1c:0001a4658730c413:026231bb8b8430db:85796e3490e3124d

                segment [0000000000000000, 0000000000008000) size   32K
6

Yeah I’m not sure. Guess I was hoping for some warning/error but everything looks fine. Maybe the ZFS mailing list would have more ideas.

7

I haven’t really used mailing lists before and I don’t know which ZFS mailing list I should choose for this (if any)…