Orphaned dataset directory after moving using rename

TrueNAS General
1

Hi! Brand new to TrueNAS and this is such a breath of fresh air to manage at the dataset level!

I did a zfs rename to move a dataset to a subdirectory:

zfs rename Pool1/Home Pool1/shares/Home

TrueNAS left what looks like an orphaned dataset mount:

root@matrix[/mnt/Pool1]# ls -altr
total 20
drwxr-xr-x 3 root     root   3 Oct  9 00:18 ..
drwxr-xr-x 2 root     root   2 Oct  9 03:19 Home
drwxrwx--- 2 root     root   2 Oct  9 15:08 apps
drwxr-xr-x 2 root     root   2 Oct  9 15:24 VMs
drwxrwxr-x 4 squatter common 5 Oct  9 20:53 media
drwxr-xr-x 8 root     root   8 Oct 10 03:44 shares
drwxr-xr-x 7 root     root   7 Oct 10 03:44 .

the directory is empty:

root@matrix[/mnt/Pool1]# ls -al Home
total 9
drwxr-xr-x 2 root root 2 Oct  9 03:19 .
drwxr-xr-x 7 root root 7 Oct 10 03:44 ..

and I verified my Home dataset is indeed in the right place:

root@matrix[/mnt/Pool1]# /sbin/zfs list | grep Home
Pool1/shares/Home                                        492G  1.02T   492G  /mnt/Pool1/shares/Home

While I realize I could have broken my system, I tried anyways:

root@matrix[/mnt/Pool1]# rmdir Home
rmdir: failed to remove 'Home': Operation not permitted

It’s just cosmetic so I’m not overly concerned but my OCD is kicking in :sweat_smile:

Even after a reboot (DXP2800 running SCALE 25.04), the orphaned directory is squatting pretty well :smiley:

The Home dataset works fine in its new location.

2

Any special permissions on it?

SCALE/CE:

nfs4xdr_getfacl /mnt/Pool1/Home

Core:

getfacl /mnt/Pool1/Home
3

You can also check that nothing is using the path.

lsof +D /mnt/Pool1/Home
4 
$ nfs4xdr_getfacl /mnt/Pool1/Home
nfs4xdr_getfacl: Failed to get NFSv4 ACL
$ lsof +D /mnt/Pool1/Home
$
5

it’s been a few weeks - the unit has been rebooted several times for upgrades and hardware changes and the pool has since been scrubbed. still getting the same error:

$ sudo rmdir /mnt/Pool1/Home
rmdir: failed to remove '/mnt/Pool1/Home': Operation not permitted
$ sudo lsof /mnt/Pool1/Home
$ sudo nfs4xdr_getfacl  /mnt/Pool1/Home 
nfs4xdr_getfacl: Failed to get NFSv4 ACL
$ sudo zfs list | grep Home
Pool1/shares/Home                                            506G  1.02T   495G  /mnt/Pool1/shares/Home

how do we find out what else TrueNAS believes /mnt/Pool1/Home is used for?

6

It might be immutable directory. Depending on circumstances we will create immutable directory when dataset is unmounted to prevent people writing into the mountpoint of a ZFS dataset and then “losing” their data when the dataset is mounted over the path.

c.f. man lsattr and man chattr

7

Remove the NFS share.

1 Like
8

not sure if this was an invitation to use the lsattr command but

lsattr /mnt/Pool1/Home

doesn’t return anything.

no NFS shares active (in fact, the NFS service/daemon is not active) - only SMB and SSH.

9

@udance4ever : I could not have written my question any better. In fact I think the only difference is our pool / dataset naming conventions. :slight_smile:

I am in exactly the same situation:

  • zfs rename pool/foo pool/bar
  • Destination “bar” is working perfectly fine
  • Leftover pool/foo “object” that looks like a plain ol’ directory (not dataset)
  • zfs destroy pool/foo results in “dataset does not exist”
  • rm “foo” results in “Operation not permitted”
  • lsattr foo results in no output at all
  • removed snapshots
  • no containers
  • no docker apps
  • no vms
  • rebooted several times
  • The ‘OCD’ :wink: – this is no obvious reason to think this thing is negatively impacting the server… it’s just my sanity that is affected.

Have you made any progress getting rid of this interloper?

Thanks!

10

Never had that issue. Therefore I can only give some generic troubleshotting/diagnostic commands that I would try. Have you checked the mount table like this:

mount | grep /mnt/pool/foo

There shouldn’t be anything mounted there, but maybe check to make sure.

It would be interesting to know if maybe the dataset is read-only or something.
Can you try if you can create files in /mnt/pool and /mnt/pool/foo. Something like:

touch /mnt/pool/test
touch /mnt/pool/foo/test

And remove them afterwards.

Readonly state can be tested with:
/sbin/zfs get readonly pool
I think mount would also show ro for /mnt/pool if it were mounted read-only.

Last but not least, this might give some clues:

stat /mnt/pool/foo
11

@bacon - thanks for the crisp and tasty feedback!

Just getting back to this now and… wouldn’t you know it, it disappeared on its own.

I have rebooted one or twice since then, but honestly I’m not sure if that was the magic fix or not.

Just the same, I appreciate the time you took.

paq237