Files vanished while still taking up space and error "Failed retrieving USER quotas"

After a recent reboot, I encountered an issue where my datasets are not showing the files, but the storage space is still occupied. I did not mess with ACL or any other permissions, but I suspect this might be related to user perimissions somehow.
I use NFS to access my data and I mount as root. Advanced NFS settings are set to Mapall User: root and Mapall Group: wheel. I suspect this is not the best way to do it, but I`m not sure this is the cause of the issue. This is how I mount on my PC:

sudo mount -t nfs -v $available_ip:mnt/MrBig/Big $mount_location

Specifically, I see the following behavior:

  • The main datasets storing data show used space in accordance to the data stored, but the data is not accessible (the folders are empty when mounting)
  • There are no files when mounting with NFS, butI still have full permissions. I can create new files that seem to show up directly to parent pool quota (MrBig) instead of counting as files stored to that children (MrBig/Big).
  • Clicking on the pools in the GUI results in an error related to space quotas.
  • Error message: [EFAULT] Failed retreiving USER quotas for MrBig/Big
    *Sometimes I get a similar error message: 'Failed retreiving GROUP quotas for MrBig/Big`

Full error message: [EFAULT] Failed retreiving USER quotas for MrBig/Big

 Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 760, in get_quota
    quotas = resource.userspace(quota_props)
  File "libzfs.pyx", line 465, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 760, in get_quota
    quotas = resource.userspace(quota_props)
  File "libzfs.pyx", line 3532, in libzfs.ZFSResource.userspace
libzfs.ZFSException: cannot get used/quota for MrBig/Big: dataset is busy

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 115, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 762, in get_quota
    raise CallError(f'Failed retreiving {quota_type} quotas for {ds}')
middlewared.service_exception.CallError: [EFAULT] Failed retreiving USER quotas for MrBig/Big
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 184, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1317, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1379, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool.py", line 4116, in get_quota
    quota_list = await self.middleware.call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1368, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1325, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1331, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1246, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1231, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [EFAULT] Failed retreiving USER quotas for MrBig/Big

Here is more info about the pool

Type: FILESYSTEM
Sync: STANDARD
Compression Level: LZ4
Enable Atime: OFF
ZFS Deduplication: OFF
Case Sensitivity: ON
Path: MrBig/Big

**Permissions**
Owner: root
Group: root
Unix Permissions
root: Read | Write | Execute
root: Read | Write | Execute
Other: Read | Write | Execute


**User quotas**
 Name ID Data Quota DQ Used DQ % Used Object Quota OQ Used OQ % Used
root 	0 	— 	47.93 GiB 	— 	— 	35859 	— 	
— 	1000 	— 	2.18 TiB 	— 	— 	984939 	— 

**Group quotas**
 Name ID Data Quota DQ Used DQ % Used Object Quota OQ Used OQ % Used
root 	0 	— 	15.89 MiB 	— 	— 	13 	— 	
— 	1000 	— 	1.85 TiB 	— 	— 	775977 	— 	
— 	1001 	— 	392.80 GiB 	— 	— 	244808 	— 

For context, I got this issue weeks ago and after some failed attemps at fixing it, I deleted and recreated my pools. This had solved the issue for the moment. I was able to upload all my data to the server and access it properly. But now,after some time and a reboot the issue is back again and I`m not sure how to properly debug it.
I hope I can fix any config mistakes so I can acces the files again. I have a backup so I can set it up again properly if necesary, but I am afraid that simply reseting will produce the issue again.

I am quite new and inexperienced with this, so I apologise for any rookie mistakes and for potenlitally not following the forum guidelines. Let me know if any information is necesary and thank you for taking the time to help me!

TrueNAS is designed to use ZFS not NFS. Obviously you can try to use the underlying Debian with an NFS file system but you should not be surprised if the GUI fails to do things on an NFS file system.

Did you have snapshots enabled? I ran into this with dataset quotas, where the actively accessible contents were significantly less than the dataset contents. Snapshots made up the difference.

Mounting NFS exports within a zpool is an unsupported configuration and liable to break backend expectations. Don’t do that. We expect the TrueNAS backend to own everything within /mnt.

2 Likes

Thank you so much to everyone for their contributions!
In regards to space taken up by snapshots, I don’t think this is the issue. Here is the output for zfs list -o space

NAME      AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
MrBig       4.41T  2.73T        0B    216K             0B      2.73T
MrBig/Big   1.77T  2.23T     1.85G   2.23T             0B         0B

I understand that I did not use NFS right. I thought this was an intended way to access and transfer files. I have two questions in order to fix my mistakes:

1.

We expect the TrueNAS backend to own everything within /mnt.

Can I restore access by changing the permissions back for MrBig/Big? If so, what is the best way to do it?
Changing it via GUI gives me an error (Error: [EFAULT] [EINVAL] filesystem.chown.uid: Please specify either user or group to change. )

2.

TrueNAS is designed to use ZFS not NFS

What is the best way access to the files in truenas from a GNU/linux device? While following official documentation I found instructions for setting up NFS share (Adding NFS Shares | TrueNAS Documentation Hub) and this is the way I thought this is the recommended way.

I know I am still very inexperienced and I appreciate your patience and support!

Oh, I may have misunderstood your comments. I presumed you were mounting an external NFS export within the pool on the TrueNAS itself (sometimes we have users do this to bypass our path selector validation in the UI).

What is output of df -T /mnt/MrBig/Big on the TrueNAS server?

1 Like

I did not mount an external NFS export, I am using the Unix NFS Share service to access my data on the NAS.

Here is the output of df -T /mnt/MrBig/Big:

Filesystem     Type  1K-blocks  Used  Available Use% Mounted on
MrBig          zfs  4732453376   256 4732453120   1% /mnt/MrBig

Okay. The dataset MrBig/Big doesn’t appear to be mounted. What is output of zfs get mountpoint MrBig/Big and zfs get mounted MrBig/Big?

1 Like

The output of zfs get mountpoint MrBig/Big is:

NAME       PROPERTY    VALUE           SOURCE
MrBig/Big  mountpoint  /mnt/MrBig/Big  default

The output of zfs get mounted MrBig/Big is:

NAME       PROPERTY  VALUE    SOURCE
MrBig/Big  mounted   no       -

Okay. So there’s your problem. Your dataset is not mounted and therefore not accessible. Is this dataset being replicated from another server? Is the dataset encrypted?

1 Like

The dataset is not replicated from another server.
It is encrypted with a keyfile but the GUI shows it as unlocked.

What is output of zfs get all MrBig/Big?

1 Like

The output is:

NAME       PROPERTY               VALUE                  SOURCE
MrBig/Big  type                   filesystem             -
MrBig/Big  creation               Tue May 28  2:39 2024  -
MrBig/Big  used                   2.23T                  -
MrBig/Big  available              1.77T                  -
MrBig/Big  referenced             2.23T                  -
MrBig/Big  compressratio          1.03x                  -
MrBig/Big  mounted                no                     -
MrBig/Big  quota                  none                   local
MrBig/Big  reservation            none                   local
MrBig/Big  recordsize             128K                   default
MrBig/Big  mountpoint             /mnt/MrBig/Big         default
MrBig/Big  sharenfs               off                    inherited from MrBig
MrBig/Big  checksum               on                     default
MrBig/Big  compression            lz4                    inherited from MrBig
MrBig/Big  atime                  off                    inherited from MrBig
MrBig/Big  devices                on                     default
MrBig/Big  exec                   on                     default
MrBig/Big  setuid                 on                     default
MrBig/Big  readonly               off                    default
MrBig/Big  zoned                  off                    default
MrBig/Big  snapdir                hidden                 default
MrBig/Big  aclmode                discard                inherited from MrBig
MrBig/Big  aclinherit             discard                local
MrBig/Big  createtxg              25                     -
MrBig/Big  canmount               on                     default
MrBig/Big  xattr                  sa                     local
MrBig/Big  copies                 1                      local
MrBig/Big  version                5                      -
MrBig/Big  utf8only               off                    -
MrBig/Big  normalization          none                   -
MrBig/Big  casesensitivity        sensitive              -
MrBig/Big  vscan                  off                    default
MrBig/Big  nbmand                 off                    default
MrBig/Big  sharesmb               off                    inherited from MrBig
MrBig/Big  refquota               4T                     local
MrBig/Big  refreservation         none                   local
MrBig/Big  guid                   763619536625961138     -
MrBig/Big  primarycache           all                    default
MrBig/Big  secondarycache         all                    default
MrBig/Big  usedbysnapshots        1.85G                  -
MrBig/Big  usedbydataset          2.23T                  -
MrBig/Big  usedbychildren         0B                     -
MrBig/Big  usedbyrefreservation   0B                     -
MrBig/Big  logbias                latency                default
MrBig/Big  objsetid               646                    -
MrBig/Big  dedup                  off                    default
MrBig/Big  mlslabel               none                   default
MrBig/Big  sync                   standard               default
MrBig/Big  dnodesize              legacy                 default
MrBig/Big  refcompressratio       1.03x                  -
MrBig/Big  written                0                      -
MrBig/Big  logicalused            2.30T                  -
MrBig/Big  logicalreferenced      2.29T                  -
MrBig/Big  volmode                default                default
MrBig/Big  filesystem_limit       none                   default
MrBig/Big  snapshot_limit         none                   default
MrBig/Big  filesystem_count       none                   default
MrBig/Big  snapshot_count         none                   default
MrBig/Big  snapdev                hidden                 default
MrBig/Big  acltype                posix                  local
MrBig/Big  context                none                   default
MrBig/Big  fscontext              none                   default
MrBig/Big  defcontext             none                   default
MrBig/Big  rootcontext            none                   default
MrBig/Big  relatime               off                    default
MrBig/Big  redundant_metadata     all                    default
MrBig/Big  overlay                on                     default
MrBig/Big  encryption             aes-256-gcm            -
MrBig/Big  keylocation            none                   default
MrBig/Big  keyformat              hex                    -
MrBig/Big  pbkdf2iters            0                      default
MrBig/Big  encryptionroot         MrBig                  -
MrBig/Big  keystatus              unavailable            -
MrBig/Big  special_small_blocks   0                      default
MrBig/Big  snapshots_changed      Tue Jun 11 23:00:00 2024  -
MrBig/Big  org.truenas:managedby  100.109.39.111         local

This was very insightful!
The encryption key was not loaded for some reason even though the GUI showed the pools as unlocked.

sudo zfs mount MrBig/Big
cannot mount 'MrBig/Big': encryption key not loaded

I loaded the keys for the root (MrBig) succesfully ( zfs load-key MrBig), as it was a requirement to have the keys loaded for the root pool.
I tried to load the key for the dataset MrBig/Big but the same error kept popping up ( that the encryption root key for MrBig needed to be loaded first. The next command shows the key for the parent dataset (MrBig)is already loaded.
Following this, I attempted to mount the ‘MrBig/Big’ dataset but faced a permission error instead of the previous missing keys error. I also tried creating the mount point directory manually using but this operation was also not permitted. Finally, I checked the permissions of the ‘/mnt’ directory to understand the issue better.

root@truenas[~]#zfs load-key MrBig/Big
Key load error: Keys must be loaded for encryption root of 'MrBig/Big' (MrBig).
root@truenas[~]# sudo zfs load-key MrBig
Key load error: Key already loaded for 'MrBig'.
root@truenas[~]# sudo zfs mount MrBig/Big
cannot mount '/mnt/MrBig/Big': failed to create mountpoint: Operation not permitted
root@truenas[~]# sudo mkdir -p /mnt/MrBig/Big
mkdir: cannot create directory ‘/mnt/MrBig/Big’: Operation not permitted
root@truenas[~]# sudo ls -ld /mnt
drwxr-xr-x 4 root root 4 Jun 10 16:21 /mnt

It’s not clear to me from this what the underlying issue is. I will keep attempting to mount the dataset.

After a reboot, I had to manually load the key file. After unlocking the datasets running sudo zfs mount MrBig/Big worked and the data is accesible again via NFS and the error [EFAULT] Failed retreiving USER quotas for MrBig/Big no longer shows up.
Thank you for all the help, I was feeling very stuck!

I have no idea what has caused this issue and I will play with it a little, in order to set the key file again and hopefully auto-mount on boot. That being said, I am so glad I don’t have to reset the pools and start all over again!

1 Like