Correct way to script ZFS replication in TrueNAS Scale?

TrueNAS General
,
22

From my testing with the TrueNAS GUI, it was perfectly ok to have a replication target dataset configured non-read-only, shared in SMB and even written to it…

The only thing that is required is to rollback to the common snapshot before trying a new replication. The newly written data on the target dataset is then of course lost, which is perfectly acceptable for me, as I know this will happen…

The target dataset will only be written to in very short / exceptional circumstances where it is used for testing / developing something. So all writes to it are ok to be temporary / redundant.

Am I missing something on why I still shouldn’t do this (even though it seems to work perfectly fine)?

I actually did mean the parent dataset on the target, but, if I understand you correctly, the dataset properties will always come from the source dataset when using ZFS replication?

Is this only for the initial full replication? Or also for the incremental replications later? If I, for example, change a dataset property on my target dataset and then perform an incremental replication, will this changed-dataset-property on the target be overwritten with the property value of the source?

23

Then you have a very unique and “interesting” use-case. If you’re okay with losing newly written data on these shares, then I guess this isn’t a drawback.

Most people would not find that acceptable. (After all, what’s the point of saving files that will soon be destroyed by another server’s replication schedule? What if you need to access these files for more than a day, yet the next replication is about to run in a few minutes?)

As long as you’re okay with the above, then there’s no issue.

If you use the -R flag (“full filesystem”) or any of the properties flags, such as compression, encryption, and so on, then yes, it will use the properties from the source dataset; not inherit from the parent dataset that already lives on the destination. (However, the GUI puts all the properties under a single checkbox named “Include Properties”. The command-line gives you more control, but requires you to know the different flags.)

Yes, unless you explicitly “exclude” the particular property in question. However, this does not work for all properties. Some of them are immutable and are only applied at dataset creation, such as encryption and case-sensitivity. One way to think of it: If you can’t change a certain property on a local dataset? Then it means the replication cannot change a property on a target dataset. Same “rules of ZFS” apply.

1 Like
24

Thanks again for your clarifying reply! Much appreciated…

To be clear, my backup TrueNAS server will be powered off most of the time and will only run replication on demand (or with a long interval, like once per month). So there will be no daily replication schedule that will interfere with my exceptional testing on it.

For most zfs dataset properties, I don’t see an issue in them being copied over from source to target.
Only for the encryption this would be problematic I’m afraid… On both source and target server, most datasets have a single parent dataset that is encrypted with a passphrase. The children datasets of this encrypted dataset on the source inherit this encryption from their parent.
Also on the target TrueNAS server I’d like these children datasets to inherit their encryption from their encrypted parent dataset (and not from the source).

In the TrueNAS GUI replication task, I do this by enabling “Encryption” and “Inherit encryption”. I also have “Include dataset properties” and “Full filesystem replication” disabled.

Any idea idea how I could replicate these settings on the command line?

Also I still wonder if it is “ok” to destroy and create datasets outside the TrueNAS GUI (in my scripts)? Or will this potentially damage my TrueNAS configuration (perhaps the state of the filesystem is then out of sync with what TrueNAS expects it to be?)

25

That must be new for SCALE. I don’t have such an option in TrueNAS Core.

1 Like
26

On the “send” side, you can use -Lec, while on the “recv” you can use -x encryption

This will essentially retain large blocks, embedded blocks, and compression, all “as is” found on the source, which modern versions of ZFS support. Then on the other side, the userkey will be “inherited” from whatever the parent’s encryption userkey is. So after you do your one-time full replication, you can make the destination dataset inherit its parent’s encryption properties, and then -x encryption will prevent the source from overriding this property. Keep in mind that this requires the destination dataset to be unlocked in order for it to receive any ZFS replications.

1 Like
27

Have you thought about adding your script as an init/shudown script using the option in System Settings > Advanced? I don’t know if it would work but that is what this feature is for, running a script outside a normal schedule. If I understand correctly, the remote backup server is a TrueNAS system. Set this up in that system.

https://www.truenas.com/docs/scale/scaletutorials/systemsettings/advanced/manageinitshutdownscale/

1 Like
28

To answer a question from way above, yes, handles encryption perfectly and yes I use encryption. Read the zfs-autobackup docs on the code github site for all those sorts of details, the flags are important. It can even decrypt then re-encrypt, go from no encryption to encryption, etc.

The problem with Scale UI replication is you will find yourself needing to do full replications over after a time as things happen. The tool I suggested handles this for you without this issue, handles restarts due to failed communications without problem, etc. Then there is features like like not copying certain datasets that I needed, that were not easy with Scale UI.

ZFS autobackup is widely used. It’s discussed a lot by ZFS nerds. The code is there, and yes, it uses as @winnielinnie points out standard zfs commands, it just crafts them for you. So, backups are one command in a script. The issue is picking the right options for what you want. I keep my backup un-mounted too. But you may want it mounted. You can copy all properties or only certain ones, etc.

Here’s the command I use for one of my backups, the main offline one:

autobackup-venv/bin/python -m zfs_autobackup.ZfsAutobackup -v -d --no-progress --clear-refreservation --exclude-received --keep-source=10,1d1w,1w1m,1m3m --keep-target=10,1d1w,1w1m,1m3m --allow-empty --clear-mountpoint --rollback --destroy-missing=2d --zfs-compressed --ssh-config /mnt/tank/Scripts/Keys/config --ssh-target backup backup Backup/Replicate

2 Likes
29

Just a followup on this post. I was referring to vanilla zfs commands.

You have to refer to the wrapper / tool documentation (such as “zfs_autobackup”) for the relevant options.

So for example, in vanilla zfs you’d use the -c flag on the send side, whereas for zfs_autobackup, you’d use the --zfs-compressed option.

1 Like
30

What I like about autobackup (well, another like as I’ve posted some of them) is it does a separate zfs send/receive for each dataset. That makes things so much more flexible avoids problems with replication at the root level.

1 Like
31

Thanks a lot again for all replies!

The encryption settings in TrueNAS Scale GUI for Replication Tasks are indeed different from TrueNAS Core (I’m upgrading my main server from Core to Scale, so I saw and worked with both)

Thanks for the command line options for zfs send and zfs recv. I am aware that similar but different options exist for zfs-autobackup.
If I can get zfs-autobackup to do what I want, I’ll probably use that instead of my own script.

I know about the on-init/shutdown script options in the GUI yes and might use them in the future to automate things a bit more.

I know that zfs-autobackup is widely used in the ZFS community, but I couldn’t really find much evidence that it is also widely used for TrueNAS (your extremely useful instruction-post has 0 replies and I couldn’t find many people talk about it, except you :stuck_out_tongue: ).
That is why I still wonder how safe it really is to do zfs create and zfs destroy on the command line (or via the zfs-autobackup script) in TrueNAS and not use the GUI. As explained, perhaps the state of the filesystem is then out of sync with what TrueNAS expects it to be?
But since you have been using it successfully for quite some time and since no one so far has answered this question negatively, I think it might be safe to try?

Thanks for providing me your zfs-autobackup settings! I went through all documentation on github in meantime and composed a similar command:

cd /mnt/backup-pool/homedir-ds/home/root/bin/zfs_autobackup-3.2.2 \
  && autobackup-venv/bin/python -m zfs_autobackup.ZfsAutobackup --verbose --test --debug \
  --keep-source=0 --keep-target=0 \
  --clear-mountpoint --clear-refreservation --exclude-received \
  --strip-path 2 \
  --rollback --zfs-compressed \
  --ssh-config /mnt/backup-pool/homedir-ds/home/root/.ssh/config --ssh-target root@192.168.10.10 \
  backupondemand master-pool/encrypted-ds

I still need to try it though and finetune my keep-source and keep-target options…

32

Ok I’ve just tried to run zfs autobackup without --test for the first time and I’m afraid I immediately run into a couple issues…

First of all, also zfs autobackup cannot handle that the target dataset already exists. It doesn’t automatically remove and recreate the pre-existing dataset, even not when using --force.

root@truenas-backup:/mnt/backup-pool/homedir-ds/home/root/bin/zfs_autobackup-3.2.2# cd /mnt/backup-pool/homedir-ds/home/root/bin/zfs_autobackup-3.2.2   && autobackup-venv/bin/python -m zfs_autobackup.ZfsAutobackup --verbose --debug   --keep-source=0 --keep-target=0   --clear-mountpoint --clear-refreservation --exclude-received   --strip-path 2   --rollback --zfs-compressed   --ssh-config /mnt/backup-pool/homedir-ds/home/root/.ssh/config --ssh-target root@192.168.10.10   backupondemand master-pool/encrypted-ds
...
# [Target] CMD    > (zfs send -L -e -c --raw -v -P -p backup-pool/encrypted-ds/shared-ds@backupondemand-20240511215655) | (ssh -F /mnt/backup-pool/homedir-ds/home/root/.ssh/config root@192.168.10.10 'zfs recv -u -x refreservation -o canmount=noauto -v -s master-pool/encrypted-ds/shared-ds')
# [Source] STDERR > full        backup-pool/encrypted-ds/shared-ds@backupondemand-20240511215655        42786371016
! [Target] STDERR > cannot receive new filesystem stream: destination 'master-pool/encrypted-ds/shared-ds' exists
! [Target] STDERR > must specify -F to overwrite it
# [Source] STDERR > size        42786371016
! [Source] Command "zfs send -L -e -c --raw -v -P -p backup-pool/encrypted-ds/shared-ds@backupondemand-20240511215655" returned exit code 141 (valid codes: [0])
! [Target] Command "ssh -F /mnt/backup-pool/homedir-ds/home/root/.ssh/config root@192.168.10.10 'zfs recv -u -x refreservation -o canmount=noauto -v -s master-pool/encrypted-ds/shared-ds'" returned exit code 1 (valid codes: [0])
! [Source] backup-pool/encrypted-ds/shared-ds: FAILED: Last command returned error
  Debug mode, aborting on first error
! Exception: Last command returned error
Traceback (most recent call last):
...

root@truenas-backup:/mnt/backup-pool/homedir-ds/home/root/bin/zfs_autobackup-3.2.2# cd /mnt/backup-pool/homedir-ds/home/root/bin/zfs_autobackup-3.2.2   && autobackup-venv/bin/python -m zfs_autobackup.ZfsAutobackup --verbose --debug   --force --keep-source=0 --keep-target=0   --clear-mountpoint --clear-refreservation --exclude-received   --strip-path 2   --rollback --zfs-compressed   --ssh-config /mnt/backup-pool/homedir-ds/home/root/.ssh/config --ssh-target root@192.168.10.10   backupondemand master-pool/encrypted-ds
...
# [Source] STDERR > full        backup-pool/encrypted-ds/shared-ds@backupondemand-20240511215655        42786371016
! [Target] STDERR > cannot receive new filesystem stream: zfs receive -F cannot be used to destroy an encrypted filesystem or overwrite an unencrypted one with an encrypted one
# [Source] STDERR > size        42786371016
! [Source] Command "zfs send -L -e -c --raw -v -P -p backup-pool/encrypted-ds/shared-ds@backupondemand-20240511215655" returned exit code 141 (valid codes: [0])
! [Target] Command "ssh -F /mnt/backup-pool/homedir-ds/home/root/.ssh/config root@192.168.10.10 'zfs recv -u -x refreservation -o canmount=noauto -v -F -s master-pool/encrypted-ds/shared-ds'" returned exit code 1 (valid codes: [0])
! [Source] backup-pool/encrypted-ds/shared-ds: FAILED: Last command returned error
  Debug mode, aborting on first error
! Exception: Last command returned error
Traceback (most recent call last):
...

I’ve then removed the dataset from the TrueNAS GUI myself. There I noticed that the TrueNAS GUI also automatically removes my share.

After removing the dataset, the zfs autobackup finally works and copies my data. Also my --keep-source and --keep-target options seem to work like I want them to.

However, this command apparently handles my encryption completely wrong… It does it like this:

(source) backup-pool - encrypted-ds(encrypted) - shared-ds(encryption inherited from parent)
→
(target) master-pool - encrypted-ds(encrypted) - shared-ds(own encryption copied from the source)

It should inherit the encryption from the parent, also on the target…

(Sorry for the confusing backup going from backup-pool → master-pool , but as I’m currently recreating the master, my backup is currrently acting as master, so yes, this is confusing, but exceptionally correct)

After this failure, I’ve tried to use

  --filter-properties PROPERTY,...
                        List of properties to "filter" when receiving filesystems. (you can still restore them with zfs inherit -S)

But also that fails miserably with the error “cannot receive new filesystem stream: encryption property ‘encryption’ cannot be set or excluded for raw streams.” :frowning:

root@truenas-backup:/mnt/backup-pool/homedir-ds/home/root/bin/zfs_autobackup-3.2.2# cd /mnt/backup-pool/homedir-ds/home/root/bin/zfs_autobackup-3.2.2   && autobackup-venv/bin/python -m zfs_autobackup.ZfsAutobackup --verbose --debug --filter-properties encryption  --keep-source=0 --keep-target=0   --clear-mountpoint --clear-refreservation --exclude-received   --strip-path 2   --rollback --zfs-compressed   --ssh-config /mnt/backup-pool/homedir-ds/home/root/.ssh/config --ssh-target root@192.168.10.10   backupondemand master-pool/encrypted-ds
...
# [Target] CMD    > (zfs send -L -e -c --raw -v -P -p backup-pool/encrypted-ds/shared-ds@backupondemand-20240511215655) | (ssh -F /mnt/backup-pool/homedir-ds/home/root/.ssh/config root@192.168.10.10 'zfs recv -u -x encryption -x refreservation -o canmount=noauto -v -s master-pool/encrypted-ds/shared-ds')
# [Source] STDERR > full        backup-pool/encrypted-ds/shared-ds@backupondemand-20240511215655        42786371016
! [Target] STDERR > cannot receive new filesystem stream: encryption property 'encryption' cannot be set or excluded for raw streams.
# [Source] STDERR > size        42786371016
! [Source] Command "zfs send -L -e -c --raw -v -P -p backup-pool/encrypted-ds/shared-ds@backupondemand-20240511215655" returned exit code 141 (valid codes: [0])
! [Target] Command "ssh -F /mnt/backup-pool/homedir-ds/home/root/.ssh/config root@192.168.10.10 'zfs recv -u -x encryption -x refreservation -o canmount=noauto -v -s master-pool/encrypted-ds/shared-ds'" returned exit code 1 (valid codes: [0])
! [Source] backup-pool/encrypted-ds/shared-ds: FAILED: Last command returned error
  Debug mode, aborting on first error
! Exception: Last command returned error
Traceback (most recent call last):
...

I’ve tried looking for how to disable this “raw streams”, but I cannot find any option for this. Does anyone have an idea how to make this work?

Edit 1:
With my own script I can make it work…
zfs autobackup does it like this:
(zfs send -L -e -c --raw -v -P -p backup-pool/encrypted-ds/shared-ds@backupondemand-20240511215655) | (ssh -F /mnt/backup-pool/homedir-ds/home/root/.ssh/config root@192.168.10.10 'zfs recv -u -x encryption -x refreservation -o canmount=noauto -v -s master-pool/encrypted-ds/shared-ds')

If I drop the zfs-send ‘-p’ and ‘–raw’ then it does work…

${SOURCE_SSH} zfs send -v -Lec -P "${NEW_SNAPSHOT_NAME}" | ${TARGET_SSH} zfs recv -x encryption -x refreservation -o canmount=noauto -v -s "${TARGET_POOL}/${DATASET_TO_BACKUP}"

But I have no clue how to do that with zfs-autobackup :frowning: (and that just when zfatula convinced me to use zfs-autobackup instead of my own script :wink: )

edit 2:
I find it very confusing, but apparently the --decrypt option of zfs-autobackup does the trick… Even without using ‘-x encryption’ it then copies over the data and inherits the encryption from the parent on the target…

Now I still need to figure out how to make it properly mount the dataset (it’s unmounted and TrueNAS GUI (and me) doesn’t appreciate this at all… (error)

33

I do get following TrueNAS GUI error after creating a dataset with zfs-autobackup with a full replication:

image
image700×209 5.3 KB

 Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 76, in get_quota
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 529, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 78, in get_quota
    quotas = resource.userspace(quota_props)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "libzfs.pyx", line 3651, in libzfs.ZFSResource.userspace
libzfs.ZFSException: cannot get used/quota for master-pool/encrypted-ds/shared-ds: unsupported version or feature

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 80, in get_quota
    raise CallError(f'Failed retreiving {quota_type} quotas for {ds}')
middlewared.service_exception.CallError: [EFAULT] Failed retreiving USER quotas for master-pool/encrypted-ds/shared-ds
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1417, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_quota_and_perms.py", line 225, in get_quota
    quota_list = await self.middleware.call(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1425, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1431, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1337, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed retreiving USER quotas for master-pool/encrypted-ds/shared-ds

Any ideas on how to resolve this?

34

Ok I think I finally figured out why I was getting the user and group quota errors…

It was because my backed-up datasets were not mounted after a reboot :cry:
This is apparently because of the highly-recommended zfs-autobackup --clear-mountpoint which sets the zfs property canmount to noauto, causing the dataset not to be mounted at boot.

Because zfs-autobackup warns of the danger of not doing this, I blindly followed this advice. But as it actually breaks TrueNAS, I will now stop doing that and set canmount to on instead (as it is on the source).

After investigating why it could be dangerous to do this, this seems related to backing up datasets in various ‘levels’. For example, if you backup dataset-depth1 and also dataset-level1/dataset-level2/dataset-level3 you are in danger of unintentionally modifying dataset-level1/dataset-level2.

Since I will only be modifying 1 dataset level (level3 that is), I think this risk does not apply to me (but please do correct me if I’m wrong!)

edit:
I’ve just done another incremental backup and I’m again getting the same errors :cry:
Even though the datasets are still mounted and canmount is still set to on

A reboot does seem to solve it :confused: (probably only until the next backup)

1 Like
35

Whether or not a ZFS dataset is mounted does not matter when it comes to replicating snapshots.

It’s odd that the “noauto mount” would make a difference.

1 Like
36

The problem is not that the replication doesn’t work. The problem is that after the full replication (which creates a new dataset - not using the GUI - on the target side) the TrueNAS GUI has issues with this new dataset (actually exactly as I predicted) and has trouble with obtaining the user and group quota (although that is probably just a consequence and not the root-issue).

At first, I guess, this was because the filesystem wasn’t mounted after boot (canmoun=noauto). But also when setting the property canmoun=on during a new incremental replication, TrueNAS has issues obtaining the user and group quotas (even though the filesystem seems correctly mounted by zfs-autobackup).

After a reboot (and TrueNAS itself mounting the dataset), then it does work…

It actually seems to me that TrueNAS needs to do the mounting itself for it to properly work.
Manually running ‘zfs mount’ or letting zfs-autobackup do the mounting seems to cause this problem…

37

Below a list of all errors that I’m getting on the target side in the TrueNAS GUI after replicating zfs snapshots with zfs-autobackup of a dataset that is mounted on both source and target server.

I suspect this is because zfs-autobackup doesn’t “properly” mount the target dataset, because after a reboot (where TrueNAS does the mounting itself), there are no errors…

[EFAULT] Failed retreiving USER quotas for master-pool/encrypted-ds/test-ds

 Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 76, in get_quota
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 529, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 78, in get_quota
    quotas = resource.userspace(quota_props)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "libzfs.pyx", line 3651, in libzfs.ZFSResource.userspace
libzfs.ZFSException: cannot get used/quota for master-pool/encrypted-ds/test-ds: unsupported version or feature

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 80, in get_quota
    raise CallError(f'Failed retreiving {quota_type} quotas for {ds}')
middlewared.service_exception.CallError: [EFAULT] Failed retreiving USER quotas for master-pool/encrypted-ds/test-ds
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1417, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_quota_and_perms.py", line 225, in get_quota
    quota_list = await self.middleware.call(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1425, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1431, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1337, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed retreiving USER quotas for master-pool/encrypted-ds/test-ds

[EFAULT] Failed retreiving GROUP quotas for master-pool/encrypted-ds/test-ds

 Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 76, in get_quota
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 529, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 78, in get_quota
    quotas = resource.userspace(quota_props)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "libzfs.pyx", line 3651, in libzfs.ZFSResource.userspace
libzfs.ZFSException: cannot get used/quota for master-pool/encrypted-ds/test-ds: dataset is busy

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 80, in get_quota
    raise CallError(f'Failed retreiving {quota_type} quotas for {ds}')
middlewared.service_exception.CallError: [EFAULT] Failed retreiving GROUP quotas for master-pool/encrypted-ds/test-ds
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1417, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_quota_and_perms.py", line 225, in get_quota
    quota_list = await self.middleware.call(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1425, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1431, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1337, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed retreiving GROUP quotas for master-pool/encrypted-ds/test-ds

[ENOENT] Path /mnt/master-pool/encrypted-ds/test-ds not found

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1428, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 191, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 53, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/filesystem.py", line 422, in stat
    raise CallError(f'Path {_path} not found', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Path /mnt/master-pool/encrypted-ds/test-ds not found

Is there perhaps a way to force TrueNAS to remount a specific (or all) datasets from the command line?

38

I think I figured out how to resolve the above issues :partying_face:

By running zfs_autobackup with --debug and then running all zfs commands manually, I was able to “debug” the issue.
The problems occur after performing an incremental replication over a mounted dataset.

The solution is to simply unmount and remount the dataset.

I’ve raised below issue on the zfs_autobackup github (in case they would like to be compatible with TrueNAS)

This also contains some more details on this problem…

1 Like
39

If you’re referring to the destination, then they should preferably not be mounted, let alone “used”, if they are to receive replications.

See flowchart #4 in this post.

1 Like