Storj Cloud Sync Task does not work recursively?

Hi everyone,

I started to send some backups from truenas scale to Storj with the integration from truenas itself. All is fine until I noticed that one of the folders I want to copy to storj has files (which are copied successfully) but the subfolders are not.
Please someone knows if there is a way to recursively push all the files and subfolders of the specified path?

Thanks to anyone

Which version of TrueNAS?

Are you using Cloud sync or Cloud backup?

@Captain_Morgan Sorry, I have to add my configuration in the signature sooner or later :smiley: Anyway I am on Truenas scale on EE RC2. I am using Cloud Sync at the moment (even though, I did not think about using Truecloud and it is also confusing on whether to use one or the other?). For the moment I have setup 2 cloud sync tasks and one is doing things correctly while the other is just copying the files in the first level of directory but nothing else (even though it has subfolders and files into these folders too)

By default Cloud Sync is going to traverse directories. However do you have nested datasets in there? It may not be crossing mount points, which is what nested datasets will present as.

Hi @kris , actually the structure looks like this:

MyPool
    appdata
        vaultwarden
        vaultwarden-postgres

where appdata is the parent dataset and vaultwarden and vaultwarden-postgres are children datasets. I basically setup a cloud sync for each one, so vaultwarden has its own task and vaultwarden-postgres has the other one, but while vaultwarden-postgres syncs files and folders correctly (with subfolder etc) vaultwarden directory does not.

What’s different in settings of vaultwarden and vaultwarden-postgres?

Cloudsync tasks, permissions, ???

If you can’t see anything, can you test with another dataset…

I’ve made 2 different cloudsync tasks for each of the datasets, so it means vaultwarden has 1 task and vaultwarden-postgres has another.
Same settings between the 2 were applied (apart of the cronjob definition to avoid uploading all together)

Regarding the rights on these datasets, vaultwarden has root as userId while vaultwarden-postgres has user 70, both group and other have 0 rights (because postgres:alpine uses this user (and I can’t modify this settings).
To be honest I am surprised that truenas doesn’t have any problem with the dataset vaultwarden-postgres (because of the rights look like) while vaultwarden which has user as root and group as apps (both with RWE rights) doesn’t push all the subfolders as expected

Are you continuing to use “root” rather than “admin” or something else?

From our AI bot:

It is recommended to use the admin user as the primary data user in TrueNAS SCALE. The root user password is disabled by default to comply with FIPS security hardening standards, and it is advised to create and use an admin user account for system administration tasks. This approach enhances security by limiting the use of the root account, which should only be activated for limited uses and then disabled again.

For more information, you can refer to the Logging Into SCALE the First Time documentation.

Ah I didn’t know about the admin user and that it should be applied by default. I can try to apply of course the admin user and retry with the sync

It may or may not be the issue, but we don’t want to waste your time if it is. Better to operate in tested and recommended territory.

No problem at all :slight_smile: Most of the times are man-made mistakes. Anyway now I’ve updated the rights on both datasets.



But same behaviour as before applies. I delete the folders from Storj, and started both tasks. Vaultwarden-postgres has pushed all the files while vaultwarden sent 1 file.
Now I understood what happens!!! Basically the cloudsync task just pushes folders that contain at least 1 file otherwise these folders are not even pushed! Isn’t this a little odd, or at least confusing? To be honest, I would prefer to be a 1:1 copy of the structure of folder I want to push. There are cases (especially with apps) where maybe the application looks for the presence of a specific directory and just crashes if it doesn’t find it (even though it is empty)

1 Like

Cloudsync uses rclone… you might want to check if rclone is changing this or has an option for it.

I just looked at rclone documentation and I enabled this checkbox:


this SHOULD copy the empty directories even though it depends IF the object storage server supports the creation of empty directories.

https://forum.rclone.org/t/solved-why-does-sync-leave-the-empty-folders/13123/6

Last post on the topic I’ve added above:

Great - glad you got it sorted. Rclone’s handling of empty directories is a little eccentric due to the fact it works on systems like s3 which can’t have empty directories…

If someone using Storj as object storage can confirm this behaviour I will close this topic