Hi,
I am setting up Cloud Sync jobs to backup 2 datasets to S3 buckets (provided by Mega S4 service). I am using encryption and filename encryption.
My broadband is slow (20Mbps upload) and ping times go to >1000ms when the transfer is running, so this could be a potential cause of my problems.
When the job is run for a 2nd time all the data is re-transferred.
Does this work correctly with other S3 providers or could it be a problem with the new Mega S4 service?
Alternatively, it could it be due to file modification times not being accurate enough. This can be resolved by an rclone setting which Truenas does not expose through the UI.
I tested without filename encryption and saw that an upload file test.avi appeared in the S3 storage as test.avi.bin. It seem like likely that this woudl cause issues!
I then tested with no encryption at all and found that the resync operation no longer uploads all the data again but it does updated the timestamp on every file. Is this expected?
To be clear, my preference is to use encryption
Any suggestions?
I have tested the same settings with an Amazon S3 account and everything works correctly. The problem here seems to be with the Mega S4 storage service. I will raise a ticket with them.
I have stopped using Mega S4 and moved to Hetzner Object Storage which works correctly with encryption enabled (including file name encryption) using the included settings of the TrueNAS cloud sync tasks.
Mega gave me the following reply as to why the features I was trying to use didn’t work and whether they are intending to fix it:
According to our developers, There are 3 possible solutions:
Use aws cli with the sync command. Which uses last modification date + file size to check if the object is the same or not.
Use --update --use-server-modtime, modification time becomes the time the object was uploaded y uploads files whose local modification time is newer than the time it was last uploaded. Files created with timestamps in the past will be missed by the sync.
Use --size-only only checks the size of files, if the file doesn’t change size then rclone won’t detect it has changed
rclone sync --size-only /path/to/source s3:bucket
About the technical explanation for this:
rclone sync in S3 compares file modification times (mtime), sizes, and MD5 hashes (for objects below the --s3-upload-cutoff value) between the source and destination to determine which files need updating.
Since S3 does not natively store mtime, rclone preserves it using user-defined object metadata (X-Amz-Meta-Mtime).
User-defined metadata allows users to attach custom key-value pairs to objects when uploading them.
These metadata entries are stored with the object but are not processed by S3, meaning they can only be retrieved when explicitly requested.
With option 2, the difference is that rclone compares the object’s upload time in S3 with the local file’s modification time (time), instead of relying on user-defined metadata (X-Amz-Meta-Mtime)
Currently, since this is not a core feature of S3, it is not supported by S4. However, it is included in our backlog, and we are considering its implementation in the future.
Facing similar issues with TrueNAS Core’s Cloud Sync Task re-copying all files to MS OneDrive Personal over and over again instead of skipping existing (unchanged) files. grumpycat
Still wokring on it, but I fear that I need to update rclone to 1.69.0 or 1.69.1 while I am using rclone 1.68.1 as of today.
See hxxps://rclone.org/changelog/ :
** Bug Fixes*
accounting*
Fix global error acounting (Benjamin Legrand)*
Fix debug printing when debug wasn’t set (Nick Craig-Wood)*
Fix race stopping/starting the stats counter (Nick Craig-Wood)*
rc/job: Use mutex for adding listeners thread safety (hayden.pan)*
Hallo, a bit offtopic. I am using AWS Cloud directly as for pure archive purposes it seems to be the cheapest option. Also it work quite nice and uncomplicated.
i agree its not the easiest to understand but the cheapest by far. here is an example of mine form last month. most costly is the 1 time bulk upload and in case of recovery the 1 time bulk downliad. incremental uploads are less expensive but the most important thing, the storage itself on deep_glacier is very cheap compared to others.
