CloudSync fails with corrupted transfer

TrueNAS General
1

I have set up a CloudSync Task to back up my NAS to an S3 (IDrive e2) cloud storage service. This is just a test system with only 12.3G of data. After 16 hours of the task running it fails with the following fault. AFAICT the task is running still (again?) and is now at 0% completion. Any ideas?

[EFAULT] Checks: 2 / 2, 100% Transferred: 1 / 1, 100% Elapsed time: 15h51m59.8s Failed to copy with 211476 errors: last error was: corrupted on transfer: md5 hash differ “766f3a88e0dde9b56fc2f67bc9341e84” vs “7b1b28789297c989af738b71c917429f”

More info:

__run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 187, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/cloud_sync.py”, line 1048, in sync
await self._sync(cloud_sync, options, job)
File “/usr/lib/python3/dist-packages/middlewared/plugins/cloud_sync.py”, line 1102, in _sync
await rclone(self.middleware, job, cloud_sync, options[“dry_run”])
File “/usr/lib/python3/dist-packages/middlewared/plugins/cloud_sync.py”, line 246, in rclone
raise CallError(message)
middlewared.service_exception.CallError: [EFAULT] Checks: 2 / 2, 100%
Transferred: 1 / 1, 100%
Elapsed time: 15h51m59.8s

Failed to copy with 211476 errors: last error was: corrupted on transfer: md5 hash differ “766f3a88e0dde9b56fc2f67bc9341e84” vs “7b1b28789297c989af738b71c917429f”

2

Can you check with a validated S3 provider??

3

Reasonable suggestion. Unfortunately IDrive e2 is the only service I’ve paid for the storage space.

4

Which software version?
Try the latest and see if anyone else has a similar issue

5

In my opinion, this problem is with CloudSync. I believe IDrive E2 is using KMS-based encryption, as indicated in the object metadata. According to AWS, if the object is encrypted through KMS, the ETag returned by the S3 provider will not be an MD5 hash (as stated in AWS documentation). However, it seems like CloudSync always considers the ETag to be an MD5 hash.

Although this validation process needs to be changed by CloudSync to comply with AWS guidelines (sending the MD5 in the PUTObject request and letting the server validate if the correct contents are uploaded), for now, you can try using a bucket with encryption disabled. I have faced this issue previously and had to disable bucket encryption to make it work.

6

Yes, I think that this must be what is happening. For the original post I had both Versioning and Encryption enabled. I just tried again with a new bucket that only had versioning enabled. The backup worked quickly and without errors!