Cloud Sync Task black hole

TrueNAS General
1

I am trying to backup my data to my own server (using SFTP) using a Cloud Sync task.

However I am having a problem in that the task never stops running, it just keeps creating more data, even though that data doesn’t exist.

This is an image of the task setup:

image
image945×1034 48 KB

The idea behind this setup was to basically just copy all data to the cloud once, then stop.

The data being uploaded was 619GB, seen here in Windows explorer.

[image removed because I am a new user and can only upload one per post… I’m sorry… What?]

However, after running for 6 days I eventually had to cancel it, because at this point it had uploaded 1.9TB of data, and it became clear to me it was never going to stop.

I ran “tree NASBackup290324/ -s” on my server and got this result: 6364 directories, 78670 files
…That’s 50k files and 6k folders LESS than what windows is reporting, and yet it’s taking up WAY more disk space than it should be.

I ran “du -hs NASBackup290324” and got this result: 1.9T NASBackup290324
…which corresponds happily to the TrueNAS Cloud Sync task status just before I cancelled it on the 4th April.

[image removed because I am a new user and can only upload one per post… I’m sorry… What?]

I have no idea how this is possible, and I am at my wits end with it at this point. If I can’t figure out how to do something as basic as sync my data to the cloud I might have to ditch TrueNAS entirely, because I need to start taking proper backups of my data.

Broadly speaking, looking at the data on the remote server as a mapped drive in Windows, it looks like my files are there, however there are too many files to check everything manually and my commands tell me that there are files missing, and there is a 1.2TB black hole, as this 619GB of data continues to take up 1.9TB on the remote server.

Has anyone else ever seen this behaviour before or can shed some light on what to do about it?

2

I’ll review this later in depth. Check out your mode COPY, you may want to SYNC.

3

Thanks, appreciate it! I was thinking about using SYNC, but in the end I chose COPY because I wasn’t sure about how I wanted to set SYNC up yet (in terms of scheduling, and perhaps I would have certain folders sync more often than others etc) so I figured I would just upload an entire copy of the data first, and then figure out those details soon.

4

How about you compare the modify times of a file in both locations, something is making rclone push the same data over and over again

5

Surely because I’m using SYNC and not COPY, it shouldn’t be comparing modify times though?

Even if it was though, that wouldn’t explain it. For context I am a home user and the only person who uses my NAS is me. During this time I probably made and edited a few dozen small files at most. If this was going to cause an issue then maybe a few MB, or even a GB or 2 extra space taken up might be understandable, but 1.2TB is another story

Edit: I just checked anyway with several files from random locations, and the date/times are the same.

6

If du is consistent with your backup task, what does the dataset size say in the true as GUI?

Can you check the size you’re seeing in windows with something like total commander? ALT + SHIFt + ENTER to see the size of a dir when you’re in it.

7

du isn’t consistent with my backup task.

Here is the windows properties of the folders that were involved in the Sync Task:
(this is one of the images I had to remove from my original post because of this new forums only allowing me to post 1 image in my post)

image

and du on the server says 1.9TB is being taken up. This is also reflected on my server providers control panel (by the way there was nothing on the server prior to me running this task)

This data isn’t really there though, at least from what I can tell… I don’t see all this extra data it has created, more than triple the size of the data that should have been uploaded. It’s not like I am seeing duplicates. I have the server storage mapped as a drive in Windows and from what I can tell, it looks how I would expect.

8

What does total commander tell you?

What is the size of the dataset in the truenas GUI?

Why is du not consistent if it tells you you used 1.9 TB and 1.9 TB were backed up?

If COPY instead of SYNC isn’t producing an overhead (a lot of deleted files for example) then you may not really know how much data you have saved. I would guess windows isn’t telling you the whole truth.

Give me some screenshots, something I can work with. I’m fishing in the dark here.

9

I haven’t got any more time to mess around with this issue so I just backed up the data to a USB hard drive for now. I’m planning to upgrade to SCALE, after which time I’ll revisit this cloud storage angle.

I really appreciate your assistance regardless though :slight_smile: thank you.

10

I recall a similar thread on the old forum… but not much else.