I have about 700TB on Dropbox with a significant number of files.
It is mainly used for backups of both PCs and websites.
Programs make backups and delete older copies.
On the PC, Dropbox is launched manually only when needed and after about ten minutes I have a replica of everything I have in the cloud. Usually the variation is a few thousand files and around 5GB.
Even on TrueNAS SCALE I launch the Cloud Sync Task by hand, but it takes hours. For example, this morning it was started at 8:30 and at 18:00 although the percentage is 100% from 13:00 it is still “Running”.
I can imagine it’s checking every single file there against those in Dropbox, but that’s still too much time.
This is a Sync Pull with 100Mb/sec in download.
If it were a Sync Push, having only 10Mb/s in upload it would be impractical to think of managing a backup.
TrueNAS SCALE Dragonfish-24.04.0
Backup on one only disk 2.63TB. Used Space 759GB.
How are you able to achieve this? I store only around 10TB and roughly 1 million files and it takes probably around 15min.
Dropbox either has a pretty strong rate limiter or just poor performance when it comes to API calls. It takes around 10h for me to run a single sync. But hey, at least API calls are free, unlike other providers.
If you need better performance, S3 providers like Backblaze are probably better suited. But of course they cost a lot more money. I suspect you have an old Dropbox account that will soon be downgraded to 5TB per user?
I’m sorry, I made a mistake.
I have a 2TB “Professional” account in which I saved 700GB (and not 700TB).
Having a paid account I expect good performance.
In any case there is an unacceptable difference between 10 minutes on the PC and 15 hours on TrueNAS. Now it’s 11pm and it’s still running, 100% from 1pm.
Not gonna happen. I wrote multiple times with dropbox support.
I could not believe how a paid pro account could be that slow.
I mean we pay thousands of dollars every year to them.
What I found out after using Dropbox intensively for years:
Pretty high CPU consumption. Probably due to the internal database and compressing every upload
Slow speeds even on a 10Gbit line. 50mbit max. Yesterday I had to restore a 200GB folder that was deleted accidentally. Took around 2 hours.
You need a SSD. Dropbox does some pretty intensive DB rw if you have a lot of files
You need a SSD with write endurance. Even though the data itself is on D: on a HDD, files will be cached in AppData on C:. Nothing really gets deleted, lots of temp files and chunks.
API calls are free but very slow
Dropbox does not only charge you for storage but also people you share files with. That makes it basically unusable in a business environment, if you share projects bigger than 2GB with externals.
Dropbox does everything to save on bandwidth. Lan Sync, compression, not deleting, you name it. You as a customer will experience some downsides because of it.
You have to live with it or switch your cloud drive provider.
But even with fast API calls, there will always be a difference. I don’t know how the Dropbox App works, but if a file changes, I suspect that there will some hash that changes and it will change only that file. For API calls (I guess S3?), it will basically ask for every single file, compare that with what is stored on the system. That has to take way longer.
We basically use it to backup Dropbox once a week. If it takes a day to do so, that is fine by us. For local data access, we use a Windows Client that has the app and shares the Dropbox folder over smb. It is a wonky solution but it works for our use case. But soon we will migrate to something more performant.
Having to update only 5GB doesn’t justify a day of work, even imagining very slow APIs.
From what I knew, Dropbox checks the folder first and if it is not modified it does not check the files. I believe he uses an identifier for this purpose.
If TrueNAS checks individual files, the times are longer; especially, as in my case, if the Upload speed is a tenth of the Download speed.
This morning he completed the update in about half an hour; still too much, but acceptable. Now, however, it is always “Running” even if at 100%, probably because it checks every single file.
Dropbox documents how to install it on a Debian server and control it from the command line.
I had it installed on my server and it worked great.
I tried to install it on a TrueNAS Scale that I use for testing and it didn’t work.
My guess is, you aren’t just updating 5GB files, you are doing API calls to for every single file to check if there was a change.
That would make sense. With a merkle tree hash, it wouldn’t need to go check every file. But I don’t know how they implemented it.
That isn’t the case for me. You sure it already finished? I would not read too much into these percentage numbers, they can be pretty flawed in TrueNAS.
Awesome! Didn’t know that, seems to be a recent change. Thanks for the hint!
The calls to the API are certainly not made by me, but by TrueNAS and as for the percentages I only have that information.
It’s TrueNAS that manages Dropbox and there’s very little you can do about it.
Furthermore, it only works in one direction, which is contrary to Dropbox’s logic. You should first do a Pull update and then a Push update (or vice versa), but in the meantime between the two updates there may be other changed files; therefore, I find it a dangerous action because there is a risk that files will be removed for no reason.
Dropbox for servers is documented on their site on this page.
I used it two or three years ago.
I had created a user with its home folder in a partition with enough space.
Once the files have been downloaded and the installation has been launched, the token to be used in the browser to authenticate the new session is then displayed.
From that moment just launch their python script with the commands to launch updates, add new folders, report which folders to avoid synchronizing, etc.
I then configured a service to automatically launch it as “root” at certain times.
I couldn’t do this again on TrueNAS SCALE. When I have some time I’ll do some new tests in a virtual machine on my PC.
If you can install it, please let me know.
Sure. But that information is not very reliable. I just looked at my Dropbox Cloud sync task.
It immediately jumps to 100%. That is because it thinks it has synced 2 out of 2mb while it only has checked 2000 files yet. It will stay at 100% forever.
If the task is running (blue arrows) the sync is not finished. If the task has finished (green check) the sync is finished.
Exactly. Because it is not the same logic as Dropbox.
It is S3 logic.
Sync means, files on the destination are changed to match thoso on the source. If deletes on source it will be deleted on destination.
Dropbox like sync is not possible, since it has no way of knowing which files it should keep. So yes, the Dropbox like sync you can’t achive with Cloud Sync Tasks!
I am not are dare devil and would never risk that
Sure SCALE might be Debian based, but since Dropbox officially only supports Ubuntu, Debian, Fedore, I would never install it on TrueNAS.
Dropbox support is bad enough or none existent as it is! No way I am getting into this mess.
My guess is, the probably best option would be to spin up a VM (no big fan of TrueNAS has hypervisor, but YMMV) and use TrueNAS NFS or iSCSI storage as storage for that VM.
But then the simplicity of Dropbox goes out of the windows in my opinion and I will ditch Dropbox completely for Nextcloud. That way I also can work with external users without that pesky 2GB limit Dropbox enforces, even though I already paid for that storage.
I don’t use Dropbox to share files, but as a backup solution.
For €120 a year I have 2TB at my disposal from any PC, Win or Linux, and 30 days to restore deleted files.
But I realize that it is not the valid solution for TrueNAS.
Do you have an alternative solution to suggest that works for all PCs, servers and TrueNAS? Both Windows and Linux?
Great. That makes this even simpler, doesn’t it? You just run a Cloud Sync Task that either pushes or sync to Dropbox (depending on what you want to achieve).
Why not? TrueNAS will run 24/7 anyway, why bother with the speed? If it can finish under the time of your backup schedule (daily?), what is not to like?
Well, depends on your needs. Basically there are other Cloud Drives you will run into the same “issue”, or you could get S3 storage from providers like Backblaze, Wasabi and so on. But you would need an S3 compatible software like FileZilla to access it on Win or Linux.
Why not? For all the reasons I have stated here.
An update of around 5GB lasts over 15 hours with Pull at 100MB/s.
If it were a Push at 10Mb/s I imagine it would take 150 hours.
Even if it were still 15 hours, in that time new files will have been saved or modified, which also means that after 15 hours the backup will also be incomplete.
And we’re talking about just 5GB.
I didn’t open the post to complain, but to check if it was my problem that could be fixed, but from what we’ve said so far it’s a generalized problem and will remain so if no one at TrueNAS gets involved.
Then I watch the video and study the documentation better, but an experimental and unsupported feature could be risky for a backup.
I should in any case assign a space of 500GB / 1TB, can this be done?
I doubt you can scale it like that. I believe, that the backup takes as long as it takes because of how many files it has to check. I would argue that 5GB changes or 100GB changes is almost no difference on a decent fiber line. We are able to sync around 10TB and million of files in 12h. That is on two HDDs in mirror.
Nothing bad about that. That is just how backups of a running system are. Unless you stop a system, you will always have that “problem”.
That is assuming that TrueNAS can somehow speed up this process. I highly doubt it. I doubt it because I am 99% sure the bottleneck here is dog slow Dropbox servers or your pool and not TrueNAS code.
TrueNAS can’t help you with that.
You can use snapshots. If you backup the snapshot, rather than a live dataset you essentially get a frozen point in time.
Meanwhile, if your destination is zfs you can replicate. This is super fast, and deltas are optimally transferred.
This is a primary reason why I moved all my backups to zfs replications as the rsync process spent longer figuring out what to copy, rather than copying.
Exactly what we do. Weekly cloud sync, one day later (just to be sure Cloud Sync has finished), we do a Snapshot that is stored for 10y. We have 1y rewind with Dropbox, this is just in case Dropbox messes things up and to have some kind of data retention. Sure a Veeam backup would be nicer, but for a small business this is a pretty good solution in my opinion.
Exactly right. This is especially true if the data being reviewed consists of bazillions of little files as has become common in OS distributions. One way to combat this plague is to consolidate as much as possible into archives / sparsebundles / etc. That makes rsync execute considerably faster. But the fastest way to transfer the data is and benefit from potential rollbacks is replication.
By the way, if anyone here now thinks about using OneDrive Business instead, don’t!
This thread gave me the idea to use my unused OneDrive Business account as an additional cloud backup for my 600GB hot data.
At first it ran at 450MBit/s. Pretty impressive.
Then I come back to see if it went through and what do I see?
500KBit/s.
Yes, I really mean KBit/s like the old ADSL speeds.
So yeah, I ran into some kind of rate limiter. That is just what happens if you don’t pay for S3 storage.
