I’m running Scale 25.04.2, build spec is in the signature but this is more of a general question. If I’ve got lots of data nice and safe on my server, what’s to stop something like Syncthing or NextCloud updating a file if it get’s corrupted on another device. I’m about to setup Syncthing to keep a folder two way synced to an SDCard in my phone but should i be worried about that eventually syncing some garbage back up to my TrueNAS server?
Nothing, really. Snapshots will help, if you catch the corruption in time.
Yup, sync is just like RAID1.
It doesn’t protect against corruption, you just have multiple copies of your corrupt data in case one goes away for some reason.
Backups/snapshots are needed in addition to replication so you can go back to a version of the file(s) from before they were corrupted.
If the data is mission critical, I suggest you centralize the storage on the main pool so the data is known-good as long as it’s at rest. That way Syncthing cannot corrupt the good stuff with bad stuff.
What happens locally, in transfer, etc. is a different matter but here snapshots can help a lot. The longer they go into the past, the easier it is to recover an old file.
If local storage is important, I wonder if you can implement something like time machine instead, i.e. give your users the ability to go back to a uncorrupted file on their own machine. But that still assumes they detect the corruption before uploading to the central server.
The key to all the above is recognizing / detecting when something got corrupted as early as possible. Ultimately, even offsite / offline data will get eventually corrupted. That’s why the paradigm of keeping everything you care about on ZFS all the time is so powerful. It covers a lot of bad scenarios.
Sigh, I was hoping this wasn’t the case. I think I’ll set my phone as a receive sync only. My laptop should be ok, got two drives in raid 1 on that. Next job, make sure I’m saving snapshots to my offsite. I really enjoyed my ignorance before got in to self hosting. Thanks for all the replies.
1 Like
It’s a testament to the folk that write drivers, protocols, stacks, OS’, etc. that most of the time, corruption doesn’t happen despite the vast universe of hardware, etc. iterations out there. That in turn can make us all a bit lazy re: backups and file integrity in particular.
It may explain why checksumming on a wider basis in APFS and elsewhere still is not a reality. IIRC Apple got as far as checksumming and COW the metadata, but no more.
Since Apple had access to ZFS back when they made a choice between writing a new FS vs. using ZFS, I presume they chose APFS because they wanted something written specifically for flash storage and Apple, not a generalized solution.
But I wasn’t there.
Anyhow, it’s one reason we see such emphasis on server-grade hardware here, including ECC RAM. Practically no SOHO computers contain it, ditto their NICs, and other peripheries.