Better make sure you only restore if you know your data exists in that snapshot otherwise proof it’s gone. Unlike traditional approaches that have incremental vs. differential backups you with ZFS can’t restore snapshots anywhere on the timeline arbitrarily.
The scenario here is restoring twice once early and another later chronologically. The snapshots/Clone names are 2019 and 2020 to represent chronological order.
A user restores 2019 and discovers that the data/application isn’t what they expected.
The user then tries to restore 2020
Whether the snapshot or clone 2020 and anything after 2019 becomes invalid in that data becomes lost.
Given the documentation as I understand it seems like anything intermediate clones/snapshots once a restore is done are invalidated.
If my understanding is correct, how do I protect myself from the scenario?
edit: Yes a replicated dataset would provide protection but seems a bit extreme.
A rollback is a rollback and not a restore. It rolls back the dataset in question to the point in time the snapshot was taken. Which implies all future (from the time the snapshot was taken) modifications will be deleted. Yes.
So your diagnosis is on the one hand correct, but then simply rolling back a snapshot is not the way to do a restore of accidentally deleted data.
You can read-only mount any snapshot, inspect the data you find, copy out any files you are missing, then unmount it again without performing a rollback!
You do not even need an explicit mount because it’s already there in the hidden .zfs subdirectory of your dataset:
freenas# cd /mnt/hdd/share/medien/.zfs/snapshot
freenas# ls -l
total 8257
drwxr-xr-x 8 nobody nogroup 8 Jun 14 2022 auto-2024-12-01_21-00
drwxr-xr-x 8 nobody nogroup 8 Jun 14 2022 auto-2024-12-01_22-00
drwxr-xr-x 8 nobody nogroup 8 Jun 14 2022 auto-2024-12-01_23-00
drwxr-xr-x 8 nobody nogroup 8 Jun 14 2022 auto-2024-12-02_00-00
[...]
if you cd to any of these snapshot directories you will find all the files present at the time the snapshot was taken. Plenty of opportunity to restore things you need.
So, yes, a rollback will do just that - roll back to a past point of time unconditionally and without preserving any later modifications. But a rollback is not a restore.
You’ve clarified for me the difference between a restore and a rollback. Rollbacks are mounted read-only snapshot for inspecting and restoring data. The key issue is the user needs to knows specifically what needs to be recovered. However that’s not always straightforward consider applications.
Considering the next scenario with the same set up. Simply the user may not know what is wrong with the application. The application could be a corrupt database, metadata, config or any other numerous point of failure. Consider the following scenario.
Two have (users A and B) 4000 in photos Immich
User B has been tagging a thousand photos over the course of a weeks
The application crashes and is unrecoverable. This goes unnoticed for two weeks.
How does the A user know when to restore when there are many bad snapshots of application not working? Assume they don’t have a deep knowledge of application (looking at logs, most people don’t).
It seems to me the user would have to use trial and error to with multiple restores of the application. However this comes at a great risk of losing data. In this case rollback isn’t helpful because the corrupt data isn’t obvious to the user. They don’t know if it application can run just by looking at files nor can they know how much of losing the metadata because it stored in a database.
So what’s the process of mitigating this issue?
What would be the best practice workflow in this kind of scenario for restoration of an application?
What kind of backup methodology provides the best protection against this?
I recognize some sort of tiered replication of the entire dataset would help but the restore process seems so complicated for very likely scenario with applications.
So the only thing I can think of the user will have to restore incrementally in chronological order to ensure they get the latest working snapshot for the application without data a loss.
This depends so much on the application - which I don’t use and so cannot advise you in any helpful way.
General things to consider:
Pick applications that store their data in some “raw” format, so for e.g. a photo management app there should be a single directory containing all the original picture files, paperless-ngx should (and does to my knowledge) have a single directory for all the original documents, etc.
Configure all storage for your app on TN SCALE as host paths, so you can snapshot and replicate these individually
The previous step also facilitates moving to a different platform altogether. Any app “X” that uses a file store and a database should be able to run if you restore the file store and the database. So prepare to have a way to get these outside of TN SCALE and the app.
Generally there is no easy way to restore anything in a multi-tier app like e.g. Nextcloud (which I use) with some simple “restore” button. Most modern applications have some storage for your data (photos in your case) and a database to store metadata about these files. How to perform a restore operation is particular to the application in question. Again, no simple button.
In case of a real life emergency restore best contact the forum/Discord/whatever for the application in question.
tldr; in the TrueNAS context always use host path for everything supported. Snapshot and replicate (backup) those.
Unfortunately, being slightly less well informed than one or two other forum members (i.e. I am a very basic beginner and pretty well everyone knows and understands more than me!) I have failed to find on my Electric Eel box where the dir containing the snapshots is.
Where might I find the snapshots please - can you help?
I think I might want to create some tiered snapshots of /mnt/.ix-apps (so I can experiment with a few things to do with apps which I sometimes mess up).
Oh! That’s simple enough, even for me. I hadn’t realised an OS could “hide” .dirs such that bash commands like ls won’t show them. Thanks.
I was thinking about creating a tiered snapshot for /mnt/ix-apps/ such that I can dig into it if I get stuck with some of the apps (Frigate in particular, although that might be less of a problem now that I have set up some of the config in a better way).
I already have tiered snapshots set up for my main data (documents, music photographs etc.) but not for the ix stuff.
That’s a good point. I expect I shall refrain from doing anything potentially risky for the time being; now that I have fixed my frigate problems (perhaps) I might stay well away from the idea anyway.
