I have run into a problem replicating a ZFS dataset from another machine (running Debian + openZFS) to TrueNAS Scale. The snapshots are generated on this machine via Sanoid and are done monthly, daily and hourly on a dataset (/home) which has several sub-datasets (for each user’s /home, named by their first and last name).
In the replication task in TrueNAS, I set the snapshot naming schema to match the one that Sanoid is producing: autosnap_%Y-%m-%d_%H:%M:%S_daily
The problem is now, that TrueNAS (at least what I gather from the error message when replication fails) seems to expect all snapshots not only to match the naming schema but also be named exactly the same for all sub-datasets?
The error reads:
Replication “home pull from storage02” failed: zettarepl: zfs send PID is 2617494 cannot send storage-fast/home@autosnap_2024-06-08_00:08:03_daily recursively: snapshot storage-fast/home/some.user@autosnap_2024-06-08_00:08:03_daily does not exist
This sounds very much like it expects the sub-dataset snapshots to be taken at exactly the same time, which (at least on my server) is not possible since it takes Sanoid some time to snapshots each user’s /home. This leads to the snapshots of the sub-datasets having slightly different timestamps like this:
So it will be impossible to ever archive the same timestamp on all of them. Does TrueNAS really expect this or is the error message misleading? Can I somehow tell TrueNAS in the replication task settings to be a little more lenient here (maybe with wildcards?) and just replicate the thing?
Looks like there is an (undocumented?) flag for doing snapshots atomically (first comment on: Reddit - Dive into anything).
But I assume if I set this flag now, I’ll have to wait for my entire retention period so that all non-atomic snapshots are pruned and only atomic ones are present? Because I assume TrueNAS will start at the earliest snapshots present?
I don’t believe so. I think that TrueNAS (zettarepl) will just ignore the non-conforming snapshot names. (They will be ignored and hence not replicated to the destination.)
On second thought, you may be right that you’ll have to wait for the expiration date to arrive, since the parent dataset will have the oldest “hit” of a matching snapshot name, which the children will be lacking. (It will use the parent’s oldest snapshot name as the “starting” point it expects from the children as well.)
You could possibly destroy your snapshots now, and start over from scratch, if that’s feasible for your case. (This might be risky in your situation.)
Yeah, deleting all snapshots will not be possible as this is a production system. Waiting for the retention policy to run out will take about 3 months, so it will probably be quicker to just swallow the bitter pill of replicating all 60 or so sub-datasets one-by-one.
Unless… there is a checkbox in the TrueNAS replication task settings that reads “Replicate specific snapshots” but the description sounds like you cannot supply a specific name here (for example: the first atomic snapshot autosnap_2024-06-17_00:00:00) but rather like TrueNAS will judge the “specific snapshots” by their timestamp? So I assume it will start with the non-atmoic ones again?
It gets complicated because of the obfuscation from the GUI, and the difference between atomic actions vs multiple distinct actions.
Because you already had been doing non-atomic snapshots, you’ve essentially created many different “train tracks” in a sense. Unless you have a common (atomic) snapshot shared by all datasets in this “nest” (and it also exists on the destination as well), then there’s nothing you can really do to “re-sync” them all together, in terms of a single replication operation.
For example, if you were so lucky to have created a manual recursive (atomic) snapshot for the parent dataset that is called manual_2024-01-01, and it still exists on all datasets, then you could create a new atomic snapshot tomorrow named autosnap_2024-06-18_00:00:00.
Then in the command line, you can format the “send” side like this:
As you can see, it requires an atomic snapshot sometime in the past. In the above example, you would have to transfer a stream from January to June, which is better than the entire filesystem(s) from scratch.
I think I made a manual snapshot like 1.5 years ago during a server transition. I have to check if that was an atomic one and if it is still there, though
Just doing a new atomic recursive snapshot and naming it something completely different (home@thismoment_2024-06-17_18:00:00) would not work? And then giving TrueNAS the naming scheme “thismoment_%Y–%m-%d_%H:%M:%S” as “specific snashots”? Or would it still try to grab all the “autosnap_xyz” snapshots before it and fail because of the non-atomicity?
That would only work if you don’t involve the “already on different train tracks” snapshots that are named “autosnap”.
So even if you could get it “up to speed” with a new (atomic) recursive manual snapshot, you’re once again at the mercy of TrueNAS’s GUI, which doesn’t know how to only start from a certain date. (Because you want to jump back into the snapshot names that start with “autosnap” moving forward again. Unless there’s some smart logic with zettarepl that will intelligently search for the first matching hits across all children that are atomic, while ignoring the naming schema of “autosnap” from before today? I’m not sure how the TrueNAS / zettarepl logic works.)
But when it’s an entire “nest” of children? It will find such a “common snapshot” of the highest parent, which it then expects (rightly so) to exist on all the children down the nest.
There is why not using atomic recursive snapshots is a bad idea for replication. The rationale by the Sanoid developers is too narrow and short-sighted, in my opinion. (As you can see in this very thread how it “breaks the flow” of a single seamless backup.)
Luckily, they do have an option to create standard atomic recursive snapshots, but it’s sort of “hidden”, and they don’t expect their users to use it.
EDIT: To phrase it another way. TrueNAS (zettarepl) is making the assumption that your automatic snapshots were created atomically down-the-nest when you run a Replication Task that sends a recursive set of datasets/children.
Oh btw, I don’t know if that changes anything but the dataset (and its children) do not exist at all, yet, on the target server. So it would be a full dataset replication in any case. But that doesn’t change anything that was discussed here, right?
On second thought, and I never tried this, you could possibly:
Bulk rename all the snapshots on the source (Debian) to something like autosnap-old or anything that will have it passed over by TrueNAS’s zettarepl.
Starting from today’s Sandoid snapshots (which are atomic), do a full filesystem replication. Zettarepl will only pick up the snapshots that match the naming schema, and will skip autosnap-old and such.
After the full filesystem replication, your subsequent replications will be incremental, starting from today (and only using the newly/correctly created atomic snapshots.)
You’ll have to figure out what to do with the old (renamed) snapshots. You might even have to manually prune them over time.
I’ve never tried such a thing, so I cannot tell you if it will work in practice.
Hmm, that could be an option. I set the Sanoid options to the setting “recursive = zfs” today for the “home” dataset, so from today on all new snapshots should be atomic.
There is also another option that came to my mind today. For this, I need to explain our current setup a little:
We have three storage servers in total:
storage01 is the active server with Debian and a very old OpenZFS (0.7.12 or so)
storage02 is the current backup server with a newer Debian and OpenZFS (2.0.2 or so)
the TrueNAS is supposed to replace the old storage01 soon (but it needs the data first)
storage01 takes hourly, daily & monthly snapshots with Sanoid (since today, recursive atmoic, up until today recursive non-atomic)
storage02 pulls the snapshots over hourly via Syncoid
Now what I did so far was trying to replicate from storage01 to TrueNAS via the TrueNAS GUI, which failed because of missing atomicity for the “home” dataset. The other datasets replicate fine because they don’t have any child-datasets.
Now my new idea from today was: Since I have the full “home” dataset backed up to storage02 anyways, couldn’t I delete all snapshots (up until the new atomic ones starting today around 4pm) on storage02 and then replicate to the TrueNAS from there?
This way, the (non-atomic) snapshots would still be available on storage01 in case a rollback is needed.
Only thing I’m not sure about is, whether Syncoid will start pulling the just-deleted old snapshots over on the next replication run between storage01 and storage02 (once per hour) or just leave them alone? I guess it will pull them? Or will this leave Syncoid with storage01 and storage02 not having any snapshots in common and doing a full replication? Not sure what is the logic inside Syncoid for distinguishing between atomic and non-atomic snapshots of the same dataset. I think it doesn’t care like TrueNAS.
The whole “home” dataset is also only ~500GB total, so it might fully replicate during the one hour until the next Syncoid synchronization.
This is one of those posts that could really use pretty pictures and flowcharts.
Yes you could.
If you resort to a rollback to any of these “before atomic” snapshots on storage01, you’ll essentially put yourself into the same situation as today; nor could you any longer replicate to storage02 (or to the new TrueNAS server) anymore. (Unless you’re willing to destroy all snapshots on storage02 and TrueNAS that are newer than the rollback of storage01.)
Whether you use Syncoid, zettarepl, or vanilla ZFS, if the -R flag is invoked on the “zfs send” command for a non-incremental stream, then it will send everything up until the specified snapshot. This includes any prior and intermediary snapshots of any naming schema. Everything. The next subsequent replications will not include such intermediary snapshots (even with -R), unless you use -I (uppercase) instead of -i (lowercase).
After some deliberation which solution has the best balance between “keeping our retention policy alive” and “no extensive effort for the replication” I think the easiest way forward is to just wait until the non-atomic snapshots get pruned out of the “hourly” (24h) and “daily” (7 days) retention periods, then delete the monthlies and then replicate to the TrueNAS. There is currently a dataset replicating which will take about 2 more weeks anyways, so waiting the 7 days should be fine. It may allow me to even wait for 1st of July and get a first atomic monthly snapshot, so I only have to delete the two older non-atomic monthlies.
Judging by the snapshots transferred tonight, Syncoid doesn’t seem to care about the atomicity for replication. All snapshots that were taken after I set “recursive = zfs” on storage01 yesterday around 4pm were replicated fine to storage02.
