How to resume failed replication task instead of starting over

TrueNAS General
1

I have the same issue as this person here:

I have a replication task that is impossible to finish without at least one connection loss.
In Germany, basically all ISPs perform a force disconnect after 24 hours.
The snapshots I have are too large to be transferred within that time frame.

Obviously I was expecting that the replication would simply resume after the disconnect, especially since the snapshots seem to be transferred in chunks anyway (“Sending 2 of 381“…).

I was quite (negatively) surprised when the replication simply started over from scratch after being triggered again.

So I hope that I am just doing something wrong and help would be appreciated.
Otherwise TrueNAS is pretty much useless for me and this would leave me wondering how the most basic of basic features (resume transfer) has not been implemented in a NAS OS of all things (I would expect this to happen to a Beta-version of vibe-coded hobby project, but not an “enterprise-grade“ solution).

2

Resumable replication can only be achieved via the CLI, and not the UI.

  1. First, verify the receiving pool has the extensible_dataset feature enabled: zpool get feature@extensible_dataset <name of receiving pool>. If this has the value active, resume is possible.
  2. Perform the replication via CLI: zfs send -<desired send flags> <source snapshot> | ssh <user@receiving host> zfs receive -s -<desired receive flags> <destination dataset>. When the replication cuts off, examine the properties of the receiving dataset to retrieve a resume token: zfs get receive_resume_token <name of receiving dataset>.
  3. Restart the replication in CLI, using zfs send -t <value of resume token> -<flags from the original replication job> <source snapshot> | ssh <user@receiving host> zfs receive -<desired receive flags> <destination dataset>.
3

Ok, thats hugely disappointing but thanks for the confirmation.

Why does it not work via the UI when it is apparently technically possible?
Does the command show the progress of the transfer / when replication gets cutt off?
Do I have to jump through the same hoops when restoring files or whole snapshots?
As the transfer seems to be based on chunks, I assume that even a text file could require transferring a multi-gigabyte chunk (just speculating here).
Is there any ETA when this basic feature (is there anything even more basic than resume after failure) will be added?

4

You are WAY better off with the CLI than with nothing. (This was an educational read for me.)

Once you’ve finally succeeded with your first replication, future ones should be incremental and therefore faster (and the GUI should work).

5

UI resume might technically be feasible, but developers have to focus their efforts using the 80-20 rule. Get the most common 80% case working well, which takes only 20% of the schedule. The 20% of oddball corner cases will then consume the remaining 80% of the schedule. There are other bugs to stomp and features to add. You’re always free to submit a feature request.

Yes, if you include -v flag for verbose output for both the send and receive.

If the snapshot already exists on the receiver, you can just roll back to it. ZFS doesn’t operate on the level of individual files, only datasets/zvols and their snapshots.

Replication is based on snapshots. Except for the very first replication, you’re not transferring the whole dataset over again, just diffs of blocks that have changed from the previous snapshot.

Feel free to create a Jira account to track developer progress, if your feature request is accepted.

6

Indeed disappointed with TN GUI options for replication.

The 80-20 target audience must be very basic users, as I see this as a basic feature (power of ZFS, right?).

But TN replication is based/dependant on zettarepl, so that’s why it is that way.

Been considering sanoid/syncoid for a long time.

Just found sanoid portable and I will end up migrating to it, so I can have more control and options.

WORKAROUND in your case:

Configure multiple replications: Include ONLY certain datasets, or exclude certain DS.

7

This is best practice.

I believe these are based on rsync, which can natively resume syncs. However, OP has a misunderstanding of the purpose of replication. It’s not file-based, and isn’t intended as a file copy/sync utility like rsync. It’s block-based, and allows for duplication of versioned filesystem snapshots.

Yes, the underlying replication engine used in the UI is zettarepl, which is supposed to be able to handle resumes. This seems to be a UI limitation not exposing this feature.

8

I may not have a clear understanding of the technical implementation, but I sure have a clear understanding of the PURPOSE of replication.

Replication should mirror my existing data to another TN instance, which should in many cases, be located offsite.

Since basically ALL traditional end-user internet connections are asymetric, this means, in almost all cases, a very limited (upload) bandwidth to those remote sites.

And it should be just as obvious that people who use TN have large datasets to manage.
I wouldnt go about setting up TN when I only had 100GB of data to manage.

So for any serious backup (3-2-1) a large amount of data will always meet limited bandwidth.
Not having a resume function for replication makes TN basically useless for the 3-2-1 setup.
In case of desaster recovery I HAVE TO KNOW that things will work and not fiddle around with some feature flags on the command line.

But yeah, in light of recent developments and seeing that devs priorities seem to differ quite a bit from mine, it might be a good idea to investiage alternatives…

Nethertheless, I appreciate the provided steps on to manually trigger the replication from the command line, even though I probably wont make use of it as restore seems to unreliable for me.

9

If sanoid was rsync based, I would not have mentioned it.

May want to inform yourself of this ZFS replication solution.