Resuming replication after failure

TrueNAS General
1

I have replication running between two Core 13.0u3 boxes. This morning, I fired things up and got the following message:

Error

resume token contents:
nvlist version: 0
	fromguid = 0x9c062e69d0e095e7
	object = 0x2e6c801a
	offset = 0xe0000
	bytes = 0x955463f7ac
	toguid = 0xfeb45dbeb9820ccc
	toname = bigpool/snapshots/nitai/Torque_big@auto-2024-12-24_00-00
	compressok = 1
	rawok = 1
client_loop: ssh_packet_write_poll: Connection to 172.16.53.174 port 22: Broken pipe
cannot receive resume stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t 1-1715b437df-148-789c636064000310a501c49c50360710a715e5e7a69766a63040c1f3a90f2e64eab1cd5100b2d991d4e52765a526973030f8bd663100a9c3904f4b2b4e2d618003903c1b927c5265496a31907e61f5e9c6322cfa4bf221ae38c3d3b4735fec967f1148f29c60f9bcc4dc54a03999e905f9f939fac5798905c519f925c5fa7999258999fa21f94585a5a9f1406987c4d2927c5d230323135d43235d23937803035d03036477713320c223393fb7a028b5b8383f1b22260175374cbe28b11c26c500002f8e38cf.

Logs

[2024/12/26 07:53:12] INFO     [Thread-373] [zettarepl.paramiko.replication_task__task_15] Connected (version 2.0, client OpenSSH_8.8-hpn14v15)
[2024/12/26 07:53:12] INFO     [Thread-373] [zettarepl.paramiko.replication_task__task_15] Authentication (publickey) successful!
[2024/12/26 07:53:13] INFO     [replication_task__task_15] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots: []
[2024/12/26 07:53:13] INFO     [replication_task__task_15] [zettarepl.replication.run] Resuming replication for destination dataset 'drpool/nitai/Torque_big'
[2024/12/26 07:53:13] INFO     [replication_task__task_15] [zettarepl.replication.run] For replication task 'task_15': doing pull from 'bigpool/snapshots/nitai/Torque_big' to 'drpool/nitai/Torque_big' of snapshot=None incremental_base=None receive_resume_token='1-193479d5a4-148-789c636064000310a501c49c50360710a715e5e7a69766a63040c1f3a90f2e64eab1cd5100b2d991d4e52765a5269730304835e4e881d461c8a7a515a7968064f81860f26c48f2499525a9c5407acdf7e490a958f497e4435c7186a769e7bed82dff2290e439c1f27989b9a9407332d30bf2f373f48bf3120b8a33f24b8af5f3324b1233f543f28b0a4b53e381d20e89a525f9ba46064626ba8646ba4626f10606ba060630ff81ece566408447727e6e41516a71717e36444c02ea6e987c5162394c8a01008d6d3801'

So I tried running the

zfs send -t …

command on the command line of the sending host, which returned the error:

Error: Stream can not be written to a terminal.
You must redirect standard output.

What is the proper way to get this replication to resume?

2

It still needs to be piped to the destination (zfs recv).

I take it this was a replication triggered via the GUI?

Does the GUI not support automatically using a resume token when one exists on the destination?

3

Okay, so do it by hand. Copy that.

It was. I grew up on TrueNAS (FreeNAS) back in the days when the adage was “Never do anything on the command line behind the GUI’s back.”

I’m not sure. In the replication task, if you edit the specific task, there is a checkbox for “replicate from scratch”, but I am not sure if that is actually a diff of the replication, or if it is a “nuke it from orbit…It’s the only way to be sure,” blunt force trauma approach.