Replication working for months without issue, suddenly failing with "bad snapshot name"

TrueNAS General
1

Hi, I had a replication set up from a ZFS debian host to Truenas Scale and it was working just fine for months but suddenly it started giving me Active side: cannot receive: invalid stream (bad snapshot name).

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 511, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/replication.py", line 484, in run
    await self.middleware.call("zettarepl.run_replication_task", id_, really_run, job)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1428, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py", line 401, in run_replication_task
    self._run_replication_task_job(f"task_{id_}", job)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py", line 461, in _run_replication_task_job
    raise CallError(make_sentence(message.error))
middlewared.service_exception.CallError: [EFAULT] Active side: cannot receive: invalid stream (bad snapshot name).

Nothing changed on ZFS debian source host nor on the Truenas Scale destination.
I checked the list of snapshots on the source and nothing seems out of the ordinary, all snapshots have same names as always.

When I press start on the replication, it flashes this for a split second:

Replication «bpool,rpool - tank/debian-backup» has started.
Task is running
replication.run
0.00%
Sending 1 of 2: bpool/BOOT@autosnap_2024-07-22_00:00:17_daily

Naming schema in the replication is: autosnap_%Y-%m-%d_%H:%M:%S_daily

2

Please open a Jira ticket and attach a system debug to the private attachment area to keep your private data protected.

1 Like
3

Opened Jira ticket as suggested. We’ll see how it goes, Thanks.

4

Okay so basically I was told that there is no support for that. Anybody has an idea what I could check? It was working fine for months and there were no changes.
Can i maybe see what exactly it’s receiving so it’s saying “bad snapshot name”? The log on the replication page is garbage.

5

No software updates?

I can’t think of a reason why it would break, other than a behavior change as part of a software update

6

tldr: suddenly started working for no apparent reason.

There was no Truenas update nor on the debian end there was no updates on 22/23rd of July when the issue started occuring. There was an update on 18th:

Unattended upgrade result: All upgrades installed

Warning: A reboot is required to complete this upgrade, or a previous one.

Packages that were upgraded:
 linux-compiler-gcc-12-x86 linux-headers-amd64 linux-image-amd64
 linux-kbuild-6.1 linux-libc-dev

Packages that were auto-removed:
 linux-headers-6.1.0-21-common linux-headers-6.1.0-21-amd64
 linux-image-6.1.0-21-amd64

Package installation log:
Log started: 2024-07-18  09:27:13

and today update:

Unattended upgrade result: All upgrades installed

Warning: A reboot is required to complete this upgrade, or a previous one.

Packages that were upgraded:
 bind9-dnsutils bind9-host bind9-libs

Package installation log:
Log started: 2024-07-26  06:14:12

I also manually updated right now and rebooted that debian host manually:

The following packages will be upgraded:
  containerd.io docker-buildx-plugin docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin
6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

After doing all that, it still didn’t work, but after that i started messing with Truenas replication settings I think with transport settings,

image

and it suddenly started working. I have no explanation what exactly was a holdup as I didn’t change anything except putting it to SSH and then back to SSH+NETCAT but it was definitely on the Truenas side.