What does this Replication Error message mean?

TrueNAS General
1

Hi
It’s been a while (years?) since I posted here so please forgive me if make basic errors.

Can anyone explain to me the meaning of this error - with a view to telling me what I need to do in order to change things and replicate successfully?

I am a very unsophisticated home user. I have the older TrueNAS because “not broke - don’t fix it”. I bought an additional 8Tb external drive in an external housing and connected it via USB. I have backup up (replicated) to it successfully in the past, but this time something went wrong and I had to wipe the external drive and start again afresh. After running for 2 days (it has circa 7Tb to backup) I wake to find the following message:

New alerts:

  • Replication “NAS - to - Backup” failed: cannot send NAS@auto-2024-04-19_21-00 recursively: snapshot NAS/iocage/templates@auto-2024-04-19_21-00 does not exist
    warning: cannot send ‘NAS@auto-2024-04-19_21-00’: backup failed
    cannot receive: failed to read from stream…

I’m of some senior years and I am not a sophisticated user. I rely on the GUI for everything and only have the thinnest grasp of what Snapshots are and how they work. That said it seems to indicate that because it cannot backup a snapshot, everything has collapsed. So why didn’t it skip that snapshot? More importantly, how do I fix this? I don’t want to run a 2 day+ replication again (my calculation says it’ll take 5 days to backup 7Tb over USB 2.0 (I have no faster USB port on my old hardware) and then find it fails again! Should I try to find this snapshot and delete it? Can I do that - delete snapshots the system doesn’t like? Help! How should I proceed please?

Than you everyone, for your thoughts.

2

Hi @NumberSix, can you post the details of the replication job “NAS - to - Backup” so we can get a better idea of what it’s trying (and failing) to do.

Also, what are you trying to achieve with this backup task? It looks like you’re trying to make backups of snapshots, and it’s failing on a snapshot from April last year. If you’re unclear on the purpose/use of snapshots, I’d query why you’re trying to back them up.

1 Like
3

Hi WiteWolf
Thank you for your interest - I am very grateful for any help with this. As you can imagine, it’s putting at risk all my NAS’ contents (having no successful backup).

Here is a screen grab of the ‘Edit’ details of the Replication task.

screenshot.52
screenshot.52971×813 71.3 KB

I am not trying to backup snapshots specifically, but the entirity of my NAS and everything it contains. A total backup. I have mirored 8Tb disks, so probably 2 X approx 7.5Tb after formatting. They are now 91% full so that’s a rough guide to the amount of data I want to copy to the external 8Tb disk.

I had one risky thought; since it looks like I may have lost part of an old snapshop that apparently should have some recursive component that’s just not there - how would it be if i simply deleted ALL of my snapshots, then ran a new one manually, and only after that’s completed, attempt the Replication to the external drive again? I never make use of the ‘roll back’ facility that apparently snapshots allow, so I imagine it might be an easy fix? I’m probably prososing something dreadfully risky, but I thought I’d voice my guess as how to proceed.

Thanks again!

4

Okay, I think it would be a great benefit to you to read some docs/guides on ZFS snapshots and replication, understand what you’re doing, and how that differs from a traditional backup.

What you’ve got configured is a replication job. Replication tasks exist to backup snapshots that you’ve taken on your local storage to a secondary location, in your case your external USB HDD.

Although you’ve pointed your replication task at the root ( ‘/’ ) of your file system, it may not be making a replica of all the files on there. It will only replicate the snapshots that it finds. If you’re already periodically snapshotting your entire system, then it will be attempting to replicate all those snapshots. While you’re pointing your replication job at a place in your file system, snapshots are performed on a per dataset basis.

While snapshots and replicas are a very efficient way of protecting your data, it’s very different to simply copying a load of files to a backup folder. Do you know how to restore data from these replicated snapshots? Have you tested it?

What I’m trying to say is that, without a stronger understanding of how snapshots work, and how to restore a broken/dead TrueNAS system, this may not be the best backup strategy for you.

That said, back to your particular error: as the replica is taking so long to create (unsurprising, as you’re moving ~7TB of data over USB2, which tops out at about 35MB/s, if you’re lucky), it looks like a snapshot that it had intended to replicate at the start of the job is no longer there on the source drive when it got around to it, possibly many hours after the job started.

I would suggest reviewing your backup strategy:

  • understand how to restore your TrueNAS after a catastrophic failure (hint: you don’t need to backup the entire file system, just the configuration and the datasets containing files you want to keep)
  • take snapshots of the datasets you specifically want to keep
  • make replicas of those datasets, not the entire system
  • consider using rsync rather than ZFS snapshot/replica if you don’t want to learn about ZFS

The initial replication of your large dataset(s) is always going to take a very long time over USB2, but subsequent replications should take a lot less time, using either rsync or ZFS.

1 Like
5

Hi Wite Wulf
Again, thank you for your reply, and especially for such a well set out clearly structured one too.

I took a look at both of the documentation links you sent and paniced. The thing is, I understand very little CLI/Linux-esque stuff. I can handle cd and ls and that’s about it. Not sure how long I’ve had TrueNAS (about 4 years?) but the whole time I live or die by what the CLI can offer. There’s a personal dimension to this too; I’m 70 now and my experience (and that of my peers, some of whom used to be world class coders) is that when you hit a certain age, you just want stuff to work. There is no pleasure, and diminishing capability, in delving deeper into trying to work with significant technical complexity (and that’s me with a Master’s degree in computer science talking, too!). As one friend put it, “programming is a young man’s game”. He is so goddam right, unfortunately.

So here’s where I am. I think I may have been bluffing my way through TrueNAS for long enough. I’ve fortunately never had a major disaster so far, and you’re right, I wouldn’t know how to use external snapshots to restore anything. Indeed I had no idea the external drive replication was only storing snapshots; I set up a SMB share to it to look what was on there and it certainly appeared to have a full file structure with many files I recognised, so naturally I thought it was copying files. I am shocked to find this was some sort of illusion. So, before catastrophie strikes, my inclination is to buy a Synology system - super intuitive to set up and use according to numerous Youtube reviews - and copy my TrueNAS’ contents over to that. Then maybe I can use the TrueNAS as a simple periodic backup from the Synology - minus any snapshots that would appear to do me no good since a) while I understand the concept of a snapshot of the state of the FS at a a given point in time well enough, I don’t understand how to actually use them to eg roll back to last week / month / year, nor how to undo that roll back, nor why my weekly automated snapshots produce many hundreds of snapshot files - one for every folder in a pool it seems, and b) since these snapshots would only be stored on the TrueNAS in that setup, if the TrueNAS failed then those snapshots would be lost too, so no point in them existing in such a configuration.

If you are following my logic, is it sound or am I sleep walking into another mistake here?

Your thoughts are once again, welcome, and very much appreciated. Thank you.

6

Footnote:
I found a very user-friendly video on Youtube that mostly explained how Snapshots work. In tandem looking at Synology and finding tough choices there (a three-way pull between the most user friendly software for any NAS according to majority reviews, the exhorbitant cost, and the really lack-lustre hardware configurtions with actual downgrades due in some pending 2025 models). So while I juggle that decision I AM trying to understand Snapshots rather better, AND upgrading to a USB 3.0 socket on both sides of the would-be backup.

One doubt persists though. If the initial Snapshot is of zero-ish size, and if some accident befalls all or part of my data, it has to be unable to restore it, right? So - how many changes need to occur and how many new snapshots generated before the snapshots become a resource worth backing up? Put differently, should I ONLY see snapshots as a ‘roll back’ tool that relies on intact existing data to roll back with, so it can calculate the differences and revert an intact set of data to a prior state, or can it ever be used to restore EVERYTHING in the event of catastropic failure?

Thank you for your thoughts.

7

Can I tempt you to share a link to that, please?

1 Like
8

Want to bet the video features a member of this community? :wink:

9

“G’day guys…”

1 Like
10

That’s what i was looking for - guessing it’s a truck or two…

11

Certainly Sir!

Nice chap who may or may not be part of the Community

Regards!

12

Well I picked the winner in yesterday’s Grand national so yes dude, I’m feeling lucky :slight_smile:

Is he or isn’t he? :slight_smile:

13

And I missed the specific ID - Sorry @winnielinnie