How to keep more snapshots on the replication target than the source?

[truenas-fan]

Yes, having multiple snapshot tasks can be sent with just one replication task as long as you add the snapshot names in the replication task.

[avidee]

That’s not the issue, though.

Having just one replication task doesn’t allow you to set different expiration lengths on different series of snapshots. I’m forced to have multiple replication tasks just so that I can get my backup machine to hold more snapshots than the main machine. The question is then if you can get multiple replication tasks to coexist, and the frustration is that we need to play this game because TrueNAS doesn’t have a feature of “keep more snapshots on the target machine than on the source machine” and declined it as a feature.

[pmh]

@avidee

TrueNAS doesn’t have a feature of “keep more snapshots on the target machine than on the source machine”

But this is something we definitely have - for a single pair of snapshot and replication tasks.

[avidee]

@pmh

But this is something we definitely have - for a single pair of snapshot and replication tasks.

Right, but I want to do it for more than just a single pair. That’s what this whole thread has been about. When the thread is “TrueNAS doesn’t support ‘keep more snapshots on the target machine than on the source machine for tiered snapshots’”, a reply of “but it does support it, just not for tiered snapshots” doesn’t make sense.

At this point I’m trying to pick between giving up and just using the stock replication so that the snapshots are identical on my remote machine, or wiping my remote machine and going with stock zrepl. I’m not sure which, yet.

[etorix]

What’s the issue with multiple tasks if you want different retention times?
How would you manage different retention times for different snapshot schemes within a single task? (Cue in some convoluted interface…)

You can have as many snapshot schemes as you want on the same data. It’s all metadata.

[pmh]

Please try my suggested scheme with three different “namespaces”.

[avidee]

@etorix

What’s the issue with multiple tasks if you want different retention times?
How would you manage different retention times for different snapshot schemes within a single task?

I’m aware of having multiple snapshot schemes, each with its own retention rule. That’s TrueNAS 101.

I’m talking about replicating the pool to the remote server, and having different retention rules for the remote server. That is not trivial, and seems to be almost impossible using standard TrueNAS UI.

The only way I can think to do this is to have multiple replication tasks, each with a different “Periodic Snapshot Tasks” set (so that you can expire them differently) but each replication task including every other task’s snapshots as “Also include snapshots with the name”, then in theory, each of the replications should be able to find the snapshots from any other replication that uploaded it, and avoid duplicates.

That is insane, but about the only way I can think I could make it work.

[avidee]

@pmh

Please try my suggested scheme with three different “namespaces”.

I think you’re referring to

Then create three different replication tasks based on these with yet different retention times on the destination. Again snapshots will only use as much space as necessary to get from one to the next younger one, regardless of the name or by which job they are created.

That’s what I’m describing. Each replication task needs to be driven off of a different snapshot task, but each replication task needs to include every other replication task’s snapshots as “Also include snapshots with the name”. The “daily” replication task has to send the “weekly” snapshots so that when the “weekly” replication task runs, it can see the “weekly” snapshots in the remote and avoid sending the remote server the data that it already has.

(Again, the weekly replication task has to run, even though the daily task has guaranteed sent all of the data, just to expire the snapshots, because there’s no way to set up an expiration-only snapshot task. If we just got the ability to schedule snapshot deletion without creation, that would solve all of this!)

I have never tried this but I am quite confident it should work.

That’s what scares me. It seems like no one has dealt with this situation, but is happy to give me advice that they can’t say for sure works. My remote server is on the other side of the world. At the rate that data flows, a replication will take weeks. I can’t risk playing a game of “oh, I set up my replication rules wrong and now TrueNAS is sending the same data three times because the replication system doesn’t realize that the remote system already has it.”

[pmh]

@avidee

The only way I can think to do this is to have multiple replication tasks, each with a different “Periodic Snapshot Tasks” set (so that you can expire them differently) but each replication task including every other task’s snapshots as “Also include snapshots with the name”,

That’s not what I suggested. I think three replication tasks each working only on the hourly/weekly/monthly naming scheme would do the trick.

[avidee]

@pmh

I think three replication tasks each working only on the hourly/weekly/monthly naming scheme would do the trick.

OK, thinking this through.

The hourly replication task runs, and replicates only the hour snapshots. It does this for a week straight.

Then the weekly replication task runs. It hits the line

zettarepl/replication/snapshots_to_send.py

in which it asks, “what data does the remote server have?” Because (in this scenario) the daily replication task isn’t replicating the weekly snapshots, the weekly replication task thinks that the remote system is a week out of date, and sends a week of data that the remote server already has. My upload link dies.

The reason I’m insisting that each replication task has to send all the snapshots is because I see that line of code. Please correct me if I misunderstand it.

[avidee]

One last completely different approach. Delete old snapshots from CLI | TrueNAS Community led me to GitHub - bahamas10/zfs-prune-snapshots: Remove snapshots from one or more zpools that match given criteria · GitHub.

So the simplest possible thing to do would probably have one replication task, have it push all snapshots, have it not expire any of them, and then set up zfs-prune-snapshots in three cron jobs in the remote server to do the cleanup.

[ProfessionalAmateur]

Maybe I don’t get it but do you mean something like this?

SOLVED - Rename a snapshot? Can I rename an "auto" snapshot to save it...

That should work, right?

@winnielinnie mind chiming in?

[pmh]

@avidee

Because (in this scenario) the daily replication task isn’t replicating the weekly snapshots, the weekly replication task thinks that the remote system is a week out of date, and sends a week of data that the remote server already has. My upload link dies.

Yes, that’s what I assume how it would be working in practice. I do not have a problem sending a week’s or a month’s worth of data.

So no solution for you. Sorry.

[avidee]

@pmh

I do not have a problem sending a week’s or a month’s worth of data.

Lucky you; my backup server is a big Hetzner box in Finland, and I’m getting something like 50Mb/s encrypted to it. I genuinely don’t know if I can do better. Is it Tailscale? Lots of small files? The 10 disk RAIDZ2 on the receiving end? This is literally going to be a month.

I set up some cron jobs with zfs-prune-snapshots and I think that’ll do. I do think that easily managing tiered snapshot systems should be easier. Switch to zrepl?

Thanks, all.

[winnielinnie]

@avidee

So I thought I’d check in to see if this had changed, with the hope that surely this would have been an added feature by now.

This post hits the nail with the hammer.

The TrueNAS way of handling the automation of ZFS snapshots and replications is very narrow in scope. It’s really designed to be used by tethering an automatic snapshot task to a replication task and be done with it. Anything beyond that requires multiple tasks for the same targets (redundant and resends data), or custom scripts, or using the command-line.^[1]

I doubt it’s going to change anytime soon.

---

1. If you want a hint, all you need to do is grab the names of the latest snapshot on the destination and source, and then use the first snapshot name as the “base” and the second snapshot name as the “end”. Pass the -I parameter (capital “i”) so that zfs send will include all intermediary snapshots. One command, one send that includes all snapshots, no matter their naming schema. The problem is, the source server will not prune snapshots on the destination, since that is handled by zettarepl via the configured Replication Task. You’d have to use another means to prune the snapshots on the destination. ↩︎

[avidee]

@winnielinnie

The TrueNAS way of handling the automation of ZFS snapshots and replications is very narrow in scope.

Thanks for the confirmation.

Re your footnote, for sending all the snapshots I’m going for a standard pull but rather than a naming schema, I’m going for “matching regular expression” of .* which seems to accomplish what I’m looking for. I’m doing a manual purge using cron jobs.

The real bummer here is that zrepl seems more along the lines of something that I’d want to use, but I can’t figure out a reasonable way of installing it. I don’t think it’s very prudent to install it directly (as Run zrepl on TrueNAS | Alan Norbauer does), but you can’t run it inside a VM, nor an LXC, nor a Docker container, so I guess not.

[pmh]

@avidee

Is it Tailscale?

Why that? An SSH connection with public key authentication is as secure as any VPN.

And Hetzner Helsinki, that’s EU, GDPR, as trustworthy as can be We have a large storage server at their location, too.

[avidee]

@pmh

Why that? An SSH connection with public key authentication is as secure as any VPN.

It’s not only about security, it’s about convenience and ease of use. Because I put the remote server on Tailscale, it can easily connect back into my home network to do pull. Fair point in that it might be worth trying doing a push replication over just ssh to see if it’s faster.

[avidee]

Amusingly (or sadly) enough, with 110ms RTT, SSH is slower than Tailscale. See my adventure at Is TrueNAS is shipping an SSH binary that throttles connections over high-latency links?

Time for plan C: zrepl. No VMs or LXCs or Docker containers. This should be doable with a shell script and a cron job. I cannot believe I am forced into doing this due to complete lack of tools on TrueNAS (you had one job, TrueNAS!), but so be it.

[samlevy0515]

@avidee

I want to set up a system that has daily/weekly/monthly snapshots that are retained for a certain length of time, and I want to replicate that system to a backup TrueNAS server on which the snapshots are retained for longer. I have searched forums and the internet as a whole, and I have dug into the menus of TrueNAS, and it seems like that is impossible. Please help me, as that seems like something that should be easy to do. Surely I am misunderstanding something.

The creation of different types of snapshots is easy. You go into “Periodic Snapshot Tasks”, and create a task for each type of snapshot. You set retention there.

As for replication, let’s look at push replication. If you set up a replication task, under “Periodic Snapshot Tasks” you can select all the snapshot tasks. Excellent. Under “Snapshot Retention Policy” you can say “none”, which isn’t what I want, or “Same as Source” which isn’t what I want, or “Custom”. That sound great, but there’s just one edit box to specify a fixed lifetime for all snapshots, and I want a different retention for daily vs weekly vs monthly ones.

One might say, “use the ‘none’ retention policy, but on the remote side purge snapshots on the schedule you want”. OK; how does one do that? Snapshot tasks only purge when run, and they’re only run to create snapshots on a schedule. If my replication target dataset is read-only, which I’d want it to be, I can’t create the snapshots, and so I can’t purge them on schedule.

Switching to pull replication doesn’t seem to help. If I use pull replication, I no longer have a popup of “Periodic Snapshot Tasks” to select tasks, so either I provide the naming schema of one type of snapshot, or craft a regex to get multiple tasks. If I craft a regex to include the multiple tasks, then the “Snapshot Retention Policy” is only “none” or “same as source” which doesn’t help, and if I give the naming schema, then I get the “custom” again.

And I can’t have multiple replications going on, one for each type of snapshot, in which I can set individual expirations, as that would replicate everything multiple times if it allows me to do that at all.

Maybe zrepl would work better for my use case, but there’s no good way to get that going on TrueNAS. Does zettarepl have a magic switch not in the UI?

I’m in shock that this basic task of keeping snapshots longer on a replication target than a source seems so impossible. Surely I’m missing something. Any help would be appreciated.

Thanks!

You’re not crazy—TrueNAS/zetta-repl just doesn’t support per-snapshot-type retention on the target side via the UI, so the usual workaround is splitting replication by naming schema or handling retention with custom scripts/zrepl instead.