Local Backup to SATA or USB HDD

Feature Requests
,
22

The lifetime of four weeks was just an example.
Having long snapshot lifetimes could serve as a workaround, but it wouldn’t truly solve the issue.

Let’s say I create a snapshot task with a lifetime of six months. To perform a backup, I run a replication task that replicates it to my backup drive.
Now, suppose I don’t perform a backup for seven months for whatever reason (not recommended). When I return to do another backup, the snapshot on the source no longer exists.
As a result, there are no common snapshots between the source and target, and incremental replication won’t be possible.

The method you described is slightly different from the one I mentioned in the original feature request.
I used that approach for a while, but I didn’t like how all the snapshots were replicated and how they cluttered everything.
More importantly, it replicates periodic snapshot tasks.

I prefer to replicate snapshots taken at specific times, since I shut down containers and VMs before creating them to avoid corrupting running databases, causing synchronization issues between files on the filesystem and the state in the database or similar issues.
When using a periodic snapshot task, those containers and VMs are likely still running when the snapshot is taken.

Suggested Improvements to better support such a manual approach and a brief recap

If the fully automated solution I originally proposed takes too much time to implement, it would still be very helpful to at least have the following features:
This would make the manual approach etorix described a more viable option.
(A fully automated solution with scripts could also be implemented more easily with these features and wouldn’t have to rely on 3rd party software)

  • Manual Snapshot Task

    • A manually triggered snapshot task that doesn’t require a schedule.
    • Allows defining snapshot retention based on the number of snapshots.
    • Example: A lifetime of 4 means a maximum of four snapshots are kept. When there are 4 snapshots and a fifth one is taken, the oldest one is deleted.

  • Attach Manual Snapshot Task to Replication

    • The ability to link a manual snapshot task to a replication task.

  • Replication Task Snapshot Retention Policy by Count

    • Support retention policies in the replication task based on the number of snapshots, not just their age.

  • Clarify Retention Policy Help Text in the replication task UI
    The current Help text states:

    “Same as Source: use the Snapshot Lifetime from the source periodic snapshot task.”

    This implies that snapshots are deleted based on age.
    However, the zettarepl documentation (apparantly used by TrueNAS for snapshot creation and replication?) says:

     # How to delete snapshots on target. "source" means "delete snapshots
 # that are no more present on source", more policies documented
 # below
 retention-policy: source

    I interpret this as deleting snapshots based on if they are present on the source or not.
    Right now this only really matters for snapshots which were replicated not because they belong to a snapshot task, but because they match a naming scheme. This disctinction would become more significant if a manual snapshot task feature were implemented.

Alternative Approach

Another possible implementation could use bookmarks and holds, similar to how zrepl does it.
Zrepl seems to already support a process similar to the one we’re describing, and they provide an example in their documentation:

Zrepl: Backup to External Disk (Quickstart Guide)

1 Like
Back up to connected USB drive
23

My plan for off-site ZFS JBOD enclosures was to basically replicate from scratch every time and then note on the enclosure as of what day it was current.

Basically, every rotation would be a replication from scratch. That’s horribly inefficient but it also eliminates the issue of snapshot retention while the system awaits the return of the enclosure.

For my semi-chaotic rotation schedule, it’s hard to gauge how far to set snapshot retention AND to deal with the storage needs of the queue of snapshots being retained. Given how chatty timemachine is, that can really add up.

I also struggle to see why ixsystems would invest capital for this task. In commercial enterprises, this feature likely has little to no value, as it will be someone’s responsibility to rotate backups offsite in a given schedule that accounts for snapshot retention, etc.

24

This is one reason why my detached ZFS backup pool disks use RSync to update. Using RSync’s delete option makes the backup identical to the source, at the time the backup was taken.

Then, I snapshot the backup disk. As long as their is room for the next backup, I don’t clean up. But, if it looks like it might be tight, I prune the oldest snapshot on the backup disk before I start the next backup.

This scheme works well for me, since I have bit rot detection, backup history via limited snapshots and multiple backup disks. One of the backup disks is 50% larger than the maximum pool size, another is same as the max pool size. The last can’t support the media, but can backup everything else. However, my NAS pool is less than 1/2 full.

1 Like
25

Funny, I had to make one myself lately as well :joy:

I made it a bit differently then yours though.

I don’t need the serial. I only look for the pool name. And I can unlock encrypted drives.

In addition, I use rsync to my source datasets to a single dataset in the backup disk.

I still have to improve it, and I want to add a way to automatically start the backup at connection. Right now, I have it in the Cron section of the advanced settings of Truenas, which lets me have it send an mail report at the end of the script.

And since I do a backup once a week, it also starts a scrub after 27 days.

I made it also so that I can configure multiple disks to backup, and skip the none connected drives.

I am not a dev, so I used AI to get the commands to run, and then I implemented it in my own (not best) code.

1 Like