Replicate Replicated Dataset

TrueNAS General
1

Hey all, I’d like to say I appreciate in advance any help offered.

I’m running Truenas Scale with three main pools setup (backup, default, software). I have snapshots setup to run on default and software following what i could find to be recommended ( 10 min, hourly, daily, weekly ) with varying retention times and everything set to create empty snapshots except the 10 min one. I have two replication tasks setup recursive and automatic from the snapshots ( hourly, daily, weekly ) with some exclusions as i only want to backup certain data. Both replication tasks are set to go to backup/replicas which is encrypted by key and set to inherit encryption (the original data sets are not encrypted).

My ultimate goal is have a backup locally ( yes i know its the same server ) and then to have an offsite backup that is encrypted (zero trust ) and doesn’t have the key. I understand the prinicples but i’m having an issue from backup to offsite backup.

The offsite backup is an ubuntu 22.04 box with zfsutils-linux installed and an unencrypted pool called default.

Locally the replication works great and the dataset shows its encrytped. I can setup a replication task from backup to offsite with recursive set, but it only replicates the main replicas dataset and not the underlying default and software data sets that were replicated. I create a snapshot with recursive and i see it create snapshots of the entire dataset and children, but after a while the children datasets dissapear, I’m guessing because the replication task for local to backup deletes them as they are not in the original dataset.

When i adjust the replication task to just include all snapshots through regex .* it doesn’t find any snapshots (the replicated ones ). so that it can replicate to offsite.

I know i can’t replicate the unencrypted to encrypted as the key is then on the remote server. If I replicate the backup it ends up only sending the root dataset (replicas) and nothing else, but its encrypted as i intended.

I know there are guides for using backblaze s3 storage, but i would like to try and have this work if i could. The vps is servarica lobster that grows 3gb a day, so over time i can slowly add storage which should be good for a while, eventually I’ll do this same setup with a server at a friends house, which is why i’d like to get this to work.
Does anyone have any suggestions or guides or alternatives?

2

You shouldn’t be creating new snapshots on a backup target. This will cause breakages.

When asking for help with these types of situations, you’ll have to split up your post into sections with clear layouts and relevant information.

I know it seems self explanatory to you (since you are sitting in front of your server/GUI), but it’s hard to follow.

I know there are three pools on your TrueNAS SCALE server:

  • default
  • software
  • backup

I think you’ve successfully replicated the root datasets of the pools “default” and “software” into your backup pool, nested like this:

  • backup/replicas/default
  • backup/replicas/software

Within each of these exist all the children (and the replicated snapshots).

Is everything encrypted with these “replicated root datasets”, including their children?

Is backup/replicas encrypted?

How did you configure the Replication Task to go from backup/replicas → Ubuntu?

3

I started to guess this wasn’t a good idea and removed it.

Makes sense thank you!

yes everything is encrypted on the backup including the backup/replicas and your layout is exactly right.

I created the replication task through the ui using the advanced settings.

gave it a name
push direction
ssh transport
ssh connection set, no compression
sudo unchecked
allow blocks larger then 128kb and allow cmpressed write records checked
retry is 5
logging level default
Source → backup/replicas recursive checked
destination → default/replicas
snapshots retention same as source
replication from scratch checked off until i get the initial one complete
encryption unchecked (as its encrypted on disk)
nothing in override, exclude or periodic snapshot task
matching regular expression selected
.* for regex
schedule (midnight daily)
Shows no matching snapshots

won’t let me save.

4

Did this dataset already exist on the Ubuntu pool before you ran the task for the first time? Or did it get created from the first-run of the replication?

What about “Properties” and “Full Filesystem Replication”? (I’m not using SCALE, so the options might be named differently, or even were removed?)

Probably because the snapshots were created for the “unrelated” (“side-by-side”) root datasets default and software, and thus the dataset backup/replicas has no such snapshots. (Note: There’s no reason to have to resort to a .* regex for a Replication Task.)

You might, unfortunataley, have to create two different replication tasks, like so:

Task 1:
backup/replicas/default → Ubuntu’s default/replicas/default

Task 2:
backup/replicas/software → Ubuntu’s default/replicas/software

The problem is that you’re trying to contain two different pools into a single replication task. Because you’re dealing with separate pools (and hence, root datasets), you can’t create a single “recursive” task that have different “roots” where the snapshots recursively begin.[1]

You would have to send these as a “raw stream” (which isn’t very clear in the GUI, docs, or tooltips), since you want the destination to have zero-access and no keyfile/passphrase to unlock the datasets.

  1. To demonstrate, run this command: zfs list -t snap backup/replicas ↩︎

1 Like
5

@Stux Thanks for the look over! I saw your pointer to winnielinnie’s comment, I used your vids to setup the snapshots ( I never used them previously and kinda been letting the whole backup thing slide). Also thanks for the Jailmkr setup, I had to redo all my stuff after the truecharts decision.

@winnielinnie
Sorry for the delay in reply. I’ve been playing around and referencing your comment here. I did finally get it working. and your comments were key!

No the non root user was given zfs permissions on the default pool and this was created by the replication.

I gave this a try with your comment / suggestion as well as splitting the replication job into two and got a different error about snapshots being created after the initial or something. It did copy the core data / datasets, but it was bugged on the snapshots and the task was failing

This returns nothing as there are no snapshots, which now makes sense, if there are no snapshots to replicate then there is no job for the replication task to do. If i create a snapshot on just the root, and try recursive or full file replication it wines because they are not of the same snapshots ( ancestor ).

This was key! I tried the full file system replication but got errors and Replication “backup software - offsite” failed: skipping snapshot backup/replicas/software@cloudserver-hourly-2024-09-27_23-00 because it was created after the destination snapshot (cloudserver-hourly-2024-09-27_22-00)

and that list had almost all the other snaps… So I set back to recursive and scratch and it worked. One thing to note, for some reason it only listed it found 1 or 2 snapshots to sync, but it did do the whole dataset/snaps.

I have a fully synced replication, I’ll know tonight ( it runs daily ) when it goes to update with todays updates. This was with my smaller software pool, now for my bigger pool.

Thanks again for the help, I super appreciate it!!!

Settings for Reference to anyone else:
Set General to your requirements
Source: Root of one of the replicas
Destination: Root of main pool
recursive checked
include dataset properties checked
matching regular expression: .*
run automatically
schedule ( midnight or whatever you prefer)

1 Like