Correct way to script ZFS replication in TrueNAS Scale?

TrueNAS General
,
1

Hi everyone,

I’ve been playing around with the TrueNAS GUI replication and rsync tasks quite a bit lately. I’m trying to create a backup/devtest TrueNAS Server. This means that I’m not only trying create a backup, but I’m also trying to make a 2nd instance of my TrueNAS server that is able to run simultaniously with my master TrueNAS server (so it is not an exact copy, but it is very similar - it has a different name, IP, a additional Plex instance with a duplicated DB/metadata/etc).

From my testing I’ve learned a couple things that are relevant for what I’m trying to do:

  • Rsync can’t properly copy ACLs. This makes it unsuitable for copying my SMB shares.
  • ZFS replication requires to have snapshots. Not all my SMB shares are suitable to have many snapshots though, as some have frequently changing contents for which I only need the latest state.
  • The TrueNAS GUI Data Protection options seems only designed for scheduled tasks. My TrueNAS backup/devtest server will be powered off most of the time, so scheduling backup tasks is not practical…
    • If the server is already powered off before a scheduled task, this task is not picked up when the server is powered on after the scheduled task should have run.
    • If the server is powered off during a scheduled task, the task is not resumed when powering the server on again.
    • Running many replication tasks manually (without a schedule) is possible, but requires starting them one by one sequentially, as TrueNAS GUI does them all in parallel if you start them all at once. So, especially with mechanical HDDs, this is not really an option either.

So, as I understand it, the GUI doesn’t really offer what I’m looking for…

What I’m trying to do is not very hard to do however using a simple bash script. I already found an example script that I can use with some minor modifications:

This script is meant to do zfs replication to an external HDD, but I can easily modify it to replicate to another server as well…

What the script does is:

  • Check if the the backup location already has been backed up to.
    • If not then create a snapshot with findable name and zfs replicate this newly created snapshot to the backup location (initial full backup)
    • If there already was a previous backup on the backup location
      • then the script will find this because it has the snapshot with the findable name
      • This same snapshot should then also still exist on the source location
      • A new snapshot will then be created on the source (also with a findable name) and this new snapshot will be replicated (incremental) to the backup server
      • Both source and backup now have 2 snapshots (old and new) and the script will then remove the oldest snapshot on both source and backup server

This way both locations only have the minimal amount of snapshots to have a working zfs incremental replication and it is possible to do an on demand backup of the current state of the filesystem with ACLs and everything…
If I feel like it, I can also easily script it to be used as a cronjob or on-init-job to for example check once per day (or on-init) if a backup was missed or is incomplete and needs resuming.

So that is all the background information and here come my actual questions :slight_smile:

I know that TrueNAS is meant as an appliance, so modifying is to be avoided if possible and if unavoidable, then it should be done with care, hence my questions…

Not all OS files and folders customizations are “persistent” after reboots or upgrades. However, I don’t know the specifics of this, so I wonder:
Is ‘/root/bin’ a good location to store my scripts? Or will it be removed after an upgrade / reboot? Or which location do you suggest for a root script? My root homedir is the default ‘/root’.

I’ve also read that it is not advised to use the GUI created SSH keypairs outside the GUI. Is this true?
If true, then should I create them manually like on a non-appliance Linux server in /root/.ssh? Is this persistent after an upgrade / reboot? Or is there another way I should do this?
If not true, then how do I use the GUI created SSH keypairs in command line? With a ‘ps-ef’ during an rsync, I noticed that it seems like they’re using temporary keyfiles in /tmp. So I’m not sure how to use those in a script…

Also if you have any other advice / comment on how to handle this, please let me know.

Thanks!

2

I’d still vote for ZFS snapshots and replication. It’s by far the fastest and most reliable option.

For your frequently changing shares - you are aware that you can set frequency and retention time individually?

I snapshot everything every hour on the source and keep it for 2 weeks maximum. You can set that period way shorter if desired.

To the destination I replicate the midnight snapshots only and keep these for 6 months.

All of that is available in the UI.

If you still prefer to go with your script, place it on your storage pool, not /root/bin. I create a dataset named scripts on each of my TrueNASes and place such things there.

3 Likes
3

The script that I’m suggesting to use, still uses ZFS snapshots and replication. I agree that this is the fastest and most reliable option. That is why I want to use it, just not using the GUI…

I don’t really see how to do this with the GUI by customizing the frequency and retention time…

If I work out 2 extreme examples:

  • If I create an hourly snapshot with a 2 week retention, I have to keep 336 snapshots, for which I don’t have the space and if I go on holiday for more than 2 weeks, then I no longer have my base snapshot and I have to perform a full backup again, since incremental requires the previous snapshot. Also I will never backup the current state, as it will be up-to-1-hour out-of-date.
  • If I create a weekly snapshot with a 6 month retention, then I can go on holiday for up to 6 months, but I still have to keep 26 snapshot, for which I also don’t have the space and my snapshot will be up to 1 week out of date instead of the current state…

So not really nice in my opinion…

Thanks, so I understand from this that /root isn’t persistent… But then how do I create SSH keypairs for root (as /root/.ssh is also in /root)?
Do I need to create a cronjob / on-init-job that contstantly recreates the ssh keypairs in /root/.ssh? Or is there a better way?
Should I change the homedir of my root user to a folder in my pool?
Should I use a different user and try to get it working with sudo?

4

ssh-keygen generates key pairs that can be stored anywhere. In your script you can use ssh -i /mnt/path/to/keyfile for example. Only on the destination do you need to place the public key in the home directory of the user in question. If that is root, you can do that in the UI and it will be persistent.

As for the space constraint - if you create hourly snapshots and keep them for two days and have replication set up, why should it matter if you go on holidays? All of this is running 24x7 on autopilot, isn’t it? It is for me and all my installations.

I backup my two home TrueNAS to a third system located at my company’s office, I backup ~100 servers to a large storage system located at Hetzner in Helsinki, Finland, … automation rules :slightly_smiling_face:

1 Like
5

This is why TrueNAS needs to adopt “Syncoid” into its GUI, and perhaps stick it into its own menu, separate from the “Periodic Snapshots” and “Replication Tasks” menus.

Call it “Simple Backup”, create a wrapper/GUI around Syncoid, and we shall all rejoice.

3 Likes
6

Ok thanks, after realizing that I need to copy the private key (not the public key) from the GUI SSH Keypair (I actually was confused and didn’t recognize that kind of format for a private key and wasn’t sure it could be used as a keyfile), it does seem to work!

No, that is exactly the problem for me… My backup server will be powered off almost always. I may power it up once a week / month to make a backup and it may sometimes even take longer, when I’m on holiday… Only when I need to do some test or have downtime on my master server, it will be powered on for longer…
That is why the GUI options, that rely on basic scheduled tasks, are not at all useable for me…

I actually did come across Syncoid in my search for a working solution, but didn’t explore it any further as soon as I discovered that it required installing a package with dependencies in the TrueNAS OS itself.
That seems even more “invasive” and possibly more dangerous to do into an appliance like TrueNAS. I think a simple script like above, which I fully understand and control, seems like a more manageable / acceptable risk to me, which shouldn’t have any negative impact on the TrueNAS OS.

Ofcourse, if IXSystems ever decide to integrate Syncoid, that would be even better :slight_smile:

7

That’s why I said what I said. For it to be included in TrueNAS, with a proper GUI as well. :wink:

2 Likes
8

So in short, I will

  • create my script(s) in a folder on storage pool
  • create a private keyfile also in a folder on my storage pool, which contains the content of the private key from the GUI SSH keypair

Thanks for all support, I’ll let you know if I can get things working like this…
And if there is interest, I might create a “resource” for my script (if that is still a thing in the new forum?)

9

If you would do it manually the public key would on the remote system go into:
/root/.ssh/authorized_keys

.ssh mode 700
authorized_keys mode 644 - it contains only public keys, never private ones

But instead better use the UI, edit the root user, paste the public key into the corresponding field, save.

2 Likes
10

But your “manual suggestion” is not persistent if I understood you well when you said that for example /root/bin isn’t “secure” to store scripts?

Anyway, I already have (and need) a working GUI keypair, so there is no reason not to use that one also on the command line (if no one objects to this - I’m not sure where I’ve read that someone advised against this)

11

Yes, so paste the public key of your GUI key pair into the matching field of the root user on the remote side.

2 Likes
12

That was already done by the semi-automatic ssh-connection wizard from the GUI :slight_smile:

As I said, it works… :wink:

14

I find zfs-autobackup superior to syncoid, more flexible, and it certainly easy to install as you just copy the software to a permanent location on disk, it’s not installed via apt or anything like that. I made a guide for it in the old forums. Think of it as just data, it’s not dangerous, will still run after a system update, etc. You would simply store your script in your data pool, anywhere, not /root/anything. You don’t necessarily need to use the UI to make keypairs, you could make them on the system to be replicated to and copy them over again to some data pool. I store all my scripts at /mnt/tank/Data/Scripts for example.

I have it not only mounting and doing a backup to external drive of certain datasets, but replicating to another system a different set of data, and, replicating to an offsite system a third set of data. Some datasets are stored on all 3, some are stored merely on 1 of them as it’s not that important of data. This is all in addition to using Kopia for a non replicated style of incremental backups.

2 Likes
15

Thanks for the advice!

I wanted to have a look at it, but I cannot find this guide that you speak about. The github instructions instruct to use pip, pipx or easy_install (which are not available on TrueNAS).
I’ve just downloaded the source and will try to see if I can simply run it from that… (I have more bash experience than python)

16

While it says 8 months, that was Sept 2023, so, 17 months now, not a single restart relplication from scratch issue. I have around 4TB replicated offsite offsite. System updates do not affect it as it uses a Python virtual environment.

1 Like
17

Thanks a lot for that link!
Not sure why I didn’t run into it myself while searching…

It was able to run the script (with the --version option) on my TrueNAS Scale by simply extracting the archive. But your method with a separate Python is probably a lot better :wink: (and I would have never figured that out from myself)

In meantime I’ve also started working on my own script, but I’m afraid it is a bit more complex then I’ve hoped for…

I’m doing ZFS replication from an encrypted filesystem to an encrypted filesystem and the initial full backup makes it complicated.
‘zfs recv -F’ doesn’t work with encrypted filesystem when there is no “previous snapshot”, so I need to delete my target backup dataset, before I can initiate the initial full backup.

cannot receive new filesystem stream: zfs receive -F cannot be used to destroy an encrypted filesystem or overwrite an unencrypted one with an encrypted one

I haven’t tried to script this yet, as I’m not sure that removing / recreating a dataset outside the GUI is such a good idea (perhaps other “house holding tasks” need to be done to keep TrueNAS healthy). Also I’m not sure if newly created dataset will inherit all properties from its parent or from the source dataset.

Are you sure that ZFS-Autobackup will properly handle all of this and keep my TrueNAS “healthy”? Are you using this script in combination with encryption?

Thanks

edit:
I just read in the manual that zfs-autobackup actually does it as well:
“If there is no snapshot in common, the only way to continue is to destroy the whole dataset (and all its snapshots) on the target and start from a full backup.”

So is it safe to do this in a script and outside of the GUI? How can we validate that this is safe? Does anyone perhaps know which “open” source file contains the TrueNAS replication code so we can check what it does? (I think replication from the TrueNAS GUI also does this)

18

Anything that is a wrapper for zfs send / recv, or uses it on the backend, is bound by the same limits and rules of ZFS and snapshots.

The option to “destroy and start all over” is basically saying “go ahead and just do this without prompting me”.

Otherwise, the replication will simply abort and nothing happens. (This might be preferable to some people, who would rather investigate why their destination could not receive an incremental replication, instead of just outright destroy the entire dataset.)

EDIT: Somewhat off-topic, but if someone is clever, they could incorporate automatic “ZFS bookmarks” on the source side, which can still allow for incremental replications, even if the source is missing a common base snapshot (e.g, it was pruned due to “expiring”.)

1 Like
19

I understand that it is best to do an incremental replication without destroying the target dataset first, but you do have to start initially with a full backup before you can proceed the next time with doing an incremental replication.

Also in case something goes wrong (e.g.: all common snapshots get deleted somehow), it may be required to (exceptionally) perform another full replication.

I also understand that most people will be ok with doing their (initial) full replication to a target pool that does not have the target dataset yet.

However, as I’ve explained earlier, I’m trying backup into a fully functional 2nd instance of my TrueNAS. Meaning that my backup TrueNAS has datasets that are shared with SMB, some datasets have apps, like Plex, using them, etc.
So I’ve fully configured my backup TrueNAS with empty datasets, that are shared in SMB, etc and now I’d like to ZFS replicate my data into those existing datasets.

Now I understand that if I do an initial replication using the TrueNAS GUI (which works), it probably also destroys and recreates the existing dataset (like zfs-autobackup and my own script need to do too).
However, what I don’t know is how TrueNAS GUI exactly does this?

  • Does it simply destroy and create a new dataset, inheriting all properties from the parent? (this would be easy to script)
  • Does it save all dataset properties of the existing/to-be-deleted empty dataset and create a new dataset restoring all those properties? (this would make scripting this quite complicated / annoying)
  • Does it do anything special for datasets that are already shared in SMB? What happens if you delete a dataset that is already shared in SMB and then recreate the same dataset? Will that pre-existing share then keep on working or does it need to be recreated as well?
  • Are there any other special “house holding tasks” that TrueNAS performs to keep things “healthy” when destroying / creating datasets?

If no one can answer these questions, then I see no other option then to do the initial replication using the GUI (to make sure that all remains healthy) and then try to switch to incremental scripted replication for the next runs.
This will probably work and be safe, but it is annoyingly overcomplicated and I’d rather prefer to use just 1 tool for both full and incremental replications…

20

The replication itself is meant to create the dataset on the target pool during the first full replication.

You cannot and should not do that. (Assuming those SMB shares will ever get written to.)

Please see this post.

I encourage you to understand some fundamentals about snapshots and replications before you start moving forward.

2 Likes
21

Yes, from the source. I think you mistakenly wrote “parent”, which means something else. (This is also highly inefficient and wasteful of time and resources.)

No. See above.

Do not make a replication/backup target dataset a path for an SMB share, unless it is strictly read-only. See my previous reply.

1 Like