Scale Rotating Backup Set Script

TrueNAS General
1

So, this may be useful to someone. It is the script I use to accomplish automated backups to external usb drives (backup sets actually). I have multiple backup sets, script supports an infinite number. I’ve been doing these backups for 2 years, never failed once. I store the backup drive in my bank vault (offsite). The idea here, is every so often, I disconnect the usb drive(s), take them to the bank, bring another set (a different pool name) back, and, plug it in. No other action is required. No need to import, export, type any passphrase, etc.

Yes, it’s usb and zfs, not ideal. But it works for me. And I rotate the sets so I have multiple backups. It’s also not my only backup method, just the easiest to restore the data I backup from since it’s local (my current set).

So, the idea here since my script likely will not be the same as yours is the beginning and end portions of this do the trickier part, import, mount, unlock and export the pool. The rest is up to you.

To accomplish this, I create a zfs pool on the external drive or drives using the gui or command line and I specify passphrase encryption. The pool name itself must start with the word “rotation”. I name mine rotation1, rotation2, etc. The assumption here is you know how to do this. Each pool is a backup set that can be rotated. And I put a sticky label on the drives so I know which drive is which backup set.

I have made no effort to pretty up this script, or, add options or whatever. Modify it as you see fit. If someone has a use and wants to make some of the replaced parts command line options, do it. I don’t have the time or desire to. This assumes you know how to do the backup commands, or replication. Replication will mean you have extra work to do if not using zfs-autobackup. You might just use the API to run a truenas replication task perhaps. Maybe someone wants to script that part, sorry, but I don’t use the truenas replication feature as it is too limited for me. But for rsync, tar, etc users, it would be simple to modify this.

Consider what to backup, maybe only the most critical data if it cannot fit in the ports or drives you have avialable. This may allow you to backup the most critical stuff here and other stuff to a cloud site or whereever. I do not do a complete backup of my system with this technique, just the most important data. If I had enough drives, ports, and big enough drives, I could using this technique.

Assumptions:

  1. Pool name must start with “rotation”, add characters to that name, each set should have a different name starting with rotation
  2. The pool has passphrase encryption, if you don’t want that remove the single line below the comment “Unlock pool”. If you do use it, embed the key in the same line, replace this with your key: replacemewithyourpassphraseorremoveline
  3. I use zfs autobackup, a non built in backup technique using zfs replication which is superior to the built in truenas replication, so, you likely do not. So, replace these 2 lines:
    cd /mnt/tank/Scripts/Autobackup
    autobackup-venv/bin/python -m zfs_autobackup.ZfsAutobackup -v -d --no-progress --clear-refreservation --exclude-received --keep-source=10,1d1w,1w1m,1m3m --keep-target=10,1d1w,1w1m,1m3m --allow-empty --clear-mountpoint --rollback --destroy-missing=30d --zfs-compressed --decrypt --encrypt $POOL $POOL
    with whatever backup command you want, tar, rsync,whatever. zfs autobackup handles all the snapshots, expirations, etc. for me so no need for me to code those in.
  4. Just unplug the drives when done with a set, and plug the next rotating set in. That’s it.
  5. Relies on a cron job to run it, you have to create this. I run mine overnight.
  6. Change the path as needed for the log file, that is this line: exec 1>/mnt/tank/Scripts/Logs/Rotation-Backup.log
  7. You have the skills to modify this script for your purposes.
  8. You can use the truenas ui to import the pool and copy stuff back should you need to.

The last thing this script does is export the pool, so, the drive(s) can be removed at any time the backup is not running. Since mine runs in the middle of the night, and my bank is closed, I never worry about unplugging the drives.

#!/bin/bash
exec 1>/mnt/tank/Scripts/Logs/Rotation-Backup.log
# Find which rotation drive is connected
POOLS=$(/usr/bin/midclt call -job pool.import_find)
GUID=$(echo $POOLS | jq -r '.[] | select(.name | startswith("rotation")) | .guid')
if [ -z "$GUID" ]; then
  POOLS=$(/usr/bin/midclt call pool.query '[["name", "^", "rotation"]]')
  GUID=$(echo $POOLS | jq -r '.[] | .guid')
  if [ -z "$GUID" ]; then
    POOLS=$(/usr/bin/midclt call pool.query '[["name", "^", "rotation"]]')
    GUID=$(echo $POOLS | jq -r '.[] | .guid')
    if [ -z "$GUID" ]; then
      # No disk connected
      echo "No disk connected"
      exit 0
    fi
  fi
  POOL=$(echo $POOLS | jq -r '.[] | select(.name | startswith("rotation")) | .name')
else
  POOL=$(echo $POOLS | jq -r '.[] | select(.name | startswith("rotation")) | .name')
  RESULT=$(/usr/bin/midclt call -job pool.import_pool '{"guid":"'$GUID'"}')
  if [ "$RESULT" != "True" ]; then
    # Could not import
    echo "Could not import any rotation pool"
    exit 0
  fi
fi
STATUS=$(/usr/bin/midclt call pool.query | /usr/bin/jq -r '.[] | select(.guid == "'$GUID'").status')
if [ "$STATUS" != "ONLINE" ]; then
  echo "Offline"
  exit 0
fi
# Unlock pool
/usr/bin/midclt call -job pool.dataset.unlock "$POOL" '{"key_file":false, "recursive":true, "datasets": [{"name":"'$POOL'", "passphrase":"replacemewithyourpassphraseorremoveline"}]}'
cd /mnt/tank/Scripts/Autobackup
autobackup-venv/bin/python -m zfs_autobackup.ZfsAutobackup -v -d --no-progress --clear-refreservation --exclude-received --keep-source=10,1d1w,1w1m,1m3m --keep-target=10,1d1w,1w1m,1m3m --allow-empty --clear-mountpoint --rollback --destroy-missing=30d --zfs-compressed --decrypt --encrypt  $POOL $POOL
sleep 30
POOLID=$(midclt call pool.query | /usr/bin/jq -r '.[] | select(.name=="'$POOL'").id')
/usr/bin/midclt call pool.export $POOLID
1 Like