What is the proper way to send ZVOL data to S3

TrueNAS General
1

Hi there,
Me and me friend (both home-labers) do a so called “cloud storage share”, which in essence means that we have Site to Site VPN connectivity, between our routers, and we provide equal free space to each other via common protocols.

He requested SMB share, and I requested S3 share. So now here is my idea, to backup my VMs ZVOLs to this S3 bucket. And here comes the fun part, because currently (correct me if I’m wrong) TrueNAS Scale can do zfs send only to another TrueNAS filesystem.

So I asked the AI and he pointed me to a custom solution which uses rclone and zfs send, something like this:

zfs send --raw zpool-hdd-01/VMS/Encrypted/vm-100-disk-0@20260609 \
  | rclone --config /root/.config/rclone/rclone.conf \
    rcat s3-remote:your-bucket/vm-100-disk-0-20260609.zfs

Yeah, and first glance we can say, the volume has to be encrypted, and it is. Additionally I plan to use a custom script which iterates to all the ZVOLs, all of their snapshots and perform the aforementioned command.

The question is do you know where the rclone config is stored in TrueNAS filesystem? I understood from the AI, that “Cloud Sync task” uses rclone under the hood.

What other concerns you might have with this approach? What could go wrong =))

And last, do you think this can be wrapped as feature request? I mean - ability to send ZVOLs to the cloud by using “Cloud Sync Task”, by appending some parameters to the rclone.. Just wondering.

2

For these who are curious,

#!/bin/bash
set -uo pipefail
trap "err 'Interrupted by user'; exit 1" INT TERM

# Configuration
POOL_PATH="zpool-hdd-01/VMS"
S3_REMOTE="S3_IVAN_CRYPT"
S3_BUCKET="backup-kris"
S3_PREFIX="vm-backups"
RCLONE_CONFIG="/root/.config/rclone/rclone.conf"
RCLONE_PARAMS=(
  --config "${RCLONE_CONFIG}"
  --s3-chunk-size 64M
  --log-level ERROR
  --transfers 4
  --bwlimit 5M
)
SNAP_FILTER="auto-"
LOG="/var/log/zfs-s3-backup.log"
LOCK_FILE="/var/run/rclone-zfs-raw-backup.lock"

# Pre-flight checks
if [[ ! -f ${RCLONE_CONFIG} ]]; then
  err 'ERROR: Rclone config not found!'
  exit 1
fi

if ! grep -qw -- "${S3_REMOTE}" "${RCLONE_CONFIG}"; then
  err 'ERROR: S3_REMOTE not configured!'
  exit 1
fi

# Helpers
log()    { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "${LOG}"; }
info()   { log "INFO  $*"; }
err()    { log "ERROR $*"; }
rclone() { command rclone "${RCLONE_PARAMS[@]}" "$@"; }

if ! command -v flock >/dev/null 2>&1; then
  err 'ERROR: flock is required for single-instance protection!'
  exit 1
fi

exec 9>"${LOCK_FILE}"
if ! flock -n 9; then
  err 'Another instance is already running — exiting'
  exit 1
fi
trap 'rm -f "${LOCK_FILE}"' EXIT

s3_read()  { rclone cat "$1" 2>/dev/null | tr -d '[:space:]'; }
s3_write() { printf '%s' "$2" | rclone rcat "$1"; }

# Find matching ZVOLs
mapfile -t ZVOLS < <(
  zfs list -H -o name -t volume -r "${POOL_PATH}" \
    | grep -E '/vm-[0-9]+-disk-[0-9]+$'
)

if [[ ${#ZVOLS[@]} -eq 0 ]]; then
  info "No matching ZVOLs found under ${POOL_PATH}"
  exit 0
fi

info "Found ${#ZVOLS[@]} ZVOL(s)"
SUCCESS=0
FAILED=0

# Process each ZVOL
for ZVOL in "${ZVOLS[@]}"; do
  ZVOL_NAME=$(basename "${ZVOL}")
  S3_BASE="${S3_REMOTE}:${S3_BUCKET}/${S3_PREFIX}/${ZVOL_NAME}"
  STATE_FILE="${S3_BASE}/.last-snap"

  info "── ${ZVOL_NAME} ──"

  # Local auto-* snapshots, sorted chronologically
  mapfile -t ALL_SNAPS < <(
    zfs list -H -o name -t snapshot -r "${ZVOL}" \
      | grep "@${SNAP_FILTER}" \
      | sort
  )

  if [[ ${#ALL_SNAPS[@]} -eq 0 ]]; then
    info "  No ${SNAP_FILTER}* snapshots — skipping"
    continue
  fi

  info "  ${#ALL_SNAPS[@]} local snapshot(s)"

  # Determine starting point
  LAST_SENT=$(s3_read "${STATE_FILE}")
  PREV_SNAP=""
  START_IDX=0

  if [[ -z "${LAST_SENT}" ]]; then
    info "  No prior backup — full send from first snapshot"
  else
    info "  Last sent: ${LAST_SENT}"

    FOUND_IDX=-1
    for i in "${!ALL_SNAPS[@]}"; do
      if [[ "${ALL_SNAPS[$i]}" == "${ZVOL}@${LAST_SENT}" ]]; then
        FOUND_IDX=$i
        break
      fi
    done

    if [[ ${FOUND_IDX} -eq -1 ]]; then
      err "  Snapshot '${LAST_SENT}' no longer exists locally — chain broken, manual re-seed needed"
      (( FAILED++ ))
      continue
    fi

    START_IDX=$(( FOUND_IDX + 1 ))
    PREV_SNAP="${ZVOL}@${LAST_SENT}"

    if [[ ${START_IDX} -ge ${#ALL_SNAPS[@]} ]]; then
      info "  Already up to date"
      (( SUCCESS++ ))
      continue
    fi

    info "  $(( ${#ALL_SNAPS[@]} - START_IDX )) new snapshot(s) to send"
  fi

  # Send loop
  ZVOL_OK=true

  for (( i=START_IDX; i<${#ALL_SNAPS[@]}; i++ )); do
    SNAP="${ALL_SNAPS[$i]}"
    SNAP_NAME="${SNAP##*@}"

    if [[ -z "${PREV_SNAP}" ]]; then
      TYPE="full"
      S3_DEST="${S3_BASE}/${SNAP_NAME}.full.zfs"
      info "  Full send: ${SNAP_NAME}"
      SEND_CMD=(zfs send --raw "${SNAP}")
    else
      TYPE="incr"
      S3_DEST="${S3_BASE}/${SNAP_NAME}.incr.zfs"
      info "  Incremental: ${PREV_SNAP##*@} → ${SNAP_NAME}"
      SEND_CMD=(zfs send --raw -i "${PREV_SNAP}" "${SNAP}")
    fi

    if "${SEND_CMD[@]}" | rclone rcat "${S3_DEST}"; then
      info "  ✓ ${TYPE} OK: ${SNAP_NAME}"
      s3_write "${STATE_FILE}" "${SNAP_NAME}"
      PREV_SNAP="${SNAP}"
    else
      err "  ✗ ${TYPE} FAILED: ${SNAP_NAME}"
      ZVOL_OK=false
      break
    fi
  done

  if [[ "${ZVOL_OK}" == true ]]; then
    (( SUCCESS++ ))
  else
    (( FAILED++ ))
  fi
done

info "Done — success: ${SUCCESS}, failed: ${FAILED}"
[[ ${FAILED} -eq 0 ]] && exit 0 || exit 1

I’m executing this as a cron job each day at 8 o’clock. Rclone config looks like this:

root@sofx1010nas3012:~# cat .config/rclone/rclone.conf 
[S3_IVAN]
type = s3
provider = Other
access_key_id = GK..............82b
secret_access_key = 126.................10
endpoint = https://my-s3-endpoint.example.com
acl = private
region = kris

[S3_IVAN_CRYPT]
type = crypt
remote = S3_IVAN:backup-kris
password = cgp..............R8
password2 = DB..............IE

root@sofx1010nas3012:~#

You can extend the script to send an email or call a web hook in case of a failure, but I will leave this to your imagination.