Perform destination snapshot retention before replication to prevent quota related replication failures

Problem/Justification

When using ZFS replication with Include Dataset Properties enabled dataset properties including ZFS quotas are replicated to the destination.

This is particularly useful when using quotas that include child datasets and snapshots as it ensures that the backup dataset enforces the same storage limits as the primary dataset.

However this can cause replication failures when the source dataset reaches its quota.

For example:

  • A source dataset has a 1TB quota including snapshots and child datasets.

  • The destination dataset inherits the same 1TB quota through replication.

  • The source dataset reaches its quota and creates a new snapshot.

  • Replication attempts to transfer the new snapshot to the destination.

  • The destination cannot accommodate the incoming replication stream because it has also reached its 1TB quota.

  • Older destination snapshots that could potentially release sufficient space are not pruned until after replication completes.

Consequently replication fails even though deleting older snapshots could potentially free enough space to accommodate the incoming replication.

This is particularly problematic because the replication task itself is responsible for managing snapshot retention yet the retention process cannot release space before the replication operation that requires it.

An example error encountered:

cannot receive resume stream: destination tank/ZFS/Stuff space quota exceeded

Impact

Advantage:

  • Prevents avoidable replication failures when destination datasets reach their quota.

  • Makes replication more resilient when source and destination datasets have matching quotas.

  • Allows users to enforce consistent storage limits across primary and backup systems without requiring additional quota headroom.

  • Reduces the need for manual intervention to delete destination snapshots or increase quotas.

  • Makes snapshot retention more effective by considering the space requirements of incoming replication.

Disadvantage:

  • Additional logic would be required to determine which destination snapshots can safely be removed before replication.

  • Care would be needed to preserve snapshots required for incremental replication and resumable transfers.

  • Removing snapshots before a successful replication could reduce the available recovery history if the subsequent replication also fails.

User Story

When configuring a replication task with snapshot retention enabled the destination should ideally perform its retention/pruning operations before receiving new snapshots where this can be done safely.

The replication process should:

  1. Identify destination snapshots eligible for deletion under the configured retention policy.

  2. Preserve snapshots required for incremental replication and resumable transfers.

  3. Delete eligible older snapshots before attempting to receive the incoming replication stream.

  4. Proceed with replication once sufficient space is available.

The aim is to prevent replication failures caused purely by the ordering of snapshot retention and replication operations.

Makes sense for normal growth, but I’d want one guard on it. If the source gets hit by ransomware the next snapshot is huge because almost every block changed, and that’s often exactly when you hit the quota. If retention runs first to make room for that stream, it deletes the older snapshots on the backup box, the ones you actually want, to fit the encrypted ones.

So I’d only prune first when the incoming stream looks normal. A dry run send (zfs send -nv) tells you roughly what will be sent. Compare that to the last few runs and if it’s way bigger, fail and alert instead of pruning. In that case the quota error is doing you a favour.

Disclosure: I’m founder and CEO of CyberSnap.

The idea is that it’s pruning snaps that have already expired its just about the order its done in. Currently send most recent snap/s then prune I’m suggesting prune then send most recent snap/s.

There is no current logic in TrueNAS that says don’t prune a snapshot when ransomeware attacks it just prunes on your defined schedule. If your schedule/s are poor and don’t give you enough recovery points then that’s a different issue.

I find it best to use the grandfather, father, son method when configuring snapshot schedules.

every hour for a day
every day for two weeks
every week for a month
every month for a year

Fair, it’s only expired snaps either way, so the order doesn’t change what gets deleted. The part I’d miss is a side effect. Today an oversized stream hits the quota and fails, and that error can be the first thing anyone notices. With prune first the room gets made and the big stream just lands quietly. So if they change the order I’d still want a size check before the send, even just as an alert. Your GFS layout covers a lot of it, the weekly and monthly ones are what you’d go back to if nobody notices for a few days.

Disclosure: I’m founder and CEO of CyberSnap.