Problem/Justification
When using ZFS replication with Include Dataset Properties enabled dataset properties including ZFS quotas are replicated to the destination.
This is particularly useful when using quotas that include child datasets and snapshots as it ensures that the backup dataset enforces the same storage limits as the primary dataset.
However this can cause replication failures when the source dataset reaches its quota.
For example:
-
A source dataset has a 1TB quota including snapshots and child datasets.
-
The destination dataset inherits the same 1TB quota through replication.
-
The source dataset reaches its quota and creates a new snapshot.
-
Replication attempts to transfer the new snapshot to the destination.
-
The destination cannot accommodate the incoming replication stream because it has also reached its 1TB quota.
-
Older destination snapshots that could potentially release sufficient space are not pruned until after replication completes.
Consequently replication fails even though deleting older snapshots could potentially free enough space to accommodate the incoming replication.
This is particularly problematic because the replication task itself is responsible for managing snapshot retention yet the retention process cannot release space before the replication operation that requires it.
An example error encountered:
cannot receive resume stream: destination tank/ZFS/Stuff space quota exceeded
Impact
Advantage:
-
Prevents avoidable replication failures when destination datasets reach their quota.
-
Makes replication more resilient when source and destination datasets have matching quotas.
-
Allows users to enforce consistent storage limits across primary and backup systems without requiring additional quota headroom.
-
Reduces the need for manual intervention to delete destination snapshots or increase quotas.
-
Makes snapshot retention more effective by considering the space requirements of incoming replication.
Disadvantage:
-
Additional logic would be required to determine which destination snapshots can safely be removed before replication.
-
Care would be needed to preserve snapshots required for incremental replication and resumable transfers.
-
Removing snapshots before a successful replication could reduce the available recovery history if the subsequent replication also fails.
User Story
When configuring a replication task with snapshot retention enabled the destination should ideally perform its retention/pruning operations before receiving new snapshots where this can be done safely.
The replication process should:
-
Identify destination snapshots eligible for deletion under the configured retention policy.
-
Preserve snapshots required for incremental replication and resumable transfers.
-
Delete eligible older snapshots before attempting to receive the incoming replication stream.
-
Proceed with replication once sufficient space is available.
The aim is to prevent replication failures caused purely by the ordering of snapshot retention and replication operations.