Support for ZFS "Bookmarks"

[oxyde]

Problem/Justification
Replication tasks depend on the presence of a common snapshot on the source and destination. If replication is interrupted longer than the snapshot retention window on the source, the base snapshot may be deleted… so to restore the backup strem a replication from scratch is needed, with all the negative impact.

ZFS bookmarks instead allow incremental replication to continue even after the original snapshot has been destroyed, with a cost in pool terms space really negligible

Impact

• reduce the need of a replication from scratch, that can be a nightmare when offsite NAS are involved
• costs nothing in terms of pool space
• ideally, people not interested in this capability should be able to not select this feature

User Story
I became aware of this capability purely by chance, thanks to the resource written by @winnielinnie ©

I have experienced the feel to miss the snapshot retention window for various reasons, which forced me to restart replication from scratch. This resulted in significant time loss and, in one case, required physically relocating the destination system onto the same network as the source to complete a full re-replication smoothly.

As a TN user, I would benefit from built-in bookmark support in replication tasks to prevent these situations.

[winnielinnie]

My “vote” is a thumbs up.

There could be 2 parts to this feature.

1. Manual bookmarks. When manually creating or viewing snapshots, there can be an option for “Bookmark”. If the box is checked, a bookmark of the same name will be created. If the box is unchecked, the bookmark will be destroyed. It should have a tooltip near the “Bookmark” checkbox to explain what a ZFS bookmark is to the user. (It would be useful to have a batch operation as well if multiple snapshots are selected.)
2. Automatic for Replication Tasks. A Replication Task can have an option to “Create bookmark of latest source snapshot after successful replication”.

---

Using the bookmark is more complicated.

For now, it could be the user’s responsibility to manually use a bookmark in the command-line to do an incremental replication that would otherwise require starting all over, after which they can return to normal replications in the GUI. (I have successfully done this myself in a test scenario. Thankfully, so far, I never needed to use a fallback for my real backups.)

It’s possible for this to be automated too. There could be a fallback option to use a bookmark. “If incremental replication is not possible, allow checking for and using a bookmark to attempt an alternative way of doing an incremental replication?” This option could be placed under the “From Scratch” option in the GUI. For safety’s sake, enabling the “Bookmark Fallback” option should disable and grey out the “From Scratch” option.

---

Syncoid, a command-line tool, allows this with the --create-bookmark option:

--create-bookmark

This argument tells syncoid to create a zfs bookmark for the newest snapshot after it got replicated successfully. The bookmark name will be equal to the snapshot name. […] This can be very useful for irregular replication where the last matching snapshot on the source was already deleted but the bookmark remains so a replication is still possible.

AFAIK, this still leaves it up to the user to manually use a bookmark for an attempted fallback incremental replication, if required. (I don’t use Syncoid, so I’m not sure if they’ve improved on this feature in their latest version.)

[Schuwi]

@winnielinnie

Using the bookmark is more complicated.

Why should it not be used automatically in case the source snapshot no longer exists? In the end, as far as I understand, zfs send doesn’t care where it gets the snapshot metadata from. It mainly needs the TXG and GUID to generate an incremental stream.

I just confirmed it in the ZFS source.

Call stack:

• zfs_do_send (CLI - cmd/zfs/zfs_main.c:4780@8f6f4bc),
• calls: zfs_send_one (libzfs - lib/libzfs/libzfs_sendrecv.c:2877@8f6f4bc),
• spawns in new thread: zfs_send_one_cb(libzfs - lib/libzfs/libzfs_sendrecv.c:2869@8f6f4bc),
• ultimately calling: lzc_send_redacted(libzfs - lib/libzfs_core/libzfs_core.c:819@8f6f4bc),
• ultimately spawns in new thread: lzc_send_resume_redacted_cb(libzfs - lib/libzfs_core/libzfs_core.c:892@8f6f4bc),
• ultimately calls kernel with ioctl: zfs_ioc_send_new (kernel module - module/zfs/zfs_ioctl.c:7236@8f6f4bc),
• calling: dmu_send(kernel module - module/zfs/dmu_send.c:2781@8f6f4bc),
• where the bookmark metadata is extracted from a snapshot, or simply loaded from the given bookmark into local variable zfs_bookmark_phys_t *zb (kernel module - module/zfs/dmu_send.c:2904-2994@8f6f4bc)

module/zfs/dmu_send.c

So even if given a snapshot it just extracts the same metadata the corresponding bookmark would hold. I don’t see a reason not to fall back to a bookmark.

The only hindrance could be that zfs send -R does not yet support bookmarks as you pointed out in your explanatory post. I’m not sure whether TrueNAS uses the -R option internally or whether they iterate the children datasets manually.

[winnielinnie]

@Schuwi

I’m not sure whether TrueNAS uses the -R option internally

The -R flag is invoked if you select “Full Filesystem Replication” in the configuration page.

@Schuwi

or whether they iterate the children datasets manually.

It iterates the children manually if you select the “Recursive” option. This is not a native ZFS feature, but rather something TrueNAS does on its own.

---

TrueNAS also behaves differently, depending on if you select a single source dataset or multiple source datasets. If you select multiple source datasets, it will generate a separate stream per dataset and nest each one beneath the specified destination dataset. If you select a single source dataset, it will target the destination dataset itself without nesting it one level below.

This is why users bump into problems with TrueNAS when using replication. An easy fix is to manually type in the name of the destination dataset^[1], rather than “select” the folder icon, so that the replication’s first run will create the dataset on the destination.

---

1. It’s not obvious, but the destination side has a text field that can be typed into. Clicking the folder icons is not mandatory and it misleads users into thinking they must select an existing dataset, rather than have a new one get created. ↩︎

[PhilD13]

@winnielinnie

It’s not obvious, but the destination side has a text field that can be typed into. Clicking the folder icons is not mandatory and it misleads users into thinking they must select an existing dataset, rather than have a new one get created.

I actually did not know it would create a dataset.

[winnielinnie]

I feel that the TrueNAS implementation of ZFS replications is “incomplete” and unclear.

When you manually use zfs recv in the command-line, the dataset you type is the target for the replication.^[1]

cold/backup is different than cold/backup/media. If the dataset “media” does not exist on the destination, it will be created the first time the replication runs.

tank/media → cold/backup means that the destination’s “backup” dataset will reflect the source’s “media” dataset, just with a different name.

tank/media → cold/backup/media means that a new dataset named “media” will be created (if it doesn’t already exist), which will reflect the “media” dataset from the source.

---

1. This assumes you’re not using the -d or -e flags. ↩︎