Restore old snapshot, while still keeping intermediate/newer snapshots of dataset?

TrueNAS General
1

Hi,

Consider a dataset appdata with snapshots in pool main:

main/appdata@s1
main/appdata@s2
main/appdata@s3 # newest snapshot

Now, I’d like to restore old snapshot main/appdata@s2. This should result in appdata content to be same as main/appdata@s2. The kicker is: I still need to keep main/appdata@s3 for testing purposes.

Two remaining alternatives seem to exist:

  1. Copy over needed files from main/appdata/.zfs/snapshot/s2 back to dataset.
  2. Clone main/appdata@s2; promote and rename this clone to replace original dataset.

If I am not mistaken, ZFS rollback does not work, as it deletes all newer, intermediate snapshots (and can cause errors with ZFS incremental replication).

Alternative 1

This sounds like simplest solution. I could do

rm -rf main/appdata/big_folder 
cp -a main/appdata/.zfs/snapshot/s2/big_folder main/appdata

, but that probably isn’t optimal (size is 200GB). I guess

zfs send main/appdata@s2 | zfs recv -F main/appdata

would be much more time-efficient, leveraging ZFS block transfers?

Alternative 2

Snapshot cloning and promoting can be done as shown in TrueNAS docs. Or via command line:

zfs clone main/appdata@s2 main/appdata-clone
zfs promote main/appdata-clone

This clone needs to be renamed to original dataset name (not possible with web app):

zfs rename main/appdata main/appdata-old
zfs rename main/appdata-clone main/appdata
# optionally later on ...
# zfs destroy -r main/appdata-old

Is this rename operation OK without causing any inconsistencies in TrueNAS / do I need to stop any services before?

Being relatively new to TrueNAS/ZFS, I’d appreciate confirmation, if my understanding is correct so far and how to do proceed. Thanks!

TrueNAS SCALE 24.10

2

What about cloning s3 to appdata-test and then rolling back appdata to s2?
Simple and no rename.

3

Hey @etorix , thanks for your answer.

That probably might work as well, but my example was an over-simplification. Actually there are some more snapshots between s2 and s3, which I would like to keep.

appdata dataset name is required, so in order to test s3, I would need to rename appdata-test back. Which brings me back to original question :slight_smile: .

–

Edit:
Not sure if I got your example right - but is it possible at all to do a rollback of s2, if a clone still depends on s3 as intermediate snapshot?

4

A clone is an autonomous independent entity, and does not depend upon the original.

Back to your question, the choice is yours but renaming may interfere with replication tasks, and the manual reshuffling of data looks like an invitation to make mistakes.

5

With block-cloning enabled, it shouldn’t consume any extra space, as long as the dataset is not encrypted.

EDIT: “shouldn’t”

1 Like
6

A clone does depend on the original. You cannot destroy the snapshot that you used to create the clone.

Source:
https://openzfs.github.io/openzfs-docs/man/master/7/zfsconcepts.7.html#Clones

2 Likes
7

You would need to promote the clone in order to make it autonomous.

3 Likes
8

Wow, this lead to a rabbit hole :wink: .
At first I thought, data automatically would be de-duplicated, when cp -a from a snapshot to live dataset, similar to moving data within one dataset. But apparently, snapshots have their own filesystem, so this is not comparable to ordinary moving.

AI told me to use cp -al, which is absolutely non-sense. Then I found Hardlinks accross Datasets? and by using cat /sys/module/zfs/parameters/zfs_bclone_enabled I discovered, that block cloning already is enabled. … which confused me, because zfs list and TrueNAS UI showed me double disk size, when copying a 1GB dummy file from snapshot to live dataset.

Fortunately above thread advised to use zpool list , which is the only command that shows real diskspace … duh!

// Edit:
@winnielinnie Can you elaborate on the “as long as the dataset is not encrypted” part a bit?

  • Block cloning within one dataset (even if encrypted) should always work. Does this also hold true , when copying from a snapshot to same live dataset, as in my case?
  • So block cloning across different datasets does not work. Does it matter, if both source and destination datasets are separately encrypted (form their own encryption root) or are children of a common parent encryption root dataset?
9

Yes, agree with you. I guess the best option is to just cp -a, and not using clones + promoting on an ZFS appliance. Fortunately there is block cloning to save disk space.

Btw: Just forget about this command, it does not work.

I thought this would be more performant than the cp -a alternative (suggested by AI as well…), but now could verify on a test system, this errors, when using it on same dataset:

cannot receive new filesystem stream: destination has snapshots (eg. main/testing@manual-1) must destroy them to overwrite it

A think to consider against cp is that it does not leverage native ZFS capabilities, uses plain filesystem and may take long time to complete depending on the file/directory to be copied. In this case it might make sense to just use zfs rollback.

10

It has nothing to do with moving files within a filesystem. Block-cloning is handled at the pool level. It’s different than hardlinks or moving.

You can use block-cloning across datasets of the same pool, as long as they are unencrypted and share the same recordsize.

You need to check with zpool list <poolname> to see how much actual space is consumed. You can also check with zpool get bcloneused <poolname>.

This is because block-cloning is handled at the pool level. The zfs command and GUI are unaware of cloned blocks and will report what is perceived from an isolated filesystem.

It can work within a dataset, but it cannot work across datasets if one of them is encrypted. (This is impossible because the master key is unique.) Technically, you should be able to use block-cloning from an encrypted dataset’s snapshot into its live filesystem, but the last time I checked, this didn’t work. (It might have changed since OpenZFS 2.2 was introduced.)

It does, but only if the datasets are unencrypted and share the same recordsize.

It doesn’t matter if they share the same encryptionroot. The encryptionroot is a “logical” relationship between datasets that are unlocked/locked atomically and share the same user key and iterations. Each dataset, though, has its own unique randomly-generated master key. It is the master key that encrypts/decrypts. Not the user key.

2 Likes
11

So block-cloning won’t happen for my system between different datasets, as all are encrypted/reside as children under parent encryption root.

It would be interesting to know, if copying from main/appdata@s3 to main/appdata (encrypted DS) leads to duplication or not. For now, I only had an unencrypted test system. Probably will check that out as well :slight_smile: .

Thanks for this detailed answer!

12

There’s no technical reason why this wouldn’t work, but I remember reading on the OpenZFS Github, around the time of the 2.2 release, that this particular use-case is an exception. You are correct, it is “across filesystems”, since every snapshot is treated as a filesystem. Yet this does not prevent you from using block-cloning across datasets (filesystems) under the same pool. For some reason, the presence of “encryption + different filesystem” falls back to a normal copy operation.

You can test it out on SCALE 24.10.

  1. Create an encrypted dataset.
  2. Make a large 4GB file inside a folder in the new dataset.
  3. Take a snapshot.
  4. Delete the file.
  5. Use cp to copy the file from the hidden snapshot directory into the live filesystem folder.
  6. After about 10 seconds, check with zpool get bcloneused
13

Good to know. Yes, this test is on my todo-list!

In the meanwhile I lost patience using the cp -a main/appdata/.zfs/snapshot/s2/big_folder main/appdata approach to restore an old snapshot.

Removing and copying 200GB via plain filesystem cp call just took too long (*). I gave up, deleted those newer snapshots, used the ZFS rollback function and had an immediate filesystem update.

It’s my impression that a quick “restore snapshot to live dataset” feature would be useful in TrueNAS, when rollback is not an option. ZFS provides Cloning + Promoting exactly for this use case, but currently TrueNAS can’t rename datasets via web interface, which makes it manual driven and too error-prone operation for my taste.

(*) don’t know, if block cloning in general has huge performance benefits and whether it was enabled for this particular snapshot → live dataset copying case.

14

It sounds like it wasn’t being used, since it should have finished quickly.

I suspect this is why:

15

Yeah, unfortunate (possible) limitation :frowning: .

16

Interesting, block cloning seems active with recent versions, when copying from snapshot to same encrypted dataset:

root@truenas[/mnt/main/encrypted]# zfs get encryption main/encrypted
NAME            PROPERTY    VALUE        SOURCE
main/encrypted  encryption  aes-256-gcm  -

root@truenas[/mnt/main/encrypted]# zpool list -o name,size,alloc,free,bcloneused,bclonesaved main
NAME   SIZE  ALLOC   FREE  BCLONE_USED  BCLONE_SAVED
main  4.50G  1010M  3.51G            0             0

root@truenas[/mnt/main/encrypted]# ls -sh .zfs/snapshot/manual/random-bin
1001M .zfs/snapshot/manual/random-bin

root@truenas[/mnt/main/encrypted]# cp .zfs/snapshot/manual/random-bin .

root@truenas[/mnt/main/encrypted]# ls -sh
total 1001M
1001M random-bin

root@truenas[/mnt/main/encrypted]# zpool list -o name,size,alloc,free,bcloneused,bclonesaved main
NAME   SIZE  ALLOC   FREE  BCLONE_USED  BCLONE_SAVED
main  4.50G  1010M  3.51G        1000M         1000M

Great thing. Maybe that previous cp took so long due to amount of small files in that directory.

I also tried zfs rename via CLI and it works like a charm in TrueNAS. Definitely will pick up Clone + Promote + Rename in my toolbox as alternative to zfs rollback.

→ Add ability to rename datasets

1 Like
17

That was a possibility, but I didn’t assume you had that many small files in a folder.

Now you know that you can “recover” files from a snapshot into the live filesystem without consuming extra space.[1]

Lucky you. Those who are on Core will never be able to use block-cloning the way you did, since it is only supported starting with FreeBSD 14. :frowning_face:

  1. The application or tool needs to support this. Modern versions of Coreutils (cp) will work. I’m not sure about Windows File Explorer. I know that KDE’s Dolphin file manager does not invoke block-cloning, sadly. ↩︎

18

No worries, me neither :grin: . “Many” is relative, but with ~ 200k files the rm -rf bigfolder pre-cleanup step already took too long for my patience in context of quick snapshot testing. That hasn’t anything to do with block cloning, but illustrates, we are relying on filesystem-based tools and their compatibility with COW here. If you compare that with instant effects of ZFS-native operations, the difference is huge. At least I became a cloning fan now. Thanks again for you help!

1 Like