After updating to dragonfish: first time running a replication task to my usb backup asks for encryption key. Can decrypt backup pool, but not backup dataset

giftcardreloaded · May 5, 2024, 5:47pm

Note - After writing this all up, it seems my problem is similar to this post. I’ll explain my situation, plus details from my source and backup encryption setup

What I’d like to know:

Is my replication task setup properly?
What can I do differently to unlock everything using one top-level key?
~~unless I can use keys from some other pool – so far no luck – I worry that I’m hosed when it comes to unlocking my backup dataset. Should I just wipe the external drive and start over?~~. (see my reply)

Explanation

As best I can recall:

I’ve been running Truenas scale bluefin + keeping regular backups on an external usb drive. The backup procedure has been trouble-free while doing things bluefin-to-bluefin.
I upgraded to Dragonfish a few days ago
Just now I’ve tried importing the external usb pool to run my periodic update
Truenas prompted me to decrypt my usb pool PLUS its top-level data set.
I had my encryption key for the usb pool, but it’s just a single key. I’m not sure I ever exported the key for the dataset.

Am I pretty much boned here? The only thing I can think to do is wipe the usb drive and redo the whole backup, and pay attention to keys this time.

The fact I’m even in this position and asking this question makes me think I don’t understand the encrypted back up procedure as well as I should.

Additional details from following advice in the other thread

– see attached pics for the replication task settings–

zfs list -r -t filesystem -o name,encryptionroot,keyformat pool1

root@truenas[/home/admin]# zfs list -r -t filesystem -o name,encryptionroot,keyformat pool1     
NAME                                                    ENCROOT  KEYFORMAT
pool1                                                   pool1    hex
pool1/.system                                           pool1    hex
pool1/.system/configs-ae32c386e13840b2bf9c0083275e7941  pool1    hex
pool1/.system/cores                                     pool1    hex
pool1/.system/ctdb_shared_vol                           pool1    hex
pool1/.system/glusterd                                  pool1    hex
pool1/.system/netdata-ae32c386e13840b2bf9c0083275e7941  pool1    hex
pool1/.system/rrd-ae32c386e13840b2bf9c0083275e7941      pool1    hex
pool1/.system/samba4                                    pool1    hex
pool1/.system/services                                  pool1    hex
pool1/.system/webui                                     pool1    hex
pool1/appdata                                           pool1    hex
pool1/archive                                           pool1    hex
pool1/backups                                           pool1    hex
pool1/docker                                            pool1    hex
pool1/docker/data                                       pool1    hex
pool1/docker/stacks                                     pool1    hex
pool1/ix-applications                                   -        none
pool1/ix-applications/catalogs                          -        none
pool1/ix-applications/default_volumes                   -        none
pool1/ix-applications/k3s                               -        none
pool1/ix-applications/k3s/kubelet                       -        none
pool1/ix-applications/releases                          -        none
pool1/jailmaker                                         pool1    hex
pool1/jailmaker/jails                                   pool1    hex
pool1/jailmaker/jails/docker                            pool1    hex
pool1/library                                           pool1    hex
pool1/library/movies                                    pool1    hex
pool1/library/music                                     pool1    hex
pool1/library/shows                                     pool1    hex
pool1/library/vg                                     pool1    hex
pool1/memes                                             pool1    hex
pool1/photos                                            pool1    hex
pool1/projects                                          pool1    hex
pool1/shared                                            pool1    hex
pool1/temp                                              pool1    hex
pool1/users                                             pool1    hex
pool1/users/XXXX                                       pool1    hex
pool1/users/YYYY                                     pool1    hex
root@truenas[/home/admin]#

For my external HDD:

NAME                                                                    ENCROOT                KEYFORMAT
usb-zfs-backup                                                          usb-zfs-backup         hex
usb-zfs-backup/backup                                                   usb-zfs-backup/backup  hex
usb-zfs-backup/backup/.system                                           usb-zfs-backup/backup  hex
usb-zfs-backup/backup/.system/configs-ae32c386e13840b2bf9c0083275e7941  usb-zfs-backup/backup  hex
usb-zfs-backup/backup/.system/cores                                     usb-zfs-backup/backup  hex
usb-zfs-backup/backup/.system/ctdb_shared_vol                           usb-zfs-backup/backup  hex
usb-zfs-backup/backup/.system/glusterd                                  usb-zfs-backup/backup  hex
usb-zfs-backup/backup/.system/netdata-ae32c386e13840b2bf9c0083275e7941  usb-zfs-backup/backup  hex
usb-zfs-backup/backup/.system/rrd-ae32c386e13840b2bf9c0083275e7941      usb-zfs-backup/backup  hex
usb-zfs-backup/backup/.system/samba4                                    usb-zfs-backup/backup  hex
usb-zfs-backup/backup/.system/services                                  usb-zfs-backup/backup  hex
usb-zfs-backup/backup/.system/webui                                     usb-zfs-backup/backup  hex
usb-zfs-backup/backup/appdata                                           usb-zfs-backup/backup  hex
usb-zfs-backup/backup/archive                                           usb-zfs-backup/backup  hex
usb-zfs-backup/backup/backups                                           usb-zfs-backup/backup  hex
usb-zfs-backup/backup/docker                                            usb-zfs-backup/backup  hex
usb-zfs-backup/backup/docker/data                                       usb-zfs-backup/backup  hex
usb-zfs-backup/backup/docker/stacks                                     usb-zfs-backup/backup  hex
usb-zfs-backup/backup/ix-applications                                   -                      none
usb-zfs-backup/backup/ix-applications/catalogs                          -                      none
usb-zfs-backup/backup/ix-applications/default_volumes                   -                      none
usb-zfs-backup/backup/ix-applications/k3s                               -                      none
usb-zfs-backup/backup/ix-applications/k3s/kubelet                       -                      none
usb-zfs-backup/backup/ix-applications/releases                          -                      none
usb-zfs-backup/backup/jailmaker                                         usb-zfs-backup/backup  hex
usb-zfs-backup/backup/jailmaker/jails                                   usb-zfs-backup/backup  hex
usb-zfs-backup/backup/jailmaker/jails/docker                            usb-zfs-backup/backup  hex
usb-zfs-backup/backup/library                                           usb-zfs-backup/backup  hex
usb-zfs-backup/backup/library/movies                                    usb-zfs-backup/backup  hex
usb-zfs-backup/backup/library/music                                     usb-zfs-backup/backup  hex
usb-zfs-backup/backup/library/shows                                     usb-zfs-backup/backup  hex
usb-zfs-backup/backup/library/vg                                     usb-zfs-backup/backup  hex
usb-zfs-backup/backup/memes                                             usb-zfs-backup/backup  hex
usb-zfs-backup/backup/photos                                            usb-zfs-backup/backup  hex
usb-zfs-backup/backup/projects                                          usb-zfs-backup/backup  hex
usb-zfs-backup/backup/shared                                            usb-zfs-backup/backup  hex
usb-zfs-backup/backup/temp                                              usb-zfs-backup/backup  hex
usb-zfs-backup/backup/users                                             usb-zfs-backup/backup  hex
usb-zfs-backup/backup/users/XXXX                                       usb-zfs-backup/backup  hex
usb-zfs-backup/backup/users/YYYY                                     usb-zfs-backup/backup  hex
root@truenas[/home/admin]#

When I go to export my source pools keys (all keys), there is only one key in the json file.
@winnielinnie 's advice:

You need to open up your main pool’s “keys” file in a text editor. (It’s the file that ends with the .json extension.)

Within, you will find the relevant HEX strings, which you can copy+paste to manually unlock the backup datasets. After you unlock all of them, then you can export a working “keys” file (.json) for the “HDD-backup” pool.

If you’ve made it this far thanks for sticking with me!

giftcardreloaded · May 5, 2024, 5:53pm

Alright, nothing like posting for a forum to answer your own question.

I followed @winnielinnie 's advice to the letter and pasted in my single root encryption key to the child dataset (also selected Force) and it worked

My remaining question:

What about my replication task made this necessary?

In the OP, it was said that

It looks like the way you configured (and executed) your Replication Tasks have cleaved dependent datasets from their original “encryptionroot”.

I’d like to fix that if possible. Any suggestions welcome.

winnielinnie · May 5, 2024, 6:03pm

Short answer (until I figure out a way to explain this better) is that this is an oversight (my opinion) of ZFS’s original design.

The way I work through this is by using what I call “pseudo-roots”.

You can read more about it here and from the old forums in here.

Doing so essentially allows you to better “group” and “nest” familial datasets together, such as all those sharing the same encryptionroot, in which replicating the “pseudo-root” to a backup pool will retain the same encryption/encryptionroot relationship.

However, this doesn’t overcome the issue of using the main pool’s “keys” file to unlock the datasets on the backup pool. This is because the .json file generated when you export your keys reference the name of the main pool. Hence, why you must unlock (at least once) the encryptionroot(s) on the backup pool, so that you can export the “keys” from there, which will have proper names in the .json file.

(This particular quirk is not really an issue of ZFS, but just the nature of different pools using different names. On that note, if you’ve “cleaved” the child datasets from their encryptionroot, it will also change the resultant layout of the exported .json file.)

giftcardreloaded · May 5, 2024, 7:04pm

Thanks, winnie. I spend some time thinking through this.