Encryption key weirdness between host and backup pools. host is different from backup, but host unlocks part of backup data

[giftcardreloaded]

Sorry in advance for the novel. I’m trying to answer the following 3 questions and I hope it’s not too confusing

1. How did my Host encryption key change?
2. Why is my Host encryption key different from my backup pool encryption key?
3. Why do I need to provide my host key to unlock the root dataset of my usb backup pool?

Defining some terms and process first. This might explain the behaviors I see

• My host pool is called pool1.

• pool1 has a bunch of child datasets and folders holding my data.

• I have a replication task that backs up pool1 to a USB drive.

• The USB drive pool is called USB-ZFS-BACKUP.

• USB-ZFS-BACKUP has a root dataset called Backups,

• pool1 children get written to backups dataset on the USB drive.

(See the pic that outlines this. curious if this ultimately explains the following issues)

---

Ok, now my problem:

This post is semi-related to a previous issue where I needed to manually paste encryption keys to decrypt my usb backup pool.

I was having that issue again, but this time the USB encryption key (Z in the below table) wasn’t working to decrypt the usb root dataset (Backups). I actually found I needed to provide the host encryption key( Y) to decrypt Backups. This surprised me. I thought Z would decrypt everything – whether having to manually paste it or not.

Along the way I decided to export and compare my truenas keys with some earlier key exports. Here’s a table to summarize what I found

	Host Pool (one key / all keys)	Backup Pool(one key / all keys)
Last week’s keys	Y / X	Z / Z
This week’s keys	Y / Y	Z / Z

Observations and questions

1. The encryption key for my usb backup is entirely different from my host key. Is this expected?

2. The encryption key for my Host pool somehow changed last week from X to Y depending on whether I had exported a single key or all keys. I have no idea why my host key changed. The timestamp on the 'All keys" and “single key” json file is 5 minutes apart. I don’t remember doing anything that would have changed the key. How is this possible?

3. The process of unlocking my USB drive pool and its root-level dataset (Backups) required using the USB encryption key Z and then my host encryption key Y, respectively. Is this expected?

I would have expected that decrypting my USB backup pool would ONLY require the Z key. But maybe there’s something about how I wrote pool1 children content into the backups dataset meant the encryption key was inherited and so there are two different keys to use?

---

I hope this isn’t too confusing - it’s taken be a bit to write this up as plainly as I can. Happy to provide additional details if any of this is unclear. I’m scracthing my head over this.

paging @winnielinnie in case you can shed some light on this.

[winnielinnie]

I’m trying to follow, and this is not a dig against you (trust me, the way ZFS was designed with its terminologies does no one any favors), but I’m unable to get a clear picture of what is going on.

Pools don’t get encrypted. A pool doesn’t have encryption keys. Only datasets do. (The top-level root dataset, which has the same name as the pool, is a dataset at the end of the day. The fact that the GUI reads “export pool encryption keys” lends to the confusion.)

A replication cannot change properties or do anything to a destination dataset that the administrator could not do anyways. (Think of a replication as not only transferring data, but also applying zfs commands; no different than ssh’ing into a server and manually changing your dataset’s properties.)

If the destination dataset is locked? Then you cannot change its encryption key, even if you SSH into the remote server as the root user. (Nor can you send data to it.)

How would you unlock it? By providing the HEX-string key.

Where is this key stored? On your local server.

How was this HEX-string originally generated? It depends. Maybe you had it randomly generated upon the dataset’s creation, or you provided one yourself (copy+paste?).

---

With that said, I can try to answer some questions.

@giftcardreloaded

The encryption key for my usb backup is entirely different from my host key. Is this expected?

The root dataset, or the nested child underneath?

USB-ZFS-BACKUP
or
USB-ZFS-BACKUP/Backups

---

@giftcardreloaded

The encryption key for my Host pool somehow changed last week from X to Y depending on whether I had exported a single key or all keys.

There’s no key for the pool. Did you mean your root dataset?

---

@giftcardreloaded

The process of unlocking my USB drive pool and its root-level dataset (Backups) required using the USB encryption key Z and then my host encryption key Y, respectively. Is this expected?

What do you mean by this?

---

To help get a better understanding, you can see what encryptionroots you have. Only encryptionroots get unlocked.

If you have a single encryptionroot (let’s say the top-level root dataset), then it means unlocking it atomically unlocks all encryption-inherited children.

To demonstrate this point, if you have an encrypted child underneath an encryptionroot (which it “inherits” its encryption properties), you cannot “unlock” the child. You must unlock the encryptionroot, which decrypts all datasets that are part of this “encryption family”.

zfs list -r -t filesystem -o name,encryptionroot

[giftcardreloaded]

Thanks for trying to follow :). I’ll try to be a little more precise

Pools don’t get encrypted. A pool doesn’t have encryption keys. Only datasets do […] The fact that the GUI reads “export pool encryption keys” lends to the confusion.

I think this is what got me. In both host and backup pools I am clicking the top-level pool to get the option to export the encryption keys

I think this is a case of RTFM, but I’ll just be ignorant in public:

1. If pools don’t get encrypted – and child datasets are individually encrypted – then how can you ensure that all datasets are encrypted with the same key? To try and answer my own question, it might be the root dataset of the pool (with the same name) that gets encrypted. Then all the children datasets within it inherit that encryption key.

As shown in the above pic, pool1 has a a ton of child datasets. the USB-ZFS-BACKUP pool has a single child dataset (Backup – oops no plural) which I chuck everything in. See the diagram in my first post.

A replication cannot change properties or do anything to a destination dataset that the administrator could not do anyways

I think what happened was:

1. I turned the USB drive into it’s own pool
2. I created an encrypted dataset on the USB pool
3. I set up a replication task to move the (encrypted) contents of my host pool onto the (encrypted) root dataset of my usb drive.

In that case maybe I’m putting encrypted content into an encrypted container?

How was this HEX-string originally generated? It depends. Maybe you had it randomly generated upon the dataset’s creation, or you provided one yourself (copy+paste?).

I think I had it randomly generated.

There’s no key for the pool. Did you mean your root dataset?

probably, yes.

@winnielinnie

@giftcardreloaded

The process of unlocking my USB drive pool and its root-level dataset (Backups) required using the USB encryption key Z and then my host encryption key Y, respectively. Is this expected?

What do you mean by this?

To unlock my USB drive pool dataset(?) I provide encryption Key Z. This unlocks the top-level USB-ZFS-BACKUP, but USB-ZFS-BACKUP\Backups remains locked. to Unlock USB-ZFS-BACKUP\Backups, I need to provide my Host encryption key Y.

---

zfs list -r -t filesystem -o name,encryptionroot

NAME                                                                    ENCROOT
boot-pool                                                               -
boot-pool/ROOT                                                          -
boot-pool/ROOT/23.10.1.1                                                -
boot-pool/ROOT/23.10.2                                                  -
boot-pool/ROOT/24.04.0                                                  -
boot-pool/ROOT/24.04.0/audit                                            -
boot-pool/ROOT/24.04.0/conf                                             -
boot-pool/ROOT/24.04.0/data                                             -
boot-pool/ROOT/24.04.0/etc                                              -
boot-pool/ROOT/24.04.0/home                                             -
boot-pool/ROOT/24.04.0/mnt                                              -
boot-pool/ROOT/24.04.0/opt                                              -
boot-pool/ROOT/24.04.0/root                                             -
boot-pool/ROOT/24.04.0/usr                                              -
boot-pool/ROOT/24.04.0/var                                              -
boot-pool/ROOT/24.04.0/var/ca-certificates                              -
boot-pool/ROOT/24.04.0/var/log                                          -
boot-pool/ROOT/Initial-Install                                          -
boot-pool/grub                                                          -
pool1                                                                   pool1
pool1/.system                                                           pool1
pool1/.system/configs-ae32c386e13840b2bf9c0083275e7941                  pool1
pool1/.system/cores                                                     pool1
pool1/.system/ctdb_shared_vol                                           pool1
pool1/.system/glusterd                                                  pool1
pool1/.system/netdata-ae32c386e13840b2bf9c0083275e7941                  pool1
pool1/.system/rrd-ae32c386e13840b2bf9c0083275e7941                      pool1
pool1/.system/samba4                                                    pool1
pool1/.system/services                                                  pool1
pool1/.system/webui                                                     pool1
pool1/appdata                                                           pool1
pool1/archive                                                           pool1
pool1/backups                                                           pool1
pool1/docker                                                            pool1
pool1/docker/data                                                       pool1
pool1/docker/stacks                                                     pool1
pool1/ix-applications                                                   -
pool1/ix-applications/catalogs                                          -
pool1/ix-applications/default_volumes                                   -
pool1/ix-applications/k3s                                               -
pool1/ix-applications/k3s/kubelet                                       -
pool1/ix-applications/releases                                          -
pool1/jailmaker                                                         pool1
pool1/jailmaker/jails                                                   pool1
pool1/jailmaker/jails/docker                                            pool1
pool1/library                                                           pool1
pool1/library/movies                                                    pool1
pool1/library/music                                                     pool1
pool1/library/shows                                                     pool1
pool1/library/vg                                                       pool1
pool1/memes                                                             pool1
pool1/photos                                                            pool1
pool1/projects                                                          pool1
pool1/shared                                                            pool1
pool1/temp                                                              pool1
pool1/users                                                             pool1
pool1/users/XXXXX                                                       pool1
pool1/users/XXXXX                                                    pool1
usb-zfs-backup                                                          usb-zfs-backup
usb-zfs-backup/backup                                                   usb-zfs-backup/backup
usb-zfs-backup/backup/.system                                           usb-zfs-backup/backup
usb-zfs-backup/backup/.system/configs-ae32c386e13840b2bf9c0083275e7941  usb-zfs-backup/backup
usb-zfs-backup/backup/.system/cores                                     usb-zfs-backup/backup
usb-zfs-backup/backup/.system/ctdb_shared_vol                           usb-zfs-backup/backup
usb-zfs-backup/backup/.system/glusterd                                  usb-zfs-backup/backup
usb-zfs-backup/backup/.system/netdata-ae32c386e13840b2bf9c0083275e7941  usb-zfs-backup/backup
usb-zfs-backup/backup/.system/rrd-ae32c386e13840b2bf9c0083275e7941      usb-zfs-backup/backup
usb-zfs-backup/backup/.system/samba4                                    usb-zfs-backup/backup
usb-zfs-backup/backup/.system/services                                  usb-zfs-backup/backup
usb-zfs-backup/backup/.system/webui                                     usb-zfs-backup/backup
usb-zfs-backup/backup/appdata                                           usb-zfs-backup/backup
usb-zfs-backup/backup/archive                                           usb-zfs-backup/backup
usb-zfs-backup/backup/backups                                           usb-zfs-backup/backup
usb-zfs-backup/backup/docker                                            usb-zfs-backup/backup
usb-zfs-backup/backup/docker/data                                       usb-zfs-backup/backup
usb-zfs-backup/backup/docker/stacks                                     usb-zfs-backup/backup
usb-zfs-backup/backup/ix-applications                                   -
usb-zfs-backup/backup/ix-applications/catalogs                          -
usb-zfs-backup/backup/ix-applications/default_volumes                   -
usb-zfs-backup/backup/ix-applications/k3s                               -
111

[winnielinnie]

@giftcardreloaded

zfs list -r -t filesystem -o name,encryptionroot

This reveals you have three encryptionroots.

They are:

• pool1
• usb-zfs-backup
• usb-zfs-backup/backup

Concerning encryption keys, these are the only three datasets that matter.

If you “lock / unlock” pool1, then it means you “lock / unlock” all datasets that are part of the pool1 encryptionroot. (Encryption “family”?) No child datasets are locked or unlocked independently. (Unless you explicitly break a dataset’s encryption inheritance, in which it becomes its own encryption root, and hence uses its own key.)

The above is also true for the other encryptionroots: usb-zfs-backup and usb-zfs-backup/backup

---

Let’s ignore the top-level root dataset usb-zfs-backup for now. (In fact, you might have faced less confusion if you left it non-encrypted. It serves no real purpose other than a “glorified peg-hook”.)

Now the question is, for the remaining two encryptionroots (pool1 and usb-zfs-backup/backup), do they use the same HEX-string key?

[giftcardreloaded]

Mass correction for my above post – I think every time I say “pool” I’m really talking about the pools dataset by the same name.

@winnielinnie

Now the question is, for the remaining two encryptionroots (pool1 and usb-zfs-backup/backup), do they use the same HEX-string key?

Asking the real questions! Yes, pool1 and usb-zfs-backup\Backup share the same encryption key

@winnielinnie

Let’s ignore the top-level root dataset usb-zfs-backup for now. (In fact, you might have faced less confusion if you left it non-encrypted. It serves no real purpose other than a “glorified peg-hook”.)

I don’t remember my reasoning for having done this. I bet I wasn’t sure if replicating my data to an external drive would encrypt it, so I probably created an encrypted dataset “just in case”. I brought this on myself.

[winnielinnie]

@giftcardreloaded

I brought this on myself.

No you didn’t.

I’m telling you, the ZFS design, which we’re stuck with forever, is stupid and counter-intuitive.

---

If you look at this post, try this thought experiment:

You see the rectangle named “tank (root dataset)”? No data ever lives directly inside. It serves no purpose other than to exist as the immutable root dataset and provide “defaults” for any pseudo-roots (or children) to inherit from.

That’s it. Nothing more. You don’t want to (nor need to) send it anywhere, because any target pool you try to send it to will have its own immutable root dataset that cannot be overwritten. You can only send a dataset to be nested underneath the destination pool’s root dataset.

If this dataset (that you nest underneath) is the source pool’s root dataset? It’s redundant and pointless, and will always create an extra “ladder” in your hierarchy for no useful benefit. In fact, if you promote your backup pool to the new main pool? Then it means you don’t have the same layout as the former main pool.

By accepting the purpose of a root dataset as a “glorified peg-hook”, you can focus on your own meaningful pseudo-roots, in which it makes sense to nest underneath a destination pool’s root dataset. (You retain the same layout, ladder, and hierarchy. The only difference being the name of the root dataset.)

This way you don’t need to tell the replicated dataset to “inherit” anything from its parent on the destination pool. You can just do a raw stream or provide your own passphrase / keystring, which is unrelated to the destination pool’s root dataset. (Encrypted or not, doesn’t matter. But if the destination pool’s root dataset is encrypted? It means you have to unlock one extra encryptionroot to be able to access the encryptionroot underneath; which you had sent over.)

In the flowchart, only the colored boxes are to have their own encryption passphrase/keystring (as encryptionroots), regardless of what the root dataset is. These are the only datasets being “sent” anywhere. The root dataset (“tank”) never gets sent anywhere.

---

My personal opinion, ZFS would have been way more intuitive if there was no immutable root dataset.

Can you imagine how much more sense it would make that upon creating a pool, it’s just a… pool. No datasets. Just a pool. Then in order to get started with datasets, you can start creating your own root datasets, which are independent of each other?

Sadly, that’ll never be the case, and hence why I use the pseudo-root method.

[giftcardreloaded]

Your explanation and diagram are great, thank you very much for sharing. A classic case of poor UX “working as expected”. Glad to know nothing is wrong necessarily with my setup.

What’s the cleanest way to revise my file structure? I guess create pseudo-root datasets beneath pool1 and then move my existing datasets underneath them?

[winnielinnie]

@giftcardreloaded

Glad to know nothing is wrong necessarily with my setup.

I will warn you, there’s a high likelihood you’ll lose your data, unless you get a real clear grasp of what is going on, and you practice the routine habit of backing up your config and keys (or passphrases), and testing them regularly.

If you’re caught in this shuffle, you might accidentally change a dataset’s userkey (without realizing it), or not realize that an exported keyfile is relevant only to a dataset’s original encryption key, which has since been changed / inherited. You might even accidentally overwrite a “saved keystring” or “saved passphrase” that was meant for another dataset, confusing the two and forever locking yourself out of your own data.

[winnielinnie]

@giftcardreloaded

What’s the cleanest way to revise my file structure? I guess create pseudo-root datasets beneath pool1 and then move my existing datasets underneath them?

This might be possible, since you never saved anything directly in the root dataset “pool1”.

It will be tedious (and scary!), since you’ll have to create the pseudo-root, and then “rename” each child dataset, one by one, into the pseudo-root.

What “should” happen is that the pseudo-root will inherit pool1’s encryption properties (it will be a member of the “pool1 encryptionroot family”), and thus, the same will hold true for each child dataset you “rename” (move) into it.

After this, you can break the pseudo-root’s inheritance, and have it be its own encryptionroot.

The only problem is, you’ll have a redundant, unrelated, encrypted root dataset that must be unlocked in order to use the pseudo-root underneath. (A non-encrypted top level root dataset is much “cleaner” and easier to work with.)

I advise you create a checkpoint before doing this, as a temporary safeguard in case something going wrong. (The checkpoint can be removed later.)