Are you still able to destroy the datasets on the backup pool? (Or have you already committed?)
If you’re still in a position to start over, redo your setup with these fundamental changes:
Manually type in the exact dataset structure you want to see on the receiving pool. For example: backup/enc
(Do not “pre-create” a dataset ahead of time. Do not “select” a destination with the mouse.)
As long as no such dataset/folder exists, the replication task will create the new encrypted dataset the first time you run it.
Uncheck “Inherit Encryption”
When you run the replication task for the first time, the dataset will be created on the other side, and it will become its own encryptionroot, not tethered to anything, nor will it conflict with the non-encrypted top-level root dataset.
So, I have found that what you are trying to do only works if the parent of the destination dataset is not the encryption root. Meaning you have to create another dataset underneath backup/test_enc, and then set that as the destination. I didn’t really dig in to see why this is the case, perhaps there is a way to avoid the extra layer but wanted to share my workaround.
If this really is required, I don’t see any reason, or documentation, so perhaps a bug report is in order. Perhaps others know more.
I think the nuance here, and this is what I ran into as well, is that @jeanmich is trying to have the encrypting setting be inherited. The only way I found to have this work is to have the destination dataset inherit from a non-encryptionroot – hence the extra dataset in between. Perhaps there is some fundamental reason for this requirement, or maybe it’s a bug, I don’t know enough.
In this case, they’re not. Their top-level root dataset on the destination (“backup”) is non-encrypted. If they’re fine with that, they can have the replication task create a new encryptionroot underneath their non-encrypted root dataset.
There’s still a confusion with the way ZFS (and even TrueNAS) present such options to the user. I had filed some bug reports against the documentation and tooltips in the GUI, since there are misnomers about ZFS encryption.
There’s no such thing as an “encrypted pool”, which is what a user is suggested to believe upon creating a new pool. “Do I want my data encrypted? If yes, I should go with an encrypted pool.”
Encryption only happens at a per-dataset level. It just so happens that the top-level root dataset can be encrypted or non-encrypted. There’s no hard rule that prevents you from nesting non-encrypted children underneath encrypted parents, and vice versa.
However, TrueNAS enforces certain limits in the middleware/GUI in order to prevent unintended results where a user might not understand “Where’d my data go?” or “Why can’t I mount my filesystem?”
The way I understood this, is that they have a non-encrypted root dataset (“backup”), and then a pseudo-root encrypted dataset (“backup/test_enc”), and the destination of the replication is a child of that pseudo-root (“backup/test_enc/data”)
Following your awesome advice in the other post that @jeanmich mentioned, I had the same setup. No matter what combination of settings I tried, I could not have a destination dataset that inherited it’s enctyption settings from an “encryptionroot” dataset. I had to create a sub-dataset first (that also inherited encryption) and then put the destination (again with inherited encryption settings in the replication wizard) undeath that.
As you mentioned an alternative is to have the destination have its own encryption settings.
Because it was “pre-created” on the destination pool ahead of time. This is due to the way the TrueNAS GUI presents itself to the user. It makes you think you need to “select” something with your mouse, when in fact you can manually type in the desired destination dataset, which the task will create upon first run.
This is why I think the command-line is more clear: it forces you to type in everything. The GUI invites you to use your mouse cursor to “click” things, without indicating that you can also manually write things yourself.
Feature request, maybe? For the GUI to give you a hint that you can also type things into the destination text field without clicking anything? Maybe a button that says “Create new dataset for destination on first run”?
At the risk of highjacking the post, what if you really do want an encrypted pseudo-root on the destination and then have replication destinations underneath that. In my setup, I have a passphrase encrypted pseudo-root, and I wanted multiple replication destinations underneath that to inherit the passphrase encryption. So something like
The only way I could get this to work was to create a dataset undeath zroot_enc that inherited encryption, and then place the repl_dstX inderneath that new dataset.
I didn’t try manually doing a send | receive, perhaps thats the right next step to see if this is a TrueNAS middleware/gui issue or a ZFS limitation.
tank and nvme_fast_risky are unencrypted, zroot_enc has passphrase encryption. I manually typed in the destination dataset tank/zroot_enc/nvme_enc_bu (which does not exist), and selected encryption and inhereit encryption. When I run this I get the same error as @jeanmich. However, if I create a dataset underneath zroot_enc that inherits encryption, and change the replication task so that the destination is /tank/zroot_enc/xyz/nvme_enc_bu, then nvme_enc_bu gets created by the replication task and it inherits encryption settings just fine.
There might have been some confusion in the other thread.
The “pseudo-roots” are meant to originate in the main pool[1], to then be replicated to the backup pool (one level underneath the backup pool’s root dataset.)
This will maintain the layout and hierarchies and encryptionroot/encryption relationships.
Hence, you wouldn’t need to worry about inheriting anything, since the pseudo-roots from the main pool will carry over their own “cohesive encryption family”, with their own encryptionroot on top, and their children underneath.
The flowchart in the other thread was a hypothetical representation of a main pool that is sent from. ↩︎
I should have been more clear. I understand that your original suggestion re:psuedo-roots was for the main/source pool. I’m applying the same suggestion to the destination in this example because the destination is itself further replicated.
I think there is a more generic use-case here, which is to have a “container” dataset that sets up encryption settings, and contains replication target datasets that inherit those encryption settings. This would (1) allow for centralization of encryption settings, and (2) allow setup of replication to an encrypted destination without having to know the encryption secrets.
I’m thinking that what were seeing in the gui is a bug in the middleware/gui. I just tried the following on the command line:
As before, zroot_enc is encrypted and nvme_enc_bu does not exist. The command completes without error and creates an encrypted nvme_enc_bu underneath zroot_enc with its encryption settings inherited from zroot_enc.
So seems there is no fundamental reason ZFS can’t do this. Perhaps this is an intended limitation imposed by TrueNAS, but given that the error msg that @jeanmich and I are getting is factually incorrect (it’s saying that the parent is not encrypted when in fact it is), this seems to be a bug (or at least the error msg is a ‘bug’).
It was never really clear what the GUI is presenting, nor what is expected.
If you ask me, they slapped together this “Replication Tasks” feature for a narrow scope:
Have a Periodic Snapshot Task
Create a Replication Task based on this Periodic Snapshot Task
Maybe just don’t use encryption, since that complicates things
This can be achieved with a “raw stream” (which isn’t really obvious in the GUI, but it’s supposedly invoked if you check “Full Filesystem Replication” and the source is encrypted.)
You get so much more flexibility and control with command-line tools, such as vanilla zfs commands, syncoid, and zfs_autobackup.
Keep in mind that you cannot install software on the TrueNAS SCALE host directly. (I have zero experience in even trying to get any “zfs tools” to work in TrueNAS, other than what is already provided.)
Hi. I’ve carrefully read everything but my knowledge in ZFS are limited and I didn’t get everything.
If I’m doing the solution given by @vladatnyc with zfs send/receive using the command line, is it possible to create the replication task afterwords once all pools are correctly instantiated?
I really want the solution of “containerized”, top, encrypted dataset that would allow me to have the encryption automatically done when I backup a dataset ine the “enc” dataset.
