Limitations ZFS Encryption

Dear community,

I am in a situation where a friend of mine allows me to use his proxmox server to create VM’s. Furthermore my friend created a zfs pool NFS share on his TrueNAS that I can use to store my VM’s data.

  1. Now, for privacy reasons I want private data not be accessible to my friend.
  2. Hence, I created an encrypted ZFS pool with a passphrase keytype inside my ubuntu VM mounted to the NFS share.

On the internet I read this: When a ZFS dataset is encrypted, the encryption keys are stored in memory only in an encrypted form, known as “wrapped keys”. Wrapped keys are encrypted using a master key or a passphrase, which is not stored in memory.

The question: Is there any way for my friend to obtain my private data stored on the encrypted zfs pool inside my VM ? Snapshots, Clones, Migrations, Memory dumps ??? or what is said above is correct ?

Thank you!!

This part is too confusing for me. Maybe someone else can chime in and demystify it?

It sounds like there are two ZFS pools at play? The ZFS pool of your friend’s TrueNAS VM, hosted on Proxmox. In this virtualized TrueNAS there is a ZFS pool with a dataset that serves an NFS share for your Ubuntu VM, which will have its own ZFS pool?

I honestly think I’m reading it wrong. It’s hitting inception levels to me. (Apparently, I hate virtualization or something.)


Sort of, but not quite. There are some terms being used casually.

When an encrypted ZFS dataset is “locked”, there is nothing relevant in memory (RAM). The data-at-rest is useless garble.

In the dataset’s own metadata / “header” (the technical terms might differ), there exists an encrypted form of the Master Key. (The true, non-encrypted form of the Master Key is the 256-bit key that is used by AES to encrypt/decrypt data that is being written/read.)

This encrypted form of the Master Key itself is useless. You need to decrypt the Master Key, so that you can load the true Master Key into RAM. Then you’re back in business, encrypting/decrypting the contents of the dataset to your heart’s content!

Because it wouldn’t make sense to store the true Master Key in the dataset’s header, a function called “passphrase-based key derivation” is used, which takes the Master Key + something to create an encrypted form of the Master Key that can be safely stored in the dataset’s header.

That “something” is known by the owner of the data: a passphrase or long hex string. (With LUKS on Linux, it can even be any arbitrary file, such as a JPEG image!)

So without that “something” (passphrase, hex string, etc), the encrypted form of the Master Key is useless.


I have no idea. See the reason above at the start of my reply. (I honestly don’t know what the setup even looks like or who owns what.)

1 Like

Thank you for you reply @winnielinnie ! I am sorry for my bad explanation.

My friend provided me with a VM with x amount of resources on his Proxmox machine that he has full control off. One of these resources is a TrueNAS NFS share (ZFS pool) that he has full control of as well. For me to gain access to this VM simply SSH into it over a VPN. Now, if I would use this NFS share unencrypted then my friend would have access to all the files. Then the question is: Is there a possibility that I can create a ZFS encrypted pool or a LUKS drive inside this VM s.t my files cannot be accessed by my friend even though he has full control over the hardware my VM is stored on. Or is there always a way for the owner of the hardware or proxmox system to decrypt the encrypted pools ?

Thank you for your time!

Ok, lets see. He/She gave you a VM and from that VM you can access the NFS share. Therefore your friend can do whatever he/she wants.

Unless your story has greater clarity, that’s that.
Because you don’t say what OS runs in that VM nor what access you have to your friend’s system. Nor, … am getting aggravated. Be clear.

Thank you @argumentum for your reply.

  1. VM OS will be either ubuntu, debian or proxmox
  2. I have full access to this VM, comparable to a person’s access renting a VPS with x amount of resources
  3. Inside this ubuntu/debian/proxmox VM an encrypted zfs pool with passphrase protection will be created.
  4. Inside this encrypted zfs pool I will save all my application files e.g Nextcloud files.

Question: Can my friend who hosts this VM on his proxmox server find any way to decrypt this encrypted zfs pool that I only have the passphrase off to unlock it and mount it at boot.

Furthermore, if this is not the route, what would be the best route to take to store application data inside a VM hosted on someone else’s server. But then without the host having the possibility to access that data through some way or form ?

Thank you!

Ok, sounds good. Good friend you’ve got there.

If you encrypt it yourself then no one can unlock it without the passphrase.

Sounds good. The system will not boot without a passphrase. If you don’t control the hardware, is as good as it gets for a friend giving you resources.

Am not such friend as I would not know what the user ( you in this case ) will do from my public IP. So again, nice friend :slight_smile:

I don’t think your friend has a supercomputer to brute force passphrases until finding one that matches, therefore I’d say that you’re ok.

…if something goes wrong, like OMG it all got lost. Do set your self a remote ( from that system ) backup for your data. Because with all that encapsulation, it can go wrong in ways that wasn’t anticipated. So do backup elsewhere too.

2 Likes

What do I mean by “all that encapsulation”: You’ll set a hypervisor inside your VM and that will be open for whomever can access it. Inside your hypervisor ( Proxmox ) then you can have a VM that will not even boot without a password because everything depends on that password to even boot. That last ( 3rd level VM ) would be the secured one.

1 Like

Thank you @argumentum for the confirmation,

Yes, I have a Proxmox system myself at home as well and want to use his place for the backup.

So you’re going to essentially use ZFS on top of ZFS?

Yess that is correct indeed @winnielinnie or I would use LUKS on top of ZFS. I am not sure yet what will be best for performance ?

I’d suggest you simply set up a replication task to each other using a private VPN and send encrypted pools to each other. Bare metal is where it’s at given reports of issues with TrueNAS running inside a VM.

The default SSH approach may be a bit too much keys to the kingdom, but I am sure that SCALE developers have published a path to using less privileged users to allow replication between systems. Then neither your friend can read your data nor you can theirs. Then, each others stuff is secure, ditto the NAS’

2 Likes

Thank you @Constantin. The plan is to use syncthing for file replication and mariadb or postrgres database replication between my place and my friends place. The idea is to have high availability applications i.e if my house burns down that the application continues running at my friends place. I am thinking about a Kubernetes cluster of 5 virtual machines. Where two master nodes + 1 worker node are stored on my server and 1 master + 1 worker are stored at my friends place. They will then share data over the VPN connection.

Do you think this is a good plan ?

1 Like

My friend, that is so over my head that my toupé is somewhere mid-Atlantic at this point. Good luck! :slight_smile:

2 Likes

Technically, it’s theoretically possible to snoop the key out of the VMs memory and unlock the data.

This is non trivial. And wouldn’t be the act of a friend.

And this friend does sound like a friend giving you this level of trust on his system.

It’s more practical for you to hack his network. Than for him to hack your encryption.

So, yeah, put a layer of encryption on the nfs share if you want. How you achieve that is up to you.

Thank you for your reply @Stux, that’s what I thought but I have no idea on how to do this or how easy it is if you would try.

Besides that I am interested in your point that others pointed out as well that it is easier for me to hack his network. I would be interested in what the vulnerabilities are and how my friend could secure himself.

One thing I thought about is that my friend can make a VLAN for me which can’t connect to any of his devices on other VLAN’s.

Please tell me if this would be enough or that more should be done!!

Thank you !!

You certainly can do that (private VPN, isolated VLAN, etc). I’m pretty certain I’d know how to do that with my MikroTik gateway but a better bet may be something like Tailscale on the VMs themselves.

I have zero experience with Tailscale, so I’d consider the experiences of other TrueNAS Scale users with implementing it.

That would limit contact to just the VMs in question and isolate your respective networks. Locally, you’d have to isolate each VM to be only able to speak on Tailscale and perhaps the internet too (for software updates, NTP, and so on).