TrueNAS/ZFS encryption + deduplication for home server?

TrueNAS General
1

Hi,
I am new to TrueNAS and currently setting up a simple DIY home server NAS.
The goal is to have a simple ZFS mirror and backup data to various sources, ZFS replication is one of these alternatives.

Native ZFS encryption sounds like a great and simple recipe to protect data in case of theft and/or disk warranty. But I am a bit worried after having read posts:

It raised some questions:

1. Is ZFS encryption sufficiently “stable” for above usage scenario?
This should consider bare ZFS implementation and assume there are no user mistakes. To count as stable, the system should be recoverable without need for backup as minimal requirement.

2. Does TrueNAS mitigate or warn about buggy ZFS encryption configurations?
E.g. warnings in the GUI - or am I on my own here?

3. Regarding ZFS replication + encryption
From what I understand, encr. ZFS replication via zfs send/receive or Syncoid is prone to errors. Does that mean, replicating the ZFS mirror to an offsite-backup damages the offsite-backup irreparably in worst case? Is this immediately apparant, and am I able to retry this replication, as the main mirror is still intact?

4. Regarding ZFS encryption + deduplication

We do not recommend using GELI or ZFS encryption with deduplication because of the sizable performance impact. (TrueNAS docs)

Hm, are there some measure to support the “sizable performance impact” statement? For me both encryption and deduplication are main selling factors for ZFS, so a bit sad to read that recommendation.

5. Unencrypted boot-pool data leaks

For encrypted data at rest, child datasets need to be encrypted with a passphrase, as keys are exposed in the unencrypted boot partition (see also here). Apparently some meta data is still stored in the unencrypted boot pool - is there a comprehensive list somewhere in the docs? If it’s only the names of datasets and snapshots, I am fine with that.

(6. Tips on using TrueNAS Core/Scale under Proxmox?)

I am thinking of installing Proxmox as host with ZFS and TrueNAS in a VM, passing through all SATA controllers. Optimally this could result in full-disk encryption of Proxmox host incl. VMs, headless pre-boot passphrase (Dropbear), and ZFS filesystem-encryption for all HDDs natively managed within TrueNAS VM. With this configuration I also could use ZFS dataset keys, as protected by outer layer encryption.

Sorry for these many questions - please don’t hesitate to answer a single one, so I can subsequently get more clarity on this topic. Thanks!

2

(1) I’ve used TNS encryption without any failures but you need to be careful about key management and remote replication. What do you mean when you say “recoverable without need for backup”? What kind of failure are you talking about recovering from?

(4) Unless you have some extremely unusual use case, deduplication isn’t going to do anything for you.

(3) Unless you are doing “buddy backup”, AFAIK the only option to do remote ZFS replication to a cloud provider is limited to rsync.net (never used it). I use the TNS GUI for Cloud Sync to B2 and I’m testing TrueCloud in the 24.10 beta to backup to Storj.

(6) I’ve never had much success with hardware passthrough of motherboard SATA ports. If you want to use hardware passthrough, better to do it with an HBA. I just use virtio with my disks and that’s been fine (but there are issues).

3

Other than reports by others, I’ve never had an issuereft ZFS##########################encpr my ddata safee but bitr.fd###^^^^^^^^^^^^//////## don’t you agree?

I’ve only had a single issue with native ZFS encryption, using it since 2020. However, there was no data loss. It was a bug that caused the system crash upon hitting a special type of block during replication. The bug has since been fixed.

My main concern with native ZFS encryption is that the original author has left the OpenZFS project. If you peruse their GitHub, you get a sense that some contributors are trying to figure out this beast that we know as “ZFS”.

Fingers crossed.

Only in instances where the user might inadvertently cause mounts to occur in the wrong order. That’s why the middleware/GUI prohibits nesting a non-encrypted dataset underneath an encrypted one. There is no technical limitation in ZFS that prevents this, but it’s a measure employed by TrueNAS to safeguard against unforeseen problems.

Encryption introduces another hurdle when dealing with replications.

The target needs to be “accessible” (“unlocked”), otherwise, you must use the “raw stream” option. However, a successful replication does not mean your data is safe. If you lose your key or passphrase (or don’t realize that a backup from long ago is using a different key), then your data is as good as gone.

You also cannot use send/recv to “overwrite” a target dataset that is already encrypted. (Which is a good thing.)

You probably don’t need to use deduplication.

Furthermore, consider that each encrypted dataset uses its own “master key”, it prevents pool-wide deduplication efficiency, since the “same” blocks written on different datasets will be encrypted differently.

The boot pool contains the keys needed to automatically unlock encrypted datasets upon system bootup. The datasets need to be immediately available for SMB, NFS, jails, apps, and so on; and the System Dataset needs to be immediately available, as well.

So yes, in theory, someone who has physical possession of your server can extract the keys from the boot-pool.

As for encrypted datasets, ZFS metadata is not encrypted. (It cannot be.) Names of datasets, snapshots, clones, comments, dates, used space, free space, the pool’s "history, and possibly even the number and sizes of files.

4

Deduplication is a resource hog. Don’t touch it unless you expect some spectacular impact, such as 5x (x1.2 or x1.5 does NOT qualify), and have at least 64 GB RAM (128 GB would be better).

2 Likes
5

https://forums.truenas.com/t/memes-truenas-zfs-and-related-share-your-own/335/52

dedup-vs-fast-dedup

DON'T CLICK ME

simplicity-for-the-win

1 Like
6

Some comments from OP links raised my evebrows:

Don’t use native encryption if you’re not willing to lose the entire pool - and possibly any pool you zfs send an encrypted snapshot to. There are quite a few bugs/issues with native encryption, some of which generally result in the complete destruction of the pool.

Suffering from a constant permanent errors in zpools after trying to implement ZFS native encryption plus autosnapshots synchronization using sanoid/syncoid.

→ first link

Native ZFS encryption in Proxmox VE is experimental. Known limitations and issues include Replication with encrypted datasets
[2350 – ZFS->ZFS storage_migrate does not work if zfs feature@encryption=enabled]
, as well as checksum errors when using Snapshots or ZVOLs.
[permanent errors (ereport.fs.zfs.authentication) reported after syncoid snapshot/send workload · Issue #11688 · openzfs/zfs · GitHub]

→ second link

Basically I am worried about an irreversible pool/mirror corruption, where you can throw away disk contents and need to resort to a - possibly ZFS-independent - backup. On the other side some posts state, encryption is production-ready. And yet others say, it is fine when not using replication or condition-x. For me as new user it is not easy to judge, if switching from ZFS or ext4 over LUKS to native ZFS encryption is a safe and stable operation.

CPU needs to support IOMMU/VT-d to do PCI(e) passthrough, then TrueNAS within VM should get native access to drives. With latter iirc there was some discussion about need of HBA firmware patches to work correctly. I imagine it also could depend on the mainboard controller or HBA brand. Guess I will try out both alternatives :slight_smile: .

Interesting, thanks for the hint!
Out of interest: Are TrueNAS developers/iXsystems involved in OpenZFS development, to minimize these “single points of failures”? Honestly I don’t know anything yet about OpenZFS, their contributors and how this project kept viable financially.

Is this safe-guarded by ZFS or TrueNAS against user mistakes? What I mean is: Will it fail, before backup/pool could get damaged or corrupt?

True. But let’s neglect user mistakes for this discussion.

Sounds good. I think it mostly is important, that contents (file names, size, content, etc.) within datasets are not visible at rest. If there are concerns with meta data, one might just choose generic names.

You mean 128GB RAM? OK, de-duplication is out :wink: Thanks for mentioning.

I have used data de-duplication with restic so far. It can save a lot of disk space, when for example moving bigger directories somewhere else in the file tree. In contrast something like incremental backups with rsync (--link-dest) will duplicate all data/space. But will re-think requirments now.

7

de-duplication in restic and de-duplication in ZFS are two completely different things that happen to be called the same thing. ZFS snapshots are the rough functional equivilent of de-duplication in restic.

8

ZFS is enterprise storage. iXsystems makes money by selling their TrueNAS Enterprise products and actively contributes to OpenZFS development—though I don’t know if that includes the encryption part.
OpenZFS is likely much more sustainable financially than any BSD or Linux distribution.

ZFS replication doesn’t: It only sends the modified blocks.
And I did mean 128 GB RAM. From first-hand experience that’s the minimum to reasonably handle just over 10 TB of deduplicated data. That or a mere 64 GB RAM and then a dedicated dedup vdev or persistent L2ARC but that’s another can of worms.

1 Like
9

I seem to remember posts from last year that recommended at least 512 GB RAM for deduplication.

1 Like
10

Sharing this post Is there any known issue with dedup on truenas core 13.0? - #7 by elkmaster

Where @HoneyBadger explains this very well.

1 Like
11

Probably not. At least it seems a bit surprising to be dependent on a single developer with enterprise-level.

Thanks for the infos, de-duplication is out for me due to its requirements. (Also I haven’t understood the concept fully yet and will do some research on it.)

I still would be curious, if described ZFS encryption stability issues hold true.

12

I can only speak for myself and one other (whom I loosely “help” with their TrueNAS server). Never had any problems[1] with native ZFS encryption. No noticeable performance impact, nor any issues with data access, replications, and backups.

Hopefully I don’t get bit by some surprise bug that destroys all my data. :crossed_fingers:

  1. Except that one time where a bug in ZFS causes the entire server to crash when it hits a special type of block during a replication, which has since been fixed. ↩︎

13

The skinny of deduplication is that it skips writing a block of data that “already exists” on the pool, hence it can save space in some circumstances. Rather than write a new block of data, it just “points” to the already-existing block. In a sense, a single block of data can be used by multiple files.

How to know that a specific block of data already exists? The pool needs to maintain (and preferably keep in RAM) a large “table” of all hashes for currently existing blocks. Before “writing” a new block, its hash is computed and then there’s a search for a “hit” on this table. If it finds a hit, it discards the process of writing this new block, and simply points to the existing block (as well as bumps the “counter” for this hit by +1.)

However, there are two alternatives (which can be used together) that can save space in a ZFS pool, without the need of added complexities, resources, and the potential performance impact of using deduplication: inline compression + block cloning

Inline compression is self-explanatory.

Think of “block cloning” as an “on-demand dedup” that does not require a dedup table, nor any of the requirements for using deduplication. When you issue a “copy” of a file, the existing data blocks that exist on the pool are simply “pointed to” for the new file. Hence, you end up with two separate files (not hardlinks!) that do not consume any additional space.[1]

It also allows for “single file restores” from snapshots, without consuming additional space, since the referenced blocks are the same.[2] No special tools are needed. Just “copy” a file from a “snapshot directory”, and it is instantly restored. Zero extra space is used on the live dataset. (This was not possible before OpenZFS 2.2.)

  1. If you’re using encryption, this is limited to the dataset itself. It is not “pool wide” if you’re using encryption. ↩︎

  2. If you’re using encryption, this does not apply. ↩︎

2 Likes
14

@winnielinnie I need to read over your post a couple more times and lookup missing puzzle pieces, but thank your very much for the thorough explanation!

Then I don’t get why @tannebil framed restic and TrueNAS de-duplication to be “completely different things” conceptually. Of course internal implementation is different - and TrueNAS seems to even provide alternatives like mentioned block cloning. But in essence both save disk space by only storing one equal copy of certain data in different locations.

Thanks as well @winnielinnie for describing your own experience with ZFS encryption, that’ll make me a bit more comfortable. I am too curious anyway and will try it out :wink: .

Let’s play one case through:
Assuming ZFS encryption with plain mirrors is stable© and only replication part might cause errors. Then replicating main pool to a backup-pool via zfs send / Syncoid (unidirectional data flow) can only corrupt backup in worst case and not main, as there is no write operation on main, right?

Hence to be safe, I could have one experimental backup pool with ZFS encryption + replication and say one stable backup target with different tooling, like restic (please correct me otherwise).

15

Most likely because TrueNAS/ZFS’s implementation of de-dupe is permanently notoriously resource intensive.

Thus implementation matters.

1 Like
16

The source is not written to, correct.

But there’s no risk of corrupting the backup, either. A snapshot exists or does not exist. There’s no “halfway” snapshot. So if the replication succeeds, then it means the snapshot on the backup exists as a true replica of the source snapshot.

If the replication fails, then the snapshot does not exist on the backup. Nor are any other snapshots or datasets affected on the backup.

1 Like