How to recover the zpool encryption key after OS disk failure

Setup: TrueNAS Scale OS running from boot drive on a server with 5 4TB SATA drives until my dual m.2 pcie adapter arrived to run OS redundantly.

Break event: I set up a Linux virtual machine on the same zpool as my truenas apps and passed through the USB slots, including the one that was running the OS (WHOOPS!). Result: Could not load web GUI.

M.2’s arrive and I reinstall OS, import pool, only to discover that I had encrypted the zpool. I checked for a stored pass phrase in all my usual spots, but appear not to have saved one anywhere.

My only hope is trying to recover the encryption key from the USB OS drive, but when I try to boot from this USB it does not successfully load to TrueNAS web GUI.

Am I SOL?

What and why? Can you rephrase this? What did you do exactly?

You mean the root dataset of the pool is encrypted and all child datasets inherit this encryption?

Can you provide the results of this command:

zfs list -r -t fs -o name,encryption,keyformat,keylocation,encroot <poolname>
One more thing...

FOR THE LOVE OF ALL OF ORPHANS IN THE WORLD, PLEASE PASTE YOUR OUTPUT AS PREFORMATTED TEXT. USE THE </> BUTTON ON THE MENU BAR. DO NOT POST A SCREENSHOT, PLEASE.

…thank you. :slightly_smiling_face:

1 Like

I recommend you consider doing something like this next time.

NAME                                                    ENCRYPTION   KEYFORMAT   KEYLOCATION  ENCROOT
zpool                                                   aes-256-gcm  hex         prompt       zpool
zpool/.ix-virt                                          aes-256-gcm  hex         none         zpool
zpool/.ix-virt/buckets                                  aes-256-gcm  hex         none         zpool
zpool/.ix-virt/containers                               aes-256-gcm  hex         none         zpool
zpool/.ix-virt/custom                                   aes-256-gcm  hex         none         zpool
zpool/.ix-virt/deleted                                  aes-256-gcm  hex         none         zpool
zpool/.ix-virt/deleted/buckets                          aes-256-gcm  hex         none         zpool
zpool/.ix-virt/deleted/containers                       aes-256-gcm  hex         none         zpool
zpool/.ix-virt/deleted/custom                           aes-256-gcm  hex         none         zpool
zpool/.ix-virt/deleted/images                           aes-256-gcm  hex         none         zpool
zpool/.ix-virt/deleted/virtual-machines                 aes-256-gcm  hex         none         zpool
zpool/.ix-virt/images                                   aes-256-gcm  hex         none         zpool
zpool/.ix-virt/virtual-machines                         aes-256-gcm  hex         none         zpool
zpool/.ix-virt/virtual-machines/LM                      aes-256-gcm  hex         none         zpool
zpool/.system                                           aes-256-gcm  hex         none         zpool
zpool/.system/configs-ae32c386e13840b2bf9c0083275e7941  aes-256-gcm  hex         none         zpool
zpool/.system/cores                                     aes-256-gcm  hex         none         zpool
zpool/.system/netdata-ae32c386e13840b2bf9c0083275e7941  aes-256-gcm  hex         none         zpool
zpool/.system/nfs                                       aes-256-gcm  hex         none         zpool
zpool/.system/samba4                                    aes-256-gcm  hex         none         zpool
zpool/applications                                      aes-256-gcm  hex         none         zpool
zpool/applications/collabora                            aes-256-gcm  hex         none         zpool
zpool/applications/immich                               aes-256-gcm  hex         none         zpool
zpool/applications/immich/AI_cache                      aes-256-gcm  hex         none         zpool
zpool/applications/immich/backups                       aes-256-gcm  hex         none         zpool
zpool/applications/immich/library                       aes-256-gcm  hex         none         zpool
zpool/applications/immich/postgress                     aes-256-gcm  hex         none         zpool
zpool/applications/immich/profile                       aes-256-gcm  hex         none         zpool
zpool/applications/immich/thumbs                        aes-256-gcm  hex         none         zpool
zpool/applications/immich/uploads                       aes-256-gcm  hex         none         zpool
zpool/applications/immich/video                         aes-256-gcm  hex         none         zpool
zpool/applications/joplin                               aes-256-gcm  hex         none         zpool
zpool/applications/nextcloud                            aes-256-gcm  hex         none         zpool
zpool/applications/nextcloud/appdata                    aes-256-gcm  hex         none         zpool
zpool/applications/nextcloud/postgressdata              aes-256-gcm  hex         none         zpool
zpool/applications/nextcloud/userdata                   aes-256-gcm  hex         none         zpool
zpool/applications/nginx                                aes-256-gcm  hex         none         zpool
zpool/applications/nginx/certs                          aes-256-gcm  hex         none         zpool
zpool/applications/ollama                               aes-256-gcm  hex         none         zpool
zpool/applications/pihole                               aes-256-gcm  hex         none         zpool
zpool/applications/syncthing                            aes-256-gcm  hex         none         zpool
zpool/applications/vaultwarden                          aes-256-gcm  hex         none         zpool
zpool/applications/vaultwarden/appdata                  aes-256-gcm  hex         none         zpool
zpool/applications/vaultwarden/postgressdata            aes-256-gcm  hex         none         zpool
zpool/ix-apps                                           off          none        none         -
zpool/ix-apps/app_configs                               off          none        none         -
zpool/ix-apps/app_mounts                                off          none        none         -
zpool/ix-apps/app_mounts/omada-controller               off          none        none         -
zpool/ix-apps/app_mounts/omada-controller/data          off          none        none         -
zpool/ix-apps/app_mounts/omada-controller/logs          off          none        none         -
zpool/ix-apps/app_mounts/pihole                         off          none        none         -
zpool/ix-apps/app_mounts/pihole/config                  off          none        none         -
zpool/ix-apps/app_mounts/pihole/dnsmasq                 off          none        none         -
zpool/ix-apps/app_mounts/portainer                      off          none        none         -
zpool/ix-apps/app_mounts/portainer/data                 off          none        none         -
zpool/ix-apps/docker                                    off          none        none         -
zpool/ix-apps/truenas_catalog                           off          none        none         -

Yes, the root dataset is encrypted.

I went to Instances and created a new vm with the linux mint iso with mostly default settings. In the disk or storage options I think I selected something incorrect. In addition, under the passthrough options I selected the USB thumb drive that was running my truenas OS. The system was nonresponsive and each boot attempt to reboot truenas failed and indicated “fatal error.”

To be certain, you actually installed over the original boot drive?

If you did this, you basically wiped the keystring. Without an exported keystring, which you should have done, you cannot decrypt your data.

Is your original boot drive currently plugged in?

Can you paste the output of these two commands:

lsblk -o NAME,MODEL,PTTYPE,TYPE,SIZE,PARTTYPENAME,PARTUUID

zpool list

I would suggest not naming your pool ‘zpool’ as that is a command.

3 Likes
sdg    Extreme Pro                             gpt    disk 119.3G                          
├─sdg1                                         gpt    part     1M BIOS boot                f6bb71c1-a996-4ff5-a714-95b208ac5a91
├─sdg2                                         gpt    part   512M EFI System               37506496-8304-4828-a828-18320eb901b8
└─sdg3                                         gpt    part 118.7G Solaris /usr & Apple ZFS 6b70a97b-2db5-4e70-8691-d67a9a75c1a2

That can’t be the complete listing…

Plus, there are two commands in my post.

zpool list
NAME   MODEL                                   PTTYPE TYPE   SIZE PARTTYPENAME             PARTUUID
sda    SAMSUNG MZ7L33T8HELA-00A07              gpt    disk   3.5T                          
└─sda1                                         gpt    part   3.5T Solaris /usr & Apple ZFS 7ef6f78c-5a38-4dc9-99da-6243677b79f7
sdb    SAMSUNG MZ7L33T8HELA-00A07              gpt    disk   3.5T                          
└─sdb1                                         gpt    part   3.5T Solaris /usr & Apple ZFS ab235bf6-08e7-47e4-b70f-008689e5515f
sdc    SAMSUNG MZ7L33T8HELA-00A07              gpt    disk   3.5T                          
└─sdc1                                         gpt    part   3.5T Solaris /usr & Apple ZFS 3819f55e-fcb6-4a78-8cac-6c56cd5b2680
sdd    MTFDDAV240TGA-1BC16A 03KH111D7B08304LEN gpt    disk 223.6G                          
├─sdd1                                         gpt    part     1M BIOS boot                f220a939-33c2-4e2c-aefc-09e4a0d92151
├─sdd2                                         gpt    part   512M EFI System               cba8bdd0-6f57-46d7-a20a-6a2872bd5919
└─sdd3                                         gpt    part 223.1G Solaris /usr & Apple ZFS 1b959ea8-5d07-4c80-a856-163b9b4b8e1e
sde    SAMSUNG MZ7L33T8HELA-00A07              gpt    disk   3.5T                          
└─sde1                                         gpt    part   3.5T Solaris /usr & Apple ZFS 1fdc764f-5d69-4899-b1ea-90cbde3f7918
sdf    MTFDDAV240TGA-1BC16A 03KH111D7B08304LEN gpt    disk 223.6G                          
├─sdf1                                         gpt    part     1M BIOS boot                72ae5f1d-d8a5-417d-9a52-10b2362e72be
├─sdf2                                         gpt    part   512M EFI System               2eeef0f1-a4a2-47d5-8114-947c17e49d35
└─sdf3                                         gpt    part 223.1G Solaris /usr & Apple ZFS ec2f7f42-08ae-4af3-85f4-012c6c959869
sdg    Extreme Pro                             gpt    disk 119.3G                          
├─sdg1                                         gpt    part     1M BIOS boot                f6bb71c1-a996-4ff5-a714-95b208ac5a91
├─sdg2                                         gpt    part   512M EFI System               37506496-8304-4828-a828-18320eb901b8
└─sdg3                                         gpt    part 118.7G Solaris /usr & Apple ZFS 6b70a97b-2db5-4e70-8691-d67a9a75c1a2
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool   222G  2.83G   219G        -         -     0%     1%  1.00x    ONLINE  -
zpool      17.5T  2.66T  14.8T        -         -     4%    15%  1.00x  DEGRADED  /mnt

Won’t go off topic, but it’s concerning that your storage pool is degraded.

Can you also include this:

zpool status

The next step, we’ll see if you can mount and access your old boot-pool to retrieve the keystring, if it’s still possible.

NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool   222G  2.83G   219G        -         -     0%     1%  1.00x    ONLINE  -
zpool      17.5T  2.66T  14.8T        -         -     4%    15%  1.00x  DEGRADED  /mnt
truenas_admin@truenas[~]$ sudo zpool status
  pool: boot-pool
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdd3    ONLINE       0     0     0
            sdf3    ONLINE       0     0     0

errors: No known data errors

  pool: zpool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:

        NAME                                      STATE     READ WRITE CKSUM
        zpool                                     DEGRADED     0     0     0
          raidz2-0                                DEGRADED     0     0     0
            3819f55e-fcb6-4a78-8cac-6c56cd5b2680  ONLINE       0     0     0
            7ef6f78c-5a38-4dc9-99da-6243677b79f7  ONLINE       0     0     0
            ab235bf6-08e7-47e4-b70f-008689e5515f  ONLINE       0     0     0
            1fdc764f-5d69-4899-b1ea-90cbde3f7918  ONLINE       0     0     0
            18075899723185650379                  UNAVAIL      0     0     0  was /dev/disk/by-partuuid/4ed67b79-02cf-47fb-854b-b820f6f62d98

errors: No known data errors

Is there a reason you’re missing a drive in your RAIDZ2 vdev?

No idea. I set it up for 2 drives can fail. These were brand new SATAs.

Okay. That’s just a concern. You’re missing an entire drive. It might have something to do with your motherboard or HBA, if you’re using one.


One last thing before we continue:

zpool import

Every time you update TrueNAS you are asked if you want to save a copy of your configuration file, optionally including your encryption keys (will be saved as a .tar). You are also asked if you want to save a copy of the encryption keys when you create an encrypted dataset (saved as a .json).

Have you double-checked if you have either of those files in your downloads folder on your PC?

2 Likes
 pool: boot-pool
    id: 2650431139805676226
 state: ONLINE
status: Some supported features are not enabled on the pool.
        (Note that they may be intentionally disabled if the
        'compatibility' property is set.)
action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
config:

        boot-pool   ONLINE
          sdg3      ONLINE

I have two configuration files from May and March. I will try loading them.

You mean this whole time! :laughing:

First do what @neofusion said. I assumed you didn’t backup or export anything, since you said you never exported your keys, I figured you didn’t export your config either.

1 Like