Dataset locked after update to 25.10.3

TrueNAS General
,
1

This morning I updated my SCALE machine from 25.10.1 to 25.10.3. I downloaded the config backup including the secret key, and everything seemed OK after the update. However, I just realized that one of my encrypted datasets is now “locked”, and the key isn’t in the backup!

The configuration backup I downloaded before the update is a tar file containing freenas-v1.db and pwenc_secret. I naively tried uploading the latter as the keyfile, but that didn’t work. Some LLM-assisted troubleshooting got me this far:

$ sqlite3 freenas-v1.db ".headers on" ".mode column" "SELECT * FROM storage_encrypteddataset;"
id  name               encryption_key                kmip_uid
--  -----------------  ----------------------------  --------
10  lime-pool/Private  [redacted]

11  lime-pool/Shared   [redacted]

13  lime-pool/home     [redacted]

15  lime-pool/Backups  [redacted]

19  lime-pool/Media    [redacted]

That accounts for all of my encrypted datasets, except for the now-locked one (lime-pool/Pictures) which doesn’t appear. That would make sense to see in my live system’s database, but this was exported back when the dataset was in fact unlocked, which means the Truenas still had the key. Could the key be stored somewhere else? Maybe kept in memory, never written to disk?

On the live system, here are a couple investigative zfs queries:

# zfs get encryption,keyformat,keylocation lime-pool/Pictures
NAME                PROPERTY     VALUE        SOURCE
lime-pool/Pictures  encryption   aes-256-gcm  -
lime-pool/Pictures  keyformat    hex          -
lime-pool/Pictures  keylocation  prompt       local

# zfs get keystatus lime-pool/Pictures
NAME                PROPERTY   VALUE        SOURCE
lime-pool/Pictures  keystatus  unavailable  -

The only difference between the above and an “Unlocked” dataset is that keystatus is available.

This problem dataset is the most-recently-created of all my encrypted datasets; I did some reorganizing a few months ago and split these files out of a different dataset. Or, maybe it was a zfs rename operation? I can’t remember. I’m pretty sure I at least power-cycled the system once or twice since that point, but this was probably the first system update since then.

I know that all my encrypted datasets were set up with the default encryption settings. That makes it really odd that just one of them has bugged out like this! It also means that they all have key-files, not passphrases. I really wish I had downloaded all those keyfiles and put them in my password manager, but that’s hindsight; I assumed the backup tarfile would be enough.

I am holding out hope that the missing key is in that db file somewhere. If not, it looks like I basically hit myself with a ransomware attack! Any help is welcome.

1 Like
2

Yeah, this is why I don’t really run encryption. I don’t have any state secrets that a state actor would like to lay their hands on. I really only encrypt things on mobile devices that can be lost in public. There is zero chance of such things happening with my NAS.

You’re neither the first one nor the last one for sure in these forums that hit themselves with self-imposed ransomware.

Have you tried downgrading and seeing if the dataset goes back to unlocked?

1 Like
3

To qualify what “downgrade” should mean in this context; do NOT uninstall or reinstall anything.

Your old TrueNAS version is still in place as an inactive boot environment available without installing or -grading anything by going to System → Boot and then activating your previous version. This works as long as you…

  • …haven’t manually removed your old boot environments
  • …didn’t upgrade your ZFS versions to a version the previous boot environment doesn’t support.

If it still works in the old version you can make sure you have a working backup of any data you can’t easily recreate.

Ultimately I suspect your issue is solvable, but I lack the experience to guide you there. Perhaps someone else can assist.

2 Likes
4

It would be nice if more people knew about this simple trick.

There’s a possibility that the encrypted key exists on an older config db. TrueNAS usually creates daily backups of this on your System Dataset. You can look under /var/db/system/configs-XXXX/

There’s a possibility that renaming a dataset (which bypasses the middleware) might have the same effect as exporting a pool. The dataset doesn’t “exist” anymore, so it’s possible that the key for the “disappeared” dataset was removed from the config db.

1 Like
5

Ah! That’s really good to know; I assumed that rolling-back would entail another installation over the current (broken) system. What you describe is more promising and less risky. Lucky this happened on a minor update with no new ZFS versions or anything.

Awesome, I will dig around there and see what I can find. Fingers crossed!

6

Ok, so there is a shred of hope!

Digging through /var/db/system/configs-XXXX/ turned out very fruitful. It looks like the encryption key fell out of the database when I moved the dataset from lime-pool/Media/Pictures to lime-pool/Pictures with a zfs rename operation. This was a couple months ago so my memory is hazy, but it shows up in the daily backups at the old location up until a certain point, then it vanishes.

I haven’t tried re-activating the 25.10.1 boot environment yet. My suspicion is that, until the update, I never rebooted the system after zfs rename-ing the dataset, so it stayed unlocked for months with the key in volatile memory only (gone from the database). So, booting into the most-recent 25.10.1 state won’t fix it, I don’t think. Even if I could boot into the Feb-2026 state, the dataset path wouldn’t match up. Should be safe to try anyway, but for now I’m treading very lightly!!

Anyway, the upshot is that I have an encryption_key value I want to try out, but it’s an encrypted 141-char string from an old db file, not a 64-char hex string I can type into the webUI. Does anyone know how to use pwenc_secret to decrypt these strings?

Edit: Should have just googled before posting! There is a python script in this thread that did the job: Reddit - Please wait for verification

Reproduced here for posterity:

Summary 
#!/usr/bin/python3
# https://milek.blogspot.com/2022/02/truenas-scale-zfs-wrapping-key.html
# based on /usr/lib/migrate113/freenasUI/system/migrations/0022_cloud_sync.py

import sys
import base64
from Cryptodome.Cipher import AES
import sqlite3

PWENC_BLOCK_SIZE = 32
PWENC_FILE_SECRET = '/data/pwenc_secret'
PWENC_PADDING = b'{'

def pwenc_get_secret():
    with open(PWENC_FILE_SECRET, 'rb') as f:
        secret = f.read()
    return secret

def pwenc_decrypt(encrypted=None):
    if not encrypted:
        return ""
    from Cryptodome.Util import Counter
    encrypted = base64.b64decode(encrypted)
    nonce = encrypted[:8]
    encrypted = encrypted[8:]
    cipher = AES.new(
        pwenc_get_secret(),
        AES.MODE_CTR,
        counter=Counter.new(64, prefix=nonce),
    )
    return cipher.decrypt(encrypted).rstrip(PWENC_PADDING).decode('utf8')

if len(sys.argv) == 2:
    print(pwenc_decrypt(sys.argv[1]))
    exit(0)

dbcon = sqlite3.connect('/data/freenas-v1.db')
dbcur = dbcon.cursor()
for row in dbcur.execute('select * from storage_encrypteddataset'):
    ds_id, ds_name, ds_enc_key, kmip_enc_key = row
    #print(ds_id, ds_name, ds_enc_key, pwenc_decrypt(ds_enc_key))
    print(f'dataset: {ds_name}\n  key: {pwenc_decrypt(ds_enc_key)}\n');

I ran this on my laptop, not the truenas box, so I edited the hard-coded secret file path to my downloads folder. Make sure to put the encrypted string in quotes.

And with that…thank you guys so so so much! I never would have known where to find those daily db backups. It’s been a tough day, but I think now I can finally relax…

7

As a novice into TrueNAS, this scares me to shit.

But OP, when I created my datasets with encryption enabled, TrueNAS displayed the key so I saved them on Bitwarden.

I’m assuming I’m “safe” or am I missing something here??

8

Well, I escaped by the skin of my teeth, so don’t worry too much! Maybe worry just a tiny, appropriate amount.

I’ll admit that I checked the “encryption” box without really thinking it through, and I feel pretty dumb now. My system actually worked like this:

  1. Unauthorized network access is prevented by passwords, user permissions, etc, but can be bypassed via physical access to the drives
  2. Unauthorized physical access to the drives is prevented by at-rest data encryption
  3. Unauthorized access to the keys for this at-rest data encryption are securely encrypted at-rest on the boot drive
  4. The master key for all those other keys is helpfully stored on another physical drive in the same box, unencrypted

Real secure! As long as the attacker doesn’t have an M.2 slot on their machine… :zany_face:

Not sure if I’m really the one to ask, but if you have the non-encrypted 64-hex-char strings in your Bitwarden, that might be smart – it sure would have saved me a lot of stress! Yes, if you type that 64-char string into the webUI, it will decrypt the dataset even if the boot drive has totally lost track of the key (ask me how I know…). You can get these keys from the webUI at any time by clicking “Export Key”, so long as the dataset’s unlocked already.

But, I wouldn’t delete those keys off your TrueNAS boot drive just because they’re in a password manager; I would hold on to both as a kind of “backup”. Losing the keys is the same as losing your data, and everyone knows you need backups of your data. What am I gonna do if my boot drive bites the dust? Or, what if something happens to your password vault?

The trick mentioned above with SHA hashes of memorable files/photos is clever, and gives you an excuse to essentially have many copies of your keyfile lying around on different machines, which accomplishes this goal. Better than my current solution of three hundred config.db and secret-key files in my downloads folder :wink:

9

It did warn me to keep the key safe, so I saved it (plain text long numbers and what not) into Bitwarden coz you won’t have plain text access to it again, allegedly.

My stuff are 99% NFS shares mounted on Proxmox and my Linux PC, access is granted to groups and groups only so the group “media(jellyfin, radarr, sonarr, jackett, qbittorrent)” has access to a dataset.

Back to your case, you do have off-site backup, right??

NAS is not backup :sweat_smile:

I’m in that process right now, the best option for me to have an off-site backup appears to be Brightbox in UK, they actually have a post how to encrypt the whole backup before sending it using restic. Which again, save everything on a Vault.

You mentioned “it looks like I basically hit myself with a ransomware attack”, I’m assuming you don’t have an off-site backup plan.

I guess start looking into that for yesterday, never trust your local stuff only.

10

Nobody needs a picture in order to generate a passphrase.

I simply convert to hex a phrase/string that I know.

You can create your own pattern/string and store it in your password vault.

11

I won’t be as kind going forward with you. You pick pointless fights for the most pedantic reasons on these forums.

If people took your advice, rather than an idiot like me, they wouldn’t have found out about their bad RAM, causing system issues, from a simple memtest.

Funny how you didn’t mention this part:

Tell me, oh wise one @truenas-fan, what is the difference between

8a96262a49cbfc3251173424896233c328b4f6bf0778919da10fee6270406221

and

8a96262a49cbfc3251173424896233c328b4f6bf0778919da10fee6270406221

?

Both can be used as keystring for native ZFS encryption. Both can be exported and saved. Both can be stored in a password vault.

Does it matter that one of them is the SHA256 hash of a JPEG whose significance is known only to you?

It costs you nothing extra, but if all else fails or the worst-case scenario happens with your backed up key, you can still retrieve your keystring.

@truenas-fan: “Nobody needs a phrase/string in order to generate a passphrase.”

1 Like
12

What the OP went through is not typical. It stems from a combination of actions but primarily I would say that it’s due to using zfs commands that the GUI wasn’t aware of.

I am not saying this to place blame, simply that if you use zfs commands that change your pool make sure you have plan for how you will then have TrueNAS account for said changes.

1 Like
13

This is why requests for including native ZFS features in TrueNAS are important. Not only will the middleware be aware of the actions, but the feature will have sanity checks.

If “Rename Dataset” was a feature in the GUI, TrueNAS would have updated the stored keys in the db file, rather than assume “The dataset is gone. I’ll remove the unnecessary key from the db now.”

1 Like
14

Agreed.

There’s a Feature Request, albeit not especially eloquently worded, that asks for zfs rename in the GUI.

15

I still can’t believe that they still don’t have rename after all these years. It’s one of the basic file operations.

Instead, you have to do this janky export-import CLI → GUI handoff dance.

1 Like
16

For those that would like to take existing keys and save them to password manager, is the sqlite command above the only way to display the keys? Or can we display them in the system somewhere?

Thanks! After reading this thread I just want to get the keys and save them to keepass.

17

The GUI doesn’t have a way to show you the keystring.

You can export all the keys used in the pool to save a .json file, which can be opened up in a text editor. From here, you’ll be able to copy the keystrings. There’s an “Export Key” option on the Datasets page for any “encryptionroot” dataset.

1 Like
18

I see, that is perfect, thanks!

1 Like