Unlocking Dataset Causes Reboot

TrueNAS General
1

Not really sure what’s happening and kinda stumped. I have one pool (raidz3) that has an encrypted dataset. Not sure what the reason is but every time I load the key and unlock it the machine just restarts.

Things I’ve tried:

  • Rollbacked from 24.04.x.x to 23.10.2
  • Reverted settings I changed (ssl certs)
  • Tried loading the key from the shell
  • turned off dedupe
  • memtest

The pool is around 47Tib and zpool & dataset is 43Tib, the server has 128 GB of RAM. Not sure but it might be running out of RAM when it’s trying to unlock it. I’ve always been able to unlock it before so not sure what would make it do it now.

2

You had already enabled and used deduplication…?

1 Like
3

Yes, it was already enabled. Turning off deduplication was the last thing I did.

4

Disabling deduplication doesn’t change the existing data in place, including the memory required to load the deduplication tables.

I understand you have a single RAIDZ3 pool that’s ~47TiB - are there other datasets that mount fine, but when attempting to unlock the ~43TiB encrypted dataset, it restarts?

Is dedup set on the pool, all datasets, some datasets?

1 Like
5

Figured it wouldn’t hurt to turn it off and likely won’t do anything at this point.

It’s the only dataset in the pool and dedup was only setup on the encrypted dataset. Unfortunately, I do not have any other dataset to confirm.

6

If you can import the pool, then you can still issue zpool and zfs commands without unlocking any datasets (which is crashing the system.)

Some that might give more clues:

zpool list <poolname>

zfs list -r -t filesystem,volume -o space,dedup,encryption,encroot,keyformat <poolname>
1 Like
7

Nothing stands out to me

NAME          SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
PoolName  69.9T  58.1T  11.7T        -         -    57%    83%  1.10x    ONLINE  /mnt

NAME                                                          AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  DEDUP          ENCRYPTION   ENCROOT                      KEYFORMAT
PoolName                                                   5.39T  44.9T        0B    256K             0B      44.9T  off            off          -                            none
PoolName/.system                                           5.39T  1.91G        0B   1.23G             0B       700M  off            off          -                            none
PoolName/.system/configs-ae32c386e13840b2bf9c0083275e7941  5.39T  31.8M        0B   31.8M             0B         0B  off            off          -                            none
PoolName/.system/cores                                     1024M   256K        0B    256K             0B         0B  off            off          -                            none
PoolName/.system/ctdb_shared_vol                           5.39T   256K        0B    256K             0B         0B  off            off          -                            none
PoolName/.system/glusterd                                  5.39T   256K        0B    256K             0B         0B  off            off          -                            none
PoolName/.system/netdata-ae32c386e13840b2bf9c0083275e7941  5.39T   664M        0B    664M             0B         0B  off            off          -                            none
PoolName/.system/rrd-ae32c386e13840b2bf9c0083275e7941      5.39T   256K        0B    256K             0B         0B  off            off          -                            none
PoolName/.system/samba4                                    5.39T  1.97M     1023K    991K             0B         0B  off            off          -                            none
PoolName/.system/services                                  5.39T   256K        0B    256K             0B         0B  off            off          -                            none
PoolName/.system/webui                                     5.39T   256K        0B    256K             0B         0B  off            off          -                            none
PoolName/zvol3name                                         6.24T  1.17T        0B    320G           879G         0B  off            off          -                            none
PoolName/datasetName                                       5.39T  43.3T        0B    426K             0B      43.3T  off            aes-256-gcm  PoolName/datasetName      passphrase
PoolName/datasetName/VM2zvol                               6.78T  43.3T        0B   41.9T          1.39T         0B  off            aes-256-gcm  PoolName/datasetName      passphrase
PoolName/testdataset                                       5.39T   256K        0B    256K             0B         0B  off            off          -                            none
PoolName/VM1zvol                                           5.44T   140G        0B   80.4G          60.1G         0B  off            off          -                            none

2024-10-11.13:14:14 py-libzfs: zfs load-key  -L file:///tmp/tmp0i58_rzv -n PoolName/datasetName
2024-10-11.13:20:45 zpool import 13530795738899061307 -R /mnt -m -f -o cachefile=/data/zfs/zpool.cache
8

Sadly, you might be dealing with what @HoneyBadger explained earlier.

NAME                            AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  DEDUP   ENCRYPTION   ENCROOT                  KEYFORMAT
PoolName/datasetName/VM2zvol    6.78T  43.3T        0B   41.9T          1.39T         0B  off      aes-256-gcm  PoolName/datasetName      passphrase

I’m assuming this VM2zvol is where you had originally created it with deduplication enabled, wrote over 43 TiB worth of data, and now as suggested, your 128 GiB of RAM isn’t enough to hold the DDT in memory.

But that’s where I have to defer to others. I’m not entirely sure what the RAM requirements are for dedup.

I do believe that smaller “recordsizes” are more taxing, since it takes more blocks (and hence more entries in the table) for the same amount of total data.

What is the recordsize for VM2zvol? (Assuming you didn’t change it.)

zfs get recordsize,volblocksize PoolName/datasetName/VM2zvol
9

Just gives me a “-” likely cause it’s locked(?). Oh well. Thankfully we have 2 copies. Thank you everyone!

10

For both? One of the two properties should display a value. It will either be the recordsize or volblocksize property.

I’m assuming it’s a zvol by the “name”, but I included both properties, just in case.

11

My bad, zvol is giving me 64k volblocksize and the dataset is giving me 128k recordsize.

12

I think it may very well be that upon trying to mount the volume, there isn’t enough free RAM to hold all of those entries in the DDT.

To “un-dedup” the volume, you’d have to be able to safely mount, and then copy everything over again to a dataset/zvol with the dedup property disabled.

But you can’t even get that far, since unlocking the 44 TiB volume automatically attempts to mount it. Even if you don’t automatically mount it upon unlocking… you’d still need to mount it eventually.

But from what you said, it sounds like you have a very good, up-to-date backup of this specific data? (Is it also using deduplication?)