Can't unlock pool after upgrade from core 13.3 to SCALE 24.10.1

Hello,

as preparation for the change from core to scale, i have backup my plugin and jail, install the iso of scale 24.10.1 and imported my pool. As i realized that the config backup from Core 13.3 doesn’t work, i used the key i had also backup in my bitwarden. copied the text and paste it to open “tank” which is my pool’s name. and i got this error.

'/mnt/tank' directory is not empty (please provide "force" flag to override this error and file/directory will be renamed once the dataset is unlocked)

after looking up i saw this tread cannot unlock pool after upgrade from CORE to SCALE 24.04. pre-release. from what i got this should have bug should have been resolve for the full release many version ago. i have a very limited experience searching for bug report in the new system (was used to JIRA). but i didn’t find report like this.

on a second point. what impact will have if i flag “force” and directory will be renamed. what directory will be renamed? is it only some system directory that i don’t really interact with or it will be impacting my data folder structure?

thanks
keven

What text? Passphrase or HEX string?


Do you have residual files or ghost folders under the path /mnt/tank?

Check:

ls -lhaR /mnt/tank

It should be empty if the root dataset is not mounted/unlocked.

EDIT: It should also be empty, even if it is unlocked and mounted (or unencrypted), but the datasets beneath it are all locked.

EDIT 2: It’s also possible that empty folders will exist within that share the same name as the child datasets.

The encryption is not with a hex key 64 characters long and look like this (I changed a few characters for obvious security reasons)

d70f1ba071677be0a78fb00f479e2d03fdb9e277354f64e36a586859ee667635

I do have many child dataset of tank which show in the command you asked me to run

truenas_admin@truenas[~]$ ls -lhaR /mnt/tank
/mnt/tank:
total 2.5K
drwxr-xr-x 5 root root 5 Dec 21 13:37
drwxr-xr-x 3 root root 3 Dec 21 13:08
drwxr-xr-x 2 root root 2 Dec 21 13:36 Backup
drwxr-xr-x 5 root root 5 Dec 21 13:37 backupvol1
drwxr-xr-x 3 root root 3 Dec 21 13:36 iocage
/mnt/tank/Backup:
total 1.0K
drwxr-xr-x 2 root root 2 Dec 21 13:36
drwxr-xr-x 5 root root 5 Dec 21 13:37
/mnt/tank/backupvo11:
total 2.5K
drwxr-xr-x 5 root root 5 Dec 21 13:37
drwxr-xr-x 5 root root 5 Dec 21 13:37
drwxr-xr-x 2 root root 2 Dec 21 13:37 Bibliotheque
drwxr-xr-x 2 root root 2 Dec 21 13:37 iocage
drwxr-xr-x 2 root root 2 Dec 21 13:37 testnextcloud
/mnt/tank/backupvol1/Bibliotheque:
total 1.0K
drwxr-xr-x 2 root root 2 Dec 21 13:37
drwxr-xr-x 5 root root 5 Dec 21 13:37
/mnt/tank/backupvol1/iocage:
total 1.0K
drwxr-xr-x 2 root root 2 Dec 21 13:37
drwxr-xr-x 5 root root 5 Dec 21 13:37
/mnt/tank/backupvol1/testnextcloud:
total 1.0K
drwxr-xr-x 2 root root 2 Dec 21 13:37
drwxr-xr-x 5 root root 5 Dec 21 13:37
/mnt/tank/iocage:
total 1.5K
drwxr-xr-x 3 root root 3 Dec 21 13:36
drwxr-xr-x 5 root root 5 Dec 21 13:37
drwxr-xr-x 3 root root 3 Dec 21 13:36 releases
/mnt/tank/iocage/releases:
total 1.5K
drwxr-xr-x 3 root root 3 Dec 21 13:36
drwxr-xr-x 3 root root 3 Dec 21 13:36
drwxr-xr-x 2 root root 2 Dec 21 13:36 13.2-RELEASE
/mnt/tank/iocage/releases/13.2-RELEASE:
total 1.0K
drwxr-xr-x 2 root root 2 Dec 21 13:36
drwxr-xr-x 3 root root 3 Dec 21 13:36
truenas_admin@truenas[~]$

truenas encryption

It is.


So is “tank” or any other children currently unlocked? There should be no reason to see all of those folders and subfolders if it “tank” is locked.

zfs list -t filesystem -o name,keystatus | grep -v '\.system\|boot'

sorry misstype that i meant that i do have a hex key like you said.

tank and its child are all locked with the exception of the following child that are not encrypted:
tank/Backup/Keven
tank/Backup/VBL-PC
tank/Windows_handbrake-06jlnd

did the command suggested and i got
zsh: command not found: zfs

Sigh. SCALE. :pensive:

Put sudo in front of the command.

This is likely the problem, throwing off the middleware.

In the past we could initially create an encrypted root dataset upon a new pool’s creation, with unencrypted children nested underneath. This has since been “fixed”.

Because your pool was likely created before this change was enforced in TrueNAS, you’re left with an “unsupported” (discouraged) dataset topology.[1]

In vanilla, upstream ZFS (without TrueNAS or a middleware in between), you can do this, as long as you understand the complications and risks involved.

I’m not sure what the “safe” approach is going forward for TrueNAS SCALE systems that carry over an older pool that break this enforcement.


If I was caught in this dilemma, I would export the pool, then boot into a recent Linux live ISO (from a USB stick), and then manually do a “shuffle” of the unencrypted datasets.

  1. Import the pool in the live Linux session with the -N flag, to prevent it from automounting the unencrypted datasets
  2. Create a checkpoint for pool tank
  3. Rename tank/Backup to tank/Backup-plain
  4. Rename tank/Windows_handbrake-06jlnd to tank/Windows_handbrake-06jlnd-plain
  5. Do a full, recursive replication of the -plain datasets into new ones that will inherit tank’s encryption, and use the original dataset names
  6. Confirm everything went well
  7. Destroy the -plain datasets
  8. Import the pool into TrueNAS SCALE
  9. Discard the checkpoint if everything went well

:skull_and_crossbones: :warning:
That’s just what I, Winnie, would try to do! Don’t try anything without knowing what you’re doing.

:skull_and_crossbones: :warning:


  1. Even on TrueNAS Core 13.3-U1, we can no longer create unencrypted datasets if the root dataset is encrypted. If you try, the GUI/middleware will stop you and print an error message with red text. ↩︎

thanks for the follow up @winnielinnie with sudo it did work.

I forgot a couple of nested/nested/nested datasets that are also NOT encrypted. all of them have been created because i zfs replication those datasets from my really old pool “vol1” which has been dismantled. Fortunately for me, all the datasets that are unencrypted are either very old backups from windows 7 days and backups of my old pool before i dismantle it, in case my data that has been merge into the same folder on the “tank” pool would create some sort of problem.
so basically i can delete all those redundant unencrypted datasets.

i’m not familiar at all with linux, so if i just boot back on TrueNAS CORE 13, go into the storage section of the GUI and delete all datasets that are unencrypted then boot back into TrueNAS SCALE. will this solve my problem?

[sudo] password for truenas_admin: 
NAME                                                        KEYSTATUS
tank                                                        unavailable
tank/Backup                                                 -
tank/Backup/Keven                                           -
tank/Backup/VBL-PC                                          -
tank/Bibliotheque                                           unavailable
tank/Bibliotheque/Media                                     unavailable
tank/Bibliotheque/Secret                                    unavailable
tank/Bibliotheque/famille                                   unavailable
tank/Bibliotheque/famille/Documents                         unavailable
tank/Bibliotheque/famille/Images                            unavailable
tank/Bibliotheque/famille/Jeux                              unavailable
tank/Bibliotheque/famille/Logiciel                          unavailable
tank/Bibliotheque/famille/Musique                           unavailable
tank/Bibliotheque/famille/Videos                            unavailable
tank/Bibliotheque/qbittorrent                               unavailable
tank/Bibliotheque/qbittorrent/completed                     unavailable
tank/Bibliotheque/qbittorrent/download                      unavailable
tank/Bibliotheque/qbittorrent/torrent                       unavailable
tank/Bibliotheque/qbittorrent/upload                        unavailable
tank/VBL                                                    unavailable
tank/VM                                                     unavailable
tank/backup_qbittorrent                                     unavailable
tank/backup_qbittorrent/root                                unavailable
tank/backupvol1                                             unavailable
tank/backupvol1/Bibliotheque                                -
tank/backupvol1/Bibliotheque/deluge                         unavailable
tank/backupvol1/Bibliotheque/deluge/completed               unavailable
tank/backupvol1/Bibliotheque/deluge/download                unavailable
tank/backupvol1/Bibliotheque/deluge/torrent                 unavailable
tank/backupvol1/Bibliotheque/deluge/upload                  unavailable
tank/backupvol1/Bibliotheque/qbittorrent                    unavailable
tank/backupvol1/Bibliotheque/qbittorrent/completed          unavailable
tank/backupvol1/Bibliotheque/qbittorrent/download           unavailable
tank/backupvol1/Bibliotheque/qbittorrent/torrent            unavailable
tank/backupvol1/Bibliotheque/qbittorrent/upload             unavailable
tank/backupvol1/iocage                                      -
tank/backupvol1/iocage/download                             -
tank/backupvol1/iocage/download/11.3-RELEASE                -
tank/backupvol1/iocage/download/11.4-RELEASE                -
tank/backupvol1/iocage/download/12.0-RELEASE                -
tank/backupvol1/iocage/download/12.1-RELEASE                -
tank/backupvol1/iocage/download/12.2-RELEASE                -
tank/backupvol1/iocage/download/12.3-RELEASE                -
tank/backupvol1/iocage/download/13.2-RELEASE                -
tank/backupvol1/iocage/images                               -
tank/backupvol1/iocage/jails                                -
tank/backupvol1/iocage/jails/2unifi                         -
tank/backupvol1/iocage/jails/2unifi/root                    -
tank/backupvol1/iocage/jails/UnifiController                -
tank/backupvol1/iocage/jails/UnifiController/root           -
tank/backupvol1/iocage/jails/nextcloud                      -
tank/backupvol1/iocage/jails/nextcloud/root                 -
tank/backupvol1/iocage/jails/openspeedtest                  -
tank/backupvol1/iocage/jails/openspeedtest/root             -
tank/backupvol1/iocage/jails/plexmediaserver                -
tank/backupvol1/iocage/jails/plexmediaserver/root           -
tank/backupvol1/iocage/jails/qbittorrent                    -
tank/backupvol1/iocage/jails/qbittorrent/root               -
tank/backupvol1/iocage/jails/syncme                         -
tank/backupvol1/iocage/jails/syncme/root                    -
tank/backupvol1/iocage/log                                  -
tank/backupvol1/iocage/releases                             -
tank/backupvol1/iocage/releases/11.3-RELEASE                -
tank/backupvol1/iocage/releases/11.3-RELEASE/root           -
tank/backupvol1/iocage/releases/11.4-RELEASE                -
tank/backupvol1/iocage/releases/11.4-RELEASE/root           -
tank/backupvol1/iocage/releases/12.0-RELEASE                -
tank/backupvol1/iocage/releases/12.0-RELEASE/root           -
tank/backupvol1/iocage/releases/12.1-RELEASE                -
tank/backupvol1/iocage/releases/12.1-RELEASE/root           -
tank/backupvol1/iocage/releases/12.2-RELEASE                -
tank/backupvol1/iocage/releases/12.2-RELEASE/root           -
tank/backupvol1/iocage/releases/12.3-RELEASE                -
tank/backupvol1/iocage/releases/12.3-RELEASE/root           -
tank/backupvol1/iocage/releases/13.2-RELEASE                -
tank/backupvol1/iocage/releases/13.2-RELEASE/root           -
tank/backupvol1/iocage/templates                            -
tank/backupvol1/testnextcloud                               -
tank/iocage                                                 unavailable
tank/iocage/download                                        unavailable
tank/iocage/download/13.2-RELEASE                           unavailable
tank/iocage/download/13.3-RELEASE                           unavailable
tank/iocage/images                                          unavailable
tank/iocage/jails                                           unavailable
tank/iocage/jails/UnifiController                           unavailable
tank/iocage/jails/UnifiController/root                      unavailable
tank/iocage/jails/openspeedtest                             unavailable
tank/iocage/jails/openspeedtest/root                        unavailable
tank/iocage/jails/plexmediaserver                           unavailable
tank/iocage/jails/plexmediaserver/root                      unavailable
tank/iocage/jails/qbittorrent                               unavailable
tank/iocage/jails/qbittorrent/root                          unavailable
tank/iocage/jails/syncme                                    unavailable
tank/iocage/jails/syncme/root                               unavailable
tank/iocage/log                                             unavailable
tank/iocage/releases                                        unavailable
tank/iocage/releases/13.2-RELEASE                           -
tank/iocage/releases/13.2-RELEASE/root                      -
tank/iocage/releases/13.3-RELEASE                           unavailable
tank/iocage/releases/13.3-RELEASE/root                      unavailable
tank/iocage/templates                                       unavailable
tank/off-site_backup_andre                                  unavailable
tank/off-site_backup_andre/Backup                           unavailable
tank/off-site_backup_andre/Backup/Andre                     unavailable
tank/off-site_backup_andre/Backup/Suzie                     unavailable
tank/off-site_backup_andre/Bibliotheque                     unavailable
tank/off-site_backup_andre/Bibliotheque/Documents           unavailable
tank/off-site_backup_andre/Bibliotheque/Images              unavailable
tank/off-site_backup_andre/Bibliotheque/Logiciel            unavailable
tank/off-site_backup_andre/Bibliotheque/Musique             unavailable
tank/off-site_backup_andre/Bibliotheque/Videos              unavailable
tank/test nextcloud                                         unavailable
tank/test nextcloud/root                                    unavailable
tank/test_syncthing                                         unavailable
truenas_admin@truenas[~]