Hi.
I recently upgraded from Scale 24.04 (Dragonfish) to Scale 24.10 (Electric Eel) and managed to perform the upgrade without reading the upgrade guide thoroughly enough, ultimately missing the information that stated applications on datasets in encrypted pools would not get upgraded.
I took snapshots before the upgrade, and since I run TrueNAS in a VM on my ESXi server I also took a snapshot of the VM (boot disk) before performing the upgrade with datasets locked (maybe this was a mistake…).
24.10 installed just fine and I was able to unlock my datasets after the upgrade. I could also install applications mounted in /mnt/.ix-apps as expected.
Since I noticed that my Minio app had not migrated successfully I read on the upgrade page that users manually could run the command midclt call -job k8s_to_docker.migrate <poolname> to re-initiate a failed migration of any previously-installed Kubernetes apps to Docker at any time after upgrading to Electric Eel.
This unfortunately did not work and resulted in the error:
root@truenas[~]# midclt call -job k8s_to_docker.migrate HDD
Status: (none)
Total Progress: [________________________________________] 0.00%
Total Progress: [________________________________________] 0.00%[EFAULT] Failed to list backups for 'HDD': Unable to locate '/mnt/HDD/ix-applications/backups' backups directory
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 509, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 556, in __run_body
rv = await self.middleware.run_in_thread(self.method, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1367, in run_in_thread
return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
res = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_to_docker/migrate.py", line 59, in migrate
raise CallError(f'Failed to list backups for {kubernetes_pool!r}: {backups["error"]}')
middlewared.service_exception.CallError: [EFAULT] Failed to list backups for 'HDD': Unable to locate '/mnt/HDD/ix-applications/backups' backups directory
On the TrueNAS Discord server there was another user that had faced the similar problem after the upgrade and seemingly solved it by creating a snapshot (unknown whether he booted back into 24.04 to do this, or if he did it in 24.10):
Harrison — 10/30/24, 3:38 AM
No backups found for 'SSD'
ah nice, now it's complaining about something else
Arimil — 10/30/24, 3:39 AM
I went through this a long time ago, the best thing you can do iirc create a new pool, then unseat the pool, copy the files from your old pool over to it and then reseat the new unencrypred pool.
Harrison — 10/30/24, 3:41 AM
if that's true that's a real shame
I'm feeling brave, I'll try booting back into dragonfish and re-upgrade now that I've decrypted ix-applications
let's see if it blows up a different way
Arimil — 10/30/24, 3:42 AM
Good luck
Harrison — 10/30/24, 3:45 AM
yea, the worst that will happen is that I'll have to redo everything as you suggest anyways lol
Harrison — 10/30/24, 4:59 AM
TLDR I just had to make a snapshot like ix-applications-backup-system-update--2024-10-30_01:34:57
and magically it started working (why couldn't the script make a snapshot....)
I decided to try the same approach and verified a snapshot had been created, but when investigating whether a backup file had been created I noticed that the /backups folder did not exist at all.
I was not able to get further at this point so I decided to boot back into 24.04 and try to unlock my dataset and move the Minio app with all my data to another dataset but was met with the following error when attempting to unlock the dataset:
Error details for HDD
'/mnt/HDD' directory is not empty (please provide "force" flag to override this error and file/directory will be renamed once the dataset is unlocked)
This error also shows up when I try to click on the ix-applications folder in the GUI:
CallError
[EFAULT] Failed retreiving USER quotas for HDD/ix-applications
remove_circle_outline
More info...
Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 76, in get_quota
with libzfs.ZFS() as zfs:
File "libzfs.pyx", line 529, in libzfs.ZFS.__exit__
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 78, in get_quota
quotas = resource.userspace(quota_props)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "libzfs.pyx", line 3680, in libzfs.ZFSResource.userspace
libzfs.ZFSException: cannot get used/quota for HDD/ix-applications: dataset is busy
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
res = MIDDLEWARE._run(*call_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
return methodobj(*params)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 80, in get_quota
raise CallError(f'Failed retreiving {quota_type} quotas for {ds}')
middlewared.service_exception.CallError: [EFAULT] Failed retreiving USER quotas for HDD/ix-applications
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1417, in _call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_quota_and_perms.py", line 225, in get_quota
quota_list = await self.middleware.call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1425, in _call
return await self._call_worker(name, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1431, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1337, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed retreiving USER quotas for HDD/ix-applications
Seemingly there is content in /mnt/HDD without anything being mounted yet, so I jumped over to the shell (on 24.04) to investigate it. Here you can see that all relevant datasets are unmounted and the mount paths seem correct. It is not possible to remove this directory either as it results in a permission denied error.
root@truenas[~]# ls -la /mnt/*
/mnt/HDD:
total 2
drwxr-xr-x 3 root root 3 Jan 6 15:39 .
drwxr-xr-x 4 root root 4 Jan 6 22:36 ..
drwxr-xr-x 2 root root 2 Jan 6 15:39 ix-applications
/mnt/SSHD:
total 9
drwxr-xr-x 2 root root 2 Jan 6 17:41 .
drwxr-xr-x 4 root root 4 Jan 6 22:36 ..
root@truenas[~]# ls -la /mnt/HDD/ix-applications/
total 1
drwxr-xr-x 2 root root 2 Jan 6 15:39 .
drwxr-xr-x 3 root root 3 Jan 6 15:39 ..
root@truenas[~]# zfs list -o name,mountpoint,mounted | grep HDD | grep ix
HDD/ix-applications /mnt/HDD/ix-applications no
HDD/ix-applications/catalogs /mnt/HDD/ix-applications/catalogs no
HDD/ix-applications/default_volumes /mnt/HDD/ix-applications/default_volumes no
HDD/ix-applications/k3s /mnt/HDD/ix-applications/k3s no
HDD/ix-applications/k3s/kubelet /mnt/HDD/ix-applications/k3s/kubelet no
HDD/ix-applications/releases /mnt/HDD/ix-applications/releases no
HDD/ix-applications/releases/minio /mnt/HDD/ix-applications/releases/minio no
HDD/ix-applications/releases/minio/charts /mnt/HDD/ix-applications/releases/minio/charts no
HDD/ix-applications/releases/minio/volumes /mnt/HDD/ix-applications/releases/minio/volumes no
HDD/ix-applications/releases/minio/volumes/ix_volumes /mnt/HDD/ix-applications/releases/minio/volumes/ix_volumes no
HDD/ix-applications/releases/minio/volumes/ix_volumes/export /mnt/HDD/ix-applications/releases/minio/volumes/ix_volumes/export no
HDD/ix-apps /mnt/.ix-apps no
HDD/ix-apps/app_configs /mnt/.ix-apps/app_configs no
HDD/ix-apps/app_mounts /mnt/.ix-apps/app_mounts no
HDD/ix-apps/docker /mnt/.ix-apps/docker no
HDD/ix-apps/truenas_catalog /mnt/.ix-apps/truenas_catalog no
root@truenas[~]#
If I however boot back into 24.10 and unlock my dataset there, everything works fine and I am able to access both my encrypted data, the new /mnt/.ix-apps for apps on EE, and the old /mnt/HDD/ix-applications. But not able to migrate the old MinIO app or the data within.
Are there any ways of resolving this, or will I have to manually mirror out my data from the old no longer working MinIO app by sshfs mounting the dir and somehow mirror the S3 objects out to a new MinIO instance?
There is possibly a chance I could perform a new upgrade if I get my dataset unlocked properly while on 24.04, but I am not sure how to go forth. Are there any other commands I can run before midclt call -job k8s_to_docker.migrate HDD to manually perform the upgrade?