Upgrading from 24.04 to 24.10 .ix-apps dataset is not created

Apps and Virtualization
1

Hi,
So, I have a pool created on 24.04.2.5. All apps are up to date.
In essence, I want to update to 24.10.
I have tried the following:
Upgrade through UI manually from 24.04.2.5 to 24.10
Upgrade through UI by changing train from 24.04.2.5 to 24.10.2
Import the Pool directly to cleanly installed 24.10 and 24.10.2
All of the above result in this error:
`Failed to load datasets

[Errno 2] No such file or directory: ‘/mnt/.ix-apps/app_configs’ `

Truenas seems to not create (or mount) the /mnt/.ix-apps ([Pool]/.ix-apps) dataset in 24.10.
This crashes all datasets if the update is performed with app pool set and when unset choosing pool creates the error.
The main difference between 24.04 and 24.10 in this aspect seems to be the exclusive mount for the apps.
Is there a way to manually create this dataset i.e. CLI command?
Better yet, is there a way to manually migrate the apps (settings and data) from 24.04 to 24.10 when it fails to do this automatically when updating?

The problem is that on this Pool, I can neither get my old app setups working, nor can I force it to ignore the previous settings and setup all apps from scratch, keeping only the data.

Similar Topics but with confusing subjects have been created, specifically Nos 23439 and 22869 however they do not seem applicable on my case.

2

As someone who hasn’t updated yet but have been following the 24.10 update play out in the forums, my only guess would be that your current apps dataset is encrypted.

Is it?

You can retry a migration using the CLI with, as per the release notes:

midclt call -job k8s_to_docker.migrate poolname
3

Hi,
Thanks for the reply.
trying to run the command I get the following message:

admin@truenas[~]$ midclt call -job k8s_to_docker.migrate SSD_Pool
Status: (none)
Total Progress: [________________________________________] 0.00%Total Progress: [________________________________________] 0.00%
[EFAULT] Failed to list backups: [Errno 2] No such file or directory: '/mnt/.ix-apps/app_configs'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 488, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 535, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *args)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1361, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_to_docker/migrate.py", line 55, in migrate
    raise CallError(f'Failed to list backups: {backup_config_job.error}')
middlewared.service_exception.CallError: [EFAULT] Failed to list backups: [Errno 2] No such file or directory: '/mnt/.ix-apps/app_configs'

In a mock setup (clean 24.10, apps created directly) I see there is such a thing as [poolname]/.ix-apps dataset mounted on /mnt/.ix-apps.

My guess is that the dataset (which doesn’t exist on 24.04) fails to be created during the upgrade, hence the midctl command fails as well.
It is a system dataset and I would like to create it properly within the pool. After creating it I presume things will proceed as they should but I have no clue how to do that.

P.S. no, I have not encrypted the dataset nor the pool

4

While I have seen someone in the forums get past that error message by, as you say, create the dataset manually, I am uncertain as to what side-effects that will have down the road*.

While I’m sure iX has a lot on their plate already, I think this is another case where a proper bug report is in order.

Something is causing ix-apps/app_configs to not get created as expected and it would be good if iX could isolate that and make the migration more robust.

*Specifically, I have a faint memory of /mnt/.ix-apps not actually being a dataset, rather just a mount point for the dataset that lives on one of the existing pools.

5

Thank you, it seems you are right, it is indeed a mountpoint.
I am not too confident in truenas CLI usage and didn’t risk it at first.
Kudoz to @freshpots for his solution, which I used, only somewhat tweaked.
looking at

zfs get mountpoint tank/ix-apps

After the upgrade there was no mountpoint!

Using the command by @freshpots the system somehow created the mountpoint as
/mnt/mnt/.ix-apps
The correct command for me was:

zfs set mountpoint=/.ix-apps tank/ix-apps  # "/mnt" ommited in mountpoint

After that, your suggestion,

midclt call -job k8s_to_docker.migrate poolname

worked and everything is up and running. :smiley:

2 Likes