App Config has incorrect pool mounted on /mnt/.ix-apps, does not update

Apps and Virtualization
1

I got bitten by a seagate replacement drive being a few bytes smaller than the original drives. I updated my system to 25.x, created a new temp pool, restored a snapshot to that pool, moved my shares, apps (plex and immich) and system mount points to point at the new temp pool, deleted the original pool/dataset, re-created the pool with the four now slightly mismatched drives, then restored the temp data over to the new pool.

The data seems to have transferred great, but my Applications config is broken. There is a red X on the Application page that says ā€œApplication(s) have failed to start:
[EFAULT] ā€˜HDDs/ix-appsā€™ dataset is not mounted on ā€˜/mnt/.ix-appsā€™ā€. It turns out, this is correct, itā€™s mounted on my boot pool, but when I try to change the pool (or unset, then ā€œchoose poolā€), it throws the same error, with this stack trace:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 515, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 560, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 88, in wrapped
    result = await func(*args)
             ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/update.py", line 122, in do_update
    await self.middleware.call('docker.setup.status_change')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 977, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 692, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/state_setup.py", line 81, in status_change
    await self.middleware.call('docker.state.start_service')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 977, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 692, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/state_management.py", line 47, in start_service
    await self.before_start_check()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/state_management.py", line 16, in before_start_check
    await self.middleware.call('docker.setup.validate_fs')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 977, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 692, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/state_setup.py", line 59, in validate_fs
    raise CallError(f'{config["dataset"]!r} dataset is not mounted on {IX_APPS_MOUNT_PATH!r}')
middlewared.service_exception.CallError: [EFAULT] 'HDDs/ix-apps' dataset is not mounted on '/mnt/.ix-apps'

No matter what I do in the UI, when I look at /mnt/.ix-apps in the console, it shows:

truenas_admin@TheHive[~]$ df -h /mnt/.ix-apps
Filesystem                  Size  Used Avail Use% Mounted on
boot-pool/ROOT/25.04.0/mnt  442G  165M  442G   1% /mnt

And finally, hereā€™s my zfs mountpoints related to ix-apps:

root@TheHive[/home/truenas_admin]# zfs get mountpoint | grep .ix-apps
HDDs/ix-apps                                                      mountpoint  /mnt/HDDs/ix-apps                                default
HDDs/ix-apps@auto-2025-05-25_00-00                                mountpoint  -                                                -
HDDs/ix-apps/app_configs                                          mountpoint  /mnt/HDDs/ix-apps/app_configs                    default
HDDs/ix-apps/app_configs@auto-2025-05-25_00-00                    mountpoint  -                                                -
HDDs/ix-apps/app_mounts                                           mountpoint  /mnt/HDDs/ix-apps/app_mounts                     default
HDDs/ix-apps/app_mounts@auto-2025-05-25_00-00                     mountpoint  -                                                -
HDDs/ix-apps/app_mounts/immich                                    mountpoint  /mnt/HDDs/ix-apps/app_mounts/immich              default
HDDs/ix-apps/app_mounts/immich@auto-2025-05-25_00-00              mountpoint  -                                                -
HDDs/ix-apps/app_mounts/immich/backups                            mountpoint  /mnt/HDDs/ix-apps/app_mounts/immich/backups      default
HDDs/ix-apps/app_mounts/immich/backups@auto-2025-05-25_00-00      mountpoint  -                                                -
HDDs/ix-apps/docker                                               mountpoint  /mnt/HDDs/ix-apps/docker                         default
HDDs/ix-apps/docker@auto-2025-05-25_00-00                         mountpoint  -                                                -
HDDs/ix-apps/truenas_catalog                                      mountpoint  /mnt/HDDs/ix-apps/truenas_catalog                default
HDDs/ix-apps/truenas_catalog@auto-2025-05-25_00-00                mountpoint  -                                                -
Tempool/ix-apps                                                   mountpoint  /mnt/Tempool/ix-apps                             default
Tempool/ix-apps@auto-2025-05-25_00-00                             mountpoint  -                                                -
Tempool/ix-apps/app_configs                                       mountpoint  /mnt/Tempool/ix-apps/app_configs                 default
Tempool/ix-apps/app_configs@auto-2025-05-25_00-00                 mountpoint  -                                                -
Tempool/ix-apps/app_mounts                                        mountpoint  /mnt/Tempool/ix-apps/app_mounts                  default
Tempool/ix-apps/app_mounts@auto-2025-05-25_00-00                  mountpoint  -                                                -
Tempool/ix-apps/app_mounts/immich                                 mountpoint  /mnt/Tempool/ix-apps/app_mounts/immich           default
Tempool/ix-apps/app_mounts/immich@auto-2025-05-25_00-00           mountpoint  -                                                -
Tempool/ix-apps/app_mounts/immich/backups                         mountpoint  /mnt/Tempool/ix-apps/app_mounts/immich/backups   default
Tempool/ix-apps/app_mounts/immich/backups@auto-2025-05-25_00-00   mountpoint  -                                                -
Tempool/ix-apps/docker                                            mountpoint  /mnt/Tempool/ix-apps/docker                      default
Tempool/ix-apps/docker@auto-2025-05-25_00-00                      mountpoint  -                                                -
Tempool/ix-apps/truenas_catalog                                   mountpoint  /mnt/Tempool/ix-apps/truenas_catalog             default
Tempool/ix-apps/truenas_catalog@auto-2025-05-25_00-00             mountpoint  -                                                -
1 Like
2

Iā€™ve got the same issue here, tried migrating the pools through the UI which just ended up deleting all my .ix-apps configurations in /mnt/.ix-apps which was mounted on my boot disk. Created a new pool on my new 1TB ssd, switched the pool in the Apps UI but every time i install an app it still uses the /mnt/.ix-apps which is mounted on the boot disk folder instead of /mnt/Apps/.ix-apps.

3

Ok no I have looked a bit wrong there, it seems to just remount the /mnt/.ix-apps to the new pool, but in general just weird behaviour especially with just deleting all my old configurations. Would very much prefer when switching pools that truenas just automatically creates a new .ix-apps folder inside the pool and migrates all the data.

4

The situation with .ix-apps is a separate issue that came about with recent upgrades in both 24.10 and 25.04 paths. Your side effect with the hard is just fallout.

5

This situation seems to have self-corrected now that itā€™s after June 1st.

6

Exact same issue here, had a 2TB disk that failed, it somehow was formatted to 2048GB and all the new drives format to 2000GB.
Created a snapshot to external USB, exported my pool, created a new pool with same name as old one.

Same issue, the /mnt/.ix-apps is mounting to boot-pool and idk how to change it. I tried basic things like umount like a noob and its not working.
Iā€™d really like to just get my apps and configs thats all restored now just to work.

How is this mounting happening, canā€™t we just fix the config and reboot?