Apps pool / ix-apps stuck at /mnt/mnt/.ix-apps on SCALE 25.04.2.6, apps no longer start

TrueNAS General
,
1

Hey everyone,

first of all thanks upfront for all the helpful stuff in here - really glad for this community.
I face some serious issues with double mount (/mnt/mnt) and ix apps not working.

I spent some 4h with this forum and Perplexity to try and figure this out but I did not manage to solve it, so hopefully someone here has an inspiration on what might be the issue here.

Below I posted the generated report / discussion I had with Perplexity and what I tried in the terminal.

Thanks upfront for any help :slight_smile:

System

  • TrueNAS SCALE: 25.04.2.6 (from boot-pool/ROOT/25.04.2.6 dataset)
  • Pools:
    • raider (has its own .ix-virt and ix-apps tree, currently not the active Apps pool)
    • zfsa (intended Apps pool, holds media, Immich data, etc.)
    • zfsb (another pool with its own ix-apps tree, not active)
  • Apps: using the built‑in Apps (not TrueCharts); one key app is Immich.

I’m running into a persistent mountpoint problem with the ix-apps dataset on pool zfsa. The short version:

  • The GUI Apps page can load the catalog again, but
  • Apps refuse to start, and
  • The ix-apps dataset for zfsa keeps ending up mounted on /mnt/mnt/.ix-apps (or mounted there internally), despite multiple attempts to fix the ZFS mountpoints and export/import the pool.

I’ve gone as far as stopping middlewared, Docker, and k3s, killing dockerd, resetting properties, and performing zpool export/import, but ZFS still insists that zfsa/ix-apps is mounted at /mnt/mnt/.ix-apps according to its own view, even when mount initially looks clean.

I’d really appreciate guidance on the canonical / supported way to repair ix-apps on an existing Apps pool without destroying user data (zfsa/immich, zfsa/share, etc.).

Initial symptom

Initially, my Apps pool was zfsa. The GUI Apps page complained that the Apps dataset was not mounted at the expected path, and the catalog / apps did not work.

The core dataset looked like this:

sudo zfs get mountpoint,canmount,readonly zfsa/ix-apps
NAME          PROPERTY    VALUE              SOURCE
zfsa/ix-apps  mountpoint  /mnt/mnt/.ix-apps  local
zfsa/ix-apps  canmount    noauto             local
zfsa/ix-apps  readonly    off                inherited from zfsa

Recursive mountpoint output for the ix-apps tree on zfsa:

sudo zfs get -r mountpoint zfsa | egrep 'NAME|ix-apps'
NAME                                                                         PROPERTY    VALUE                                            SOURCE
zfsa/ix-apps                                                                 mountpoint  /mnt/mnt/.ix-apps                                local
...
zfsa/ix-apps/app_configs                                                     mountpoint  /mnt/mnt/.ix-apps/app_configs                    inherited from zfsa/ix-apps
zfsa/ix-apps/app_mounts                                                      mountpoint  /mnt/mnt/.ix-apps/app_mounts                     inherited from zfsa/ix-apps
...
zfsa/ix-apps/docker                                                          mountpoint  /mnt/mnt/.ix-apps/docker                         inherited from zfsa/ix-apps
zfsa/ix-apps/truenas_catalog                                                 mountpoint  /mnt/mnt/.ix-apps/truenas_catalog                inherited from zfsa/ix-apps

And it was effectively mounted there:

mount | egrep 'ix-apps'
zfsa/ix-apps on /mnt/mnt/.ix-apps type zfs (rw,noatime,xattr,posixacl,casesensitive)

From the docs and other threads, the expected situation for the Apps dataset is that it’s mounted under /mnt/.ix-apps (hidden directory under /mnt, not under /mnt/<pool>).

Fix attempt 1: Correct pool root mountpoint

I realized the root cause of the double /mnt/mnt might be the pool root mountpoint of zfsa itself, so I fixed that first.

After a previous export/import, I now have:

sudo zfs get mountpoint zfsa
NAME  PROPERTY    VALUE       SOURCE
zfsa  mountpoint  /mnt/zfsa   default

And ZFS mounts it correctly:

mount | egrep 'zfsa'
zfsa on /mnt/zfsa type zfs (rw,noatime,xattr,posixacl,casesensitive)
zfsa/pCloudMirror on /mnt/zfsa/pCloudMirror type zfs (...)
zfsa/share on /mnt/zfsa/share type zfs (...)
zfsa/immich on /mnt/zfsa/immich type zfs (...)
...

So the pool root is now cleanly at /mnt/zfsa (no more /mnt/mnt/zfsa).

Fix attempt 2: Set ix-apps → /mnt/.ix-apps

With zfsa root clean, I tried to update zfsa/ix-apps:

sudo zfs set mountpoint=/mnt/.ix-apps zfsa/ix-apps
sudo zfs set canmount=on zfsa/ix-apps
sudo zfs mount zfsa/ix-apps

At einem Punkt sah das sogar korrekt aus:

sudo zfs get mountpoint,canmount zfsa/ix-apps
NAME          PROPERTY    VALUE          SOURCE
zfsa/ix-apps  mountpoint  /mnt/.ix-apps  local
zfsa/ix-apps  canmount    on             local

mount | egrep 'ix-apps'
zfsa/ix-apps on /mnt/.ix-apps type zfs (rw,noatime,xattr,posixacl,casesensitive)

In diesem Zustand konnte der App‑Katalog im Web‑UI wieder geladen werden, aber Apps wie Immich starteten trotzdem nicht:

CallError: [EFAULT] Failed 'up' action for 'immich' app. 
Please check /var/log/app_lifecycle.log for more details

(I haven’t pasted the full app_lifecycle.log here to keep this post from exploding, but I can provide it if needed.)

Fix attempt 3: Stop all app-related services, deal with busy /mnt/.ix-apps/docker

To be safe while changing mountpoints, I stopped middlewared, Docker, and k3s and then checked who was still using /mnt/.ix-apps/docker:

sudo systemctl stop middlewared
sudo systemctl stop docker docker.socket containerd
sudo systemctl stop k3s

Despite that, dockerd was still holding files under /mnt/.ix-apps/docker:

sudo fuser -vm /mnt/.ix-apps/docker
                     USER        PID ACCESS COMMAND
/mnt/.ix-apps/docker:
                     root     kernel mount /mnt/.ix-apps/docker
                     root      10540 F...m dockerd

sudo lsof +f -- /mnt/.ix-apps/docker | head -n 10
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
dockerd 10540 root  mem-W  REG  0,101    32768 98311 /mnt/.ix-apps/docker/buildkit/history.db
dockerd 10540 root  mem-W  REG  0,101    32768 98614 /mnt/.ix-apps/docker/buildkit/cache.db
dockerd 10540 root  mem-W  REG  0,101    16384 98612 /mnt/.ix-apps/docker/buildkit/metadata_v2.db
...

I then killed dockerd and verified that it was gone:

sudo killall dockerd
sleep 2
ps aux | grep dockerd
admin      16484  0.0  0.0   3744  1604 pts/0    S+   03:09   0:00 grep dockerd

So no more dockerd running.

The stubborn part: ZFS still thinks zfsa/ix-apps is mounted at /mnt/mnt/.ix-apps

Despite stopping services and killing dockerd, ZFS itself insists that zfsa/ix-apps is mounted on /mnt/mnt/.ix-apps:

sudo zfs list -o name,mountpoint,mounted | grep ix-apps
raider/ix-apps                           /mnt/raider/ix-apps                  no
...
zfsa/ix-apps                             /mnt/mnt/.ix-apps                    yes
zfsa/ix-apps/app_configs                 /mnt/mnt/.ix-apps/app_configs        no
zfsa/ix-apps/app_mounts                  /mnt/mnt/.ix-apps/app_mounts         no
zfsa/ix-apps/docker                      /mnt/mnt/.ix-apps/docker             no
zfsa/ix-apps/truenas_catalog             /mnt/mnt/.ix-apps/truenas_catalog    no
zfsb/ix-apps                             /mnt/zfsb/ix-apps                    no
...

And /proc/self/mounts agrees:

cat /proc/self/mounts | egrep 'ix-apps'
zfsa/ix-apps /mnt/mnt/.ix-apps zfs rw,noatime,xattr,posixacl,casesensitive 0 0

Even when mount | egrep 'ix-apps' temporarily looked empty after umount -lf, ZFS still reported mounted=yes and /mnt/mnt/.ix-apps as the mountpoint. Any attempt to re‑mount with zfs mount fails:

sudo zfs mount zfsa/ix-apps
cannot mount 'zfsa/ix-apps': filesystem already mounted

Changing the property also doesn’t “stick” the way I expect:

sudo zfs inherit mountpoint zfsa/ix-apps
echo "RC inherit: $?"

sudo zfs set -u mountpoint=/.ix-apps zfsa/ix-apps
echo "RC set: $?"

sudo zfs get mountpoint zfsa zfsa/ix-apps
RC inherit: 0
RC set: 0
NAME          PROPERTY    VALUE          SOURCE
zfsa          mountpoint  /mnt/zfsa      default
zfsa/ix-apps  mountpoint  /mnt/.ix-apps  local

So zfs set -u mountpoint=/.ix-apps returns RC 0, but the value remains /mnt/.ix-apps, not /.ix-apps as suggested in other /mnt/mnt threads.

If I try to export the pool to reset the internal mount state:

sudo zpool export zfsa
cannot unmount '/mnt/.ix-apps/docker': pool or dataset is busy

Earlier that was due to dockerd holding files there; after killing dockerd and attempting unmounts by hand, ZFS still thinks zfsa/ix-apps is mounted at /mnt/mnt/.ix-apps, but the shell commands to unmount sometimes refuse with “does not exist” (likely because of the /mnt/mnt vs /mnt mismatch and previous attempts).

Current state

Right now:

  • zfsa root is cleanly at /mnt/zfsa.

  • ZFS believes zfsa/ix-apps is mounted at /mnt/mnt/.ix-apps:

    sudo zfs get mountpoint zfsa/ix-apps
NAME          PROPERTY    VALUE              SOURCE
zfsa/ix-apps  mountpoint  /mnt/mnt/.ix-apps  local

sudo zfs list -o name,mountpoint,mounted | grep ix-apps
zfsa/ix-apps                             /mnt/mnt/.ix-apps                    yes
...

  • /proc/self/mounts confirms the active mount at /mnt/mnt/.ix-apps.

  • Apps do not start; Immich throws Failed 'up' action and points to app_lifecycle.log.

  • The Apps catalog can load again when ix-apps was temporarily at /mnt/.ix-apps, but the system falls back to this inconsistent state and/or apps still fail.

What I’m looking for

Given all this:

  1. What is the supported way to completely reset the ix-apps mountpoint for a specific pool (zfsa) on SCALE 25.04.x so that:

    • ZFS has a consistent internal view (mounted=no when unmounted, then mounted at the correct path), and
    • TrueNAS re‑creates or re‑uses the Apps dataset at the canonical location (/mnt/.ix-apps), without breaking my user data on zfsa (zfsa/immich, zfsa/share, etc.)?

  2. In the known /mnt/mnt cases, the advice is usually “set mountpoint to /.ix-apps and import the pool with altroot=/mnt so it becomes /mnt/.ix-apps”.

    • In my case, zfs set -u mountpoint=/.ix-apps returns 0 but the value remains /mnt/.ix-apps. Is there a SCALE‑specific mechanism or middleware logic that overwrites this on import?

  3. Is there a safe, documented procedure to “factory‑reset” the Apps pool for one pool only (i.e. destroy/recreate only the ix-apps tree for zfsa, then let Apps → Settings → Choose Pool re‑initialize it), while preserving:

    • Other datasets on zfsa (immich, share, pCloudMirror), and
    • ix-apps trees on other pools (raider, zfsb)?

  4. If this is a known bug/regression in Electric Eel / 25.04.x, is there a public issue or recommended workaround I should follow?

Why I’m asking here

I’ve tried to carefully follow the usual ZFS steps (unmount, fix properties, export/import, kill processes that keep datasets busy) and the TrueNAS SCALE docs for Apps storage, but the ix-apps dataset on zfsa keeps ending up in this half‑broken /mnt/mnt/.ix-apps state according to ZFS’ own point of view.