TrueNAS Scale Apps Disappeared after Disk Full / Failed App update

Apps and Virtualization
1

I’ve hit a weird situation where all of my installed apps have disappeared from the UI and doing app.query via SSH doesn’t show anything. This began when a custom alpine based app I’d spun up to do a wget crawling operation had crashed after apparently entirely filling the data pool for apps with its temp files/logs. Before trying to diagnose that, I’d noticed that Immich had also crashed, and was listed with updates available, so I tried to run an update on that. At that point the Apps screen in the UI reloaded and said “No Apps Installed”.

That’s when I noticed the pool was entirely full, and dug around via the shell to find that the Alpine/wget custom app had consumed most of it. I used docker rm to remove that app via the shell, and the space was freed up.

Despite many more reboots and trying to diagnose with help from Claude, nothing seems to have changed. Claude ran out of suggestions, and summarised the diagnostic steps run through so far:

  1. TrueNAS SCALE, apps pool AppsDrive, dataset AppsDrive/ix-apps correctly mounted at /mnt/.ix-apps
  2. Pool previously filled to 100% by an unrelated runaway container (wget-scraper, unrelated to the apps system) — since removed, space reclaimed
  3. docker.status reports RUNNING, docker ps shows all expected containers running correctly, actual app data (app_configs, app_mounts) confirmed present and intact with sensible sizes
  4. midclt call app.query returns [] regardless of reboots, docker.update pool re-selection, or isolating individual apps (including one — Immich — that was mid-update when the disk filled)
  5. Top-level /mnt/.ix-apps/metadata.yaml is 0 bytes, but removing/renaming it does not trigger regeneration, and app.query remains empty even without it present — indicating it is not the file app.query actually reads from
  6. Per-app metadata.yaml files (checked: jellyfin) and top-level user_config.yaml all look complete and correctly structured

Any suggestions to either retrieve the current apps, or at least extract their config to rebuild them?

Running TrueNAS Scale 25.04 on a custom PC built from leftovers, with a i5-7500, 20Gb RAM, 2 x 12Tb Seagate iron wolf data drives, a 500Gb spinning drive for apps, and a 128Gb M2 boot drive.

2

Did you try unsetting & re-setting the app pool? I think I’ve also seen someone having a similar fault try installing & then removing a random app to retrigger their apps being discovered.

Any chance you have snapshots of the apps pool from before you installed your custom app you can roll back to?

Outside of these basic steps available from the GUI, I’m out of ideas.

3

Thank you so much! Installing a random app did the trick! I can’t believe I spent so long trying other diagnostics when something like that did the trick. The Immich app was still broken, and initially it wouldn’t let me roll it back, because it wasn’t in stop state, despite the UI showing it stopped. Somehow connecting to SSH seemed to kick it into action and it successfully re-deployed.

1 Like