I tried updating from 24.10.0.2 to 24.10.1 this morning. The installation appeared to go smoothly, but after the reboot all the Apps were gone. The Apps tab showed a green check mark with “Application Service Running,” but it looked like no apps were installed.
I ran docker ps which also showed nothing running and docker volumes ls showed no volumes. docker network ls only showed the three default docker networks (bridge, host and null), but none of the named networks. It looks like the 24.10.1 update reset docker settings back to factory defaults for some reason.
I tried rebooting because sometimes that fixes things, but my Apps were still missing. So, I set the boot environment to 24.10.0.2 and rebooted again, and all the Apps came back.
I don’t know what caused the issue, but thought I would post a heads-up for any other early adopters of this update.
Hi @JoelThomas. Thanks for submitting a bug report with your debug. Looks like this one is hopefully a pretty simple problem stemming from your app data being on a root-encrypted pool.
Your logs are full of tracebacks like:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/data-pool/nextcloud/config', TimeoutError: [Errno 110] Connection timed out, middlewared.service_exception.CallError: [EDATASETISLOCKED] Please unlock following dataset(s) before starting docker: data-pool
which indicates that the pool was not unlocked before docker attempted to start the containers.
My guess is that if you unlocked the dataset, or waited until it was automatically unlocked by key, then restarted the docker service, your apps should come up as expected.
Strange. That pool in the message you quoted (“data-pool”) was exported and destroyed, and those disks are no longer installed. That pool will never be unlocked. The current data pool has different disks and a different name (“data-tank”). I wonder what is causing the old one to be mounted (unsuccessfully, obviously).
Just to confirm, are you looking at error messages from today, December 18, 2024, the day I applied the 24.10.1 update? Docker should not be trying to use “data-pool” for any reason today, although messages like that would make sense in the past.
On closer examination, I see that my pool is mounted under /mnt/mnt/data-tank instead of /mnt/data-tank, and there are new docker and .ix-apps directories in /mnt. Maybe I did something when I was cleaning up the mess left by the old faulted pool that caused the new pool to be mounted in the wrong place… I can see that not being able to find its pool would cause the behavior I saw with docker.
You’re right I was looking at a summary without timestamps. The most recent ones I see in the middleware log are about a week ago.
That would definitely do it.
So to be clear, you’re saying that your whole pool is mounted a level deeper? /mnt/ contains a second /mnt/ with data-tank underneath it? What does the output of zfs get -r mountpoint data-tank indicate?
Yes, my whole pool was mounted a level deeper. Instead of mounting on /mnt as expected, it mounted on /mnt/mnt, and some other docker-related stuff got mounted on /mnt unexpectedly.
I think I mostly fixed the mount by using zfs set mountpoint and then exporting the pool before reimporting the pool through the GUI. So, now zfs get mountpoint data-tank returns the following which I think looks normal:
NAME PROPERTY VALUE SOURCE
data-tank mountpoint /mnt/data-tank local
I suppose that the docker data that seems to be missing is in these ix-apps mounts shown in zfs list. I’m not sure if they are mounted where they’re supposed to be or not. The app_configs and app_mounts folders are empty, but docker and truenas_catalog have a lot of stuff inside. In any case, my Apps screen still shows “No applications installed.”
I was mostly looking for if you had accidentally created a duplicate and ended up with /mnt/data-tank and also /mnt/mnt/data-tank, but it looks like you’ve progressed beyond that point.
What does zfs get mountpoint data-tank/ix-apps return now? It should be /mnt/.ix-apps.
That looks promising. Now we just need to make sure it’s the right ix-apps. Does ls /mnt/.ix-apps/app_configs show the apps you expect to see installed?
We may still be looking at the duplicate created when you upgraded with the mismounted pool. Hard to tell from here.
I’m grasping at straws here a bit, but…
Is this “stuff” still there?
Is there anything unexpected remaining at /mnt/ or /mnt/?
Do ls -a /mnt/data-tank/.. or ls -a /mnt/mnt/data-tank/.. possibly show a duplicate ix-apps dataset?
No, the extra mount point is still there, but it looks like there’s nothing in it.
$ ls -alR /mnt/mnt
/mnt/mnt:
total 2
drwxr-xr-x 3 root root 3 Dec 18 11:46 .
drwxr-xr-x 6 root root 6 Dec 18 11:48 ..
drwxr-xr-x 2 root root 2 Dec 18 11:46 data-tank
/mnt/mnt/data-tank:
total 1
drwxr-xr-x 2 root root 2 Dec 18 11:46 .
drwxr-xr-x 3 root root 3 Dec 18 11:46 ..
$ ls -al /mnt/data-tank/..
total 57
drwxr-xr-x 6 root root 6 Dec 18 11:48 .
drwxr-xr-x 21 root root 29 Dec 16 14:58 ..
drwxr-xr-x 6 root root 6 Dec 18 07:35 .ix-apps
drwxr-xr-x 9 root root 8 Dec 18 14:21 data-tank
drwxr-xr-x 3 root root 3 Dec 18 11:46 mnt
$ ls -al /mnt/mnt/data-tank/..
total 2
drwxr-xr-x 3 root root 3 Dec 18 11:46 .
drwxr-xr-x 6 root root 6 Dec 18 11:48 ..
drwxr-xr-x 2 root root 2 Dec 18 11:46 data-tank
$ ls -al /mnt/data-tank/../.ix-apps/app_configs/
total 12
drwxr-xr-x 2 root root 2 Dec 13 13:44 .
drwxr-xr-x 6 root root 6 Dec 18 07:35 ..
I thought maybe I could pull any files that might’ve been in .ix-apps from snapshots, but either I don’t know how to do it, or there’s nothing in there.
# ls -al /mnt/.ix-apps/.zfs/snapshot/
total 1
drwxrwxrwx 16 root root 2 Dec 18 15:00 .
drwxrwxrwx 1 root root 0 Dec 18 13:30 ..
drwxrwxrwx 1 root root 0 Dec 15 00:00 Weekly-2024-12-15_00-00
drwxrwxrwx 1 root root 0 Dec 14 00:00 daily-2024-12-14_00-00
drwxrwxrwx 1 root root 0 Dec 15 00:00 daily-2024-12-15_00-00
drwxrwxrwx 1 root root 0 Dec 16 00:00 daily-2024-12-16_00-00
drwxrwxrwx 1 root root 0 Dec 17 00:00 daily-2024-12-17_00-00
drwxr-xr-x 2 root root 2 Dec 13 13:44 daily-2024-12-18_00-00
drwxrwxrwx 1 root root 0 Dec 17 00:00 hourly-2024-12-17_00-00
drwxrwxrwx 1 root root 0 Dec 18 00:00 hourly-2024-12-18_00-00
drwxrwxrwx 1 root root 0 Dec 18 08:00 hourly-2024-12-18_08-00
drwxrwxrwx 1 root root 0 Dec 18 09:00 hourly-2024-12-18_09-00
drwxrwxrwx 1 root root 0 Dec 18 10:00 hourly-2024-12-18_10-00
drwxrwxrwx 1 root root 0 Dec 18 11:00 hourly-2024-12-18_11-00
drwxrwxrwx 1 root root 0 Dec 18 12:00 hourly-2024-12-18_12-00
drwxrwxrwx 1 root root 0 Dec 18 13:56 manual-2024-12-18_13-55
# ls -al /mnt/.ix-apps/.zfs/snapshot/daily-2024-12-18_00-00
total 1
drwxr-xr-x 2 root root 2 Dec 13 13:44 .
drwxrwxrwx 16 root root 2 Dec 18 15:00 ..