Docker settings reset to defaults by 24.10.1 update

I tried updating from 24.10.0.2 to 24.10.1 this morning. The installation appeared to go smoothly, but after the reboot all the Apps were gone. The Apps tab showed a green check mark with “Application Service Running,” but it looked like no apps were installed.

I ran docker ps which also showed nothing running and docker volumes ls showed no volumes. docker network ls only showed the three default docker networks (bridge, host and null), but none of the named networks. It looks like the 24.10.1 update reset docker settings back to factory defaults for some reason.

I tried rebooting because sometimes that fixes things, but my Apps were still missing. So, I set the boot environment to 24.10.0.2 and rebooted again, and all the Apps came back.

I don’t know what caused the issue, but thought I would post a heads-up for any other early adopters of this update.

1 Like

Hi @JoelThomas. Thanks for submitting a bug report with your debug. Looks like this one is hopefully a pretty simple problem stemming from your app data being on a root-encrypted pool.

Your logs are full of tracebacks like:

FileNotFoundError: [Errno 2] No such file or directory: '/mnt/data-pool/nextcloud/config', TimeoutError: [Errno 110] Connection timed out, middlewared.service_exception.CallError: [EDATASETISLOCKED] Please unlock following dataset(s) before starting docker: data-pool

which indicates that the pool was not unlocked before docker attempted to start the containers.

My guess is that if you unlocked the dataset, or waited until it was automatically unlocked by key, then restarted the docker service, your apps should come up as expected.

Strange. That pool in the message you quoted (“data-pool”) was exported and destroyed, and those disks are no longer installed. That pool will never be unlocked. The current data pool has different disks and a different name (“data-tank”). I wonder what is causing the old one to be mounted (unsuccessfully, obviously).

Just to confirm, are you looking at error messages from today, December 18, 2024, the day I applied the 24.10.1 update? Docker should not be trying to use “data-pool” for any reason today, although messages like that would make sense in the past.

On closer examination, I see that my pool is mounted under /mnt/mnt/data-tank instead of /mnt/data-tank, and there are new docker and .ix-apps directories in /mnt. Maybe I did something when I was cleaning up the mess left by the old faulted pool that caused the new pool to be mounted in the wrong place… I can see that not being able to find its pool would cause the behavior I saw with docker.

You’re right I was looking at a summary without timestamps. The most recent ones I see in the middleware log are about a week ago.

That would definitely do it.

So to be clear, you’re saying that your whole pool is mounted a level deeper? /mnt/ contains a second /mnt/ with data-tank underneath it? What does the output of zfs get -r mountpoint data-tank indicate?

Yes, my whole pool was mounted a level deeper. Instead of mounting on /mnt as expected, it mounted on /mnt/mnt, and some other docker-related stuff got mounted on /mnt unexpectedly.

I think I mostly fixed the mount by using zfs set mountpoint and then exporting the pool before reimporting the pool through the GUI. So, now zfs get mountpoint data-tank returns the following which I think looks normal:

NAME       PROPERTY    VALUE           SOURCE
data-tank  mountpoint  /mnt/data-tank  local

I suppose that the docker data that seems to be missing is in these ix-apps mounts shown in zfs list. I’m not sure if they are mounted where they’re supposed to be or not. The app_configs and app_mounts folders are empty, but docker and truenas_catalog have a lot of stuff inside. In any case, my Apps screen still shows “No applications installed.”

data-tank/ix-apps                                                       236M  2.73T   139K  /mnt/.ix-apps
data-tank/ix-apps/app_configs                                           128K  2.73T   128K  /mnt/.ix-apps/app_configs
data-tank/ix-apps/app_mounts                                            128K  2.73T   128K  /mnt/.ix-apps/app_mounts
data-tank/ix-apps/docker                                                778K  2.73T   229K  /mnt/.ix-apps/docker
data-tank/ix-apps/truenas_catalog                                       234M  2.73T   120M  /mnt/.ix-apps/truenas_catalog

Your command, zfs get -r mountpoint data-tank returns a thousand lines of output that starts like this:

NAME                                                                  PROPERTY    VALUE                                              SOURCE
data-tank                                                             mountpoint  /mnt/data-tank                                     local
data-tank@daily-2024-12-13_00-00                                      mountpoint  -                                                  -
data-tank@daily-2024-12-14_00-00                                      mountpoint  -                                                  -
data-tank@Weekly-2024-12-15_00-00                                     mountpoint  -                                                  -
data-tank@daily-2024-12-15_00-00                                      mountpoint  -                                                  -
data-tank@daily-2024-12-16_00-00                                      mountpoint  -                                                  -

I guess you don’t want to see all of it, but I might be missing something.

I was mostly looking for if you had accidentally created a duplicate and ended up with /mnt/data-tank and also /mnt/mnt/data-tank, but it looks like you’ve progressed beyond that point.

What does zfs get mountpoint data-tank/ix-apps return now? It should be /mnt/.ix-apps.

Yes, that’s what it returns.

root@babka:/mnt#  zfs get mountpoint data-tank/ix-apps
NAME               PROPERTY    VALUE          SOURCE
data-tank/ix-apps  mountpoint  /mnt/.ix-apps  local

That looks promising. Now we just need to make sure it’s the right ix-apps. Does ls /mnt/.ix-apps/app_configs show the apps you expect to see installed?

No, the app_configs and app_mounts directories are empty:

$ cd /mnt/.ix-apps/
$ ls -al
total 35
drwxr-xr-x  6 root root  6 Dec 18 07:35 .
drwxr-xr-x  6 root root  6 Dec 18 11:48 ..
drwxr-xr-x  2 root root  2 Dec 13 13:44 app_configs
drwxr-xr-x  2 root root  2 Dec 13 13:44 app_mounts
drwx--x--- 12 root root 13 Dec 18 13:30 docker
drwxr-xr-x  8 root root 16 Dec 18 13:30 truenas_catalog
$ ls -al app_configs/
total 12
drwxr-xr-x 2 root root 2 Dec 13 13:44 .
drwxr-xr-x 6 root root 6 Dec 18 07:35 ..
$ ls -al app_mounts/
total 12
drwxr-xr-x 2 root root 2 Dec 13 13:44 .
drwxr-xr-x 6 root root 6 Dec 18 07:35 ..

Less promising.

We may still be looking at the duplicate created when you upgraded with the mismounted pool. Hard to tell from here.

I’m grasping at straws here a bit, but…

Is this “stuff” still there?
Is there anything unexpected remaining at /mnt/ or /mnt/?
Do ls -a /mnt/data-tank/.. or ls -a /mnt/mnt/data-tank/.. possibly show a duplicate ix-apps dataset?

No, the extra mount point is still there, but it looks like there’s nothing in it.

$ ls -alR /mnt/mnt
/mnt/mnt:
total 2
drwxr-xr-x 3 root root 3 Dec 18 11:46 .
drwxr-xr-x 6 root root 6 Dec 18 11:48 ..
drwxr-xr-x 2 root root 2 Dec 18 11:46 data-tank

/mnt/mnt/data-tank:
total 1
drwxr-xr-x 2 root root 2 Dec 18 11:46 .
drwxr-xr-x 3 root root 3 Dec 18 11:46 ..
$ ls -al /mnt/data-tank/..
total 57
drwxr-xr-x  6 root root     6 Dec 18 11:48 .
drwxr-xr-x 21 root root    29 Dec 16 14:58 ..
drwxr-xr-x  6 root root     6 Dec 18 07:35 .ix-apps
drwxr-xr-x  9 root root     8 Dec 18 14:21 data-tank
drwxr-xr-x  3 root root     3 Dec 18 11:46 mnt
$ ls -al /mnt/mnt/data-tank/..
total 2
drwxr-xr-x 3 root root 3 Dec 18 11:46 .
drwxr-xr-x 6 root root 6 Dec 18 11:48 ..
drwxr-xr-x 2 root root 2 Dec 18 11:46 data-tank
$ ls -al /mnt/data-tank/../.ix-apps/app_configs/
total 12
drwxr-xr-x 2 root root 2 Dec 13 13:44 .
drwxr-xr-x 6 root root 6 Dec 18 07:35 ..

I thought maybe I could pull any files that might’ve been in .ix-apps from snapshots, but either I don’t know how to do it, or there’s nothing in there.

# ls -al /mnt/.ix-apps/.zfs/snapshot/
total 1
drwxrwxrwx 16 root root 2 Dec 18 15:00 .
drwxrwxrwx  1 root root 0 Dec 18 13:30 ..
drwxrwxrwx  1 root root 0 Dec 15 00:00 Weekly-2024-12-15_00-00
drwxrwxrwx  1 root root 0 Dec 14 00:00 daily-2024-12-14_00-00
drwxrwxrwx  1 root root 0 Dec 15 00:00 daily-2024-12-15_00-00
drwxrwxrwx  1 root root 0 Dec 16 00:00 daily-2024-12-16_00-00
drwxrwxrwx  1 root root 0 Dec 17 00:00 daily-2024-12-17_00-00
drwxr-xr-x  2 root root 2 Dec 13 13:44 daily-2024-12-18_00-00
drwxrwxrwx  1 root root 0 Dec 17 00:00 hourly-2024-12-17_00-00
drwxrwxrwx  1 root root 0 Dec 18 00:00 hourly-2024-12-18_00-00
drwxrwxrwx  1 root root 0 Dec 18 08:00 hourly-2024-12-18_08-00
drwxrwxrwx  1 root root 0 Dec 18 09:00 hourly-2024-12-18_09-00
drwxrwxrwx  1 root root 0 Dec 18 10:00 hourly-2024-12-18_10-00
drwxrwxrwx  1 root root 0 Dec 18 11:00 hourly-2024-12-18_11-00
drwxrwxrwx  1 root root 0 Dec 18 12:00 hourly-2024-12-18_12-00
drwxrwxrwx  1 root root 0 Dec 18 13:56 manual-2024-12-18_13-55
# ls -al /mnt/.ix-apps/.zfs/snapshot/daily-2024-12-18_00-00
total 1
drwxr-xr-x  2 root root 2 Dec 13 13:44 .
drwxrwxrwx 16 root root 2 Dec 18 15:00 ..

Sometimes the ix-apps dataset doesn’t mount right.

When this happens, then stuff gets written to /mnt/.ix-apps on the boot disk.

You can try umount /mnt/.ix-apps and then zfs mount data-tank/ix-apps to remount.

Once mounted it can be worth disabling and re-enabling the apps in the GUI.