Update ix-apps location

Tyrell_Midland · September 25, 2024, 1:41am

Hello, like a noob I didn’t realize I had a typo in my pool name until after it was in use. It used to be called “App Pool”, I followed the directions, disconnected the pool, renamed/reconnected, however, now it gets butthurt and threw the following error after a reboot

" Failed to configure docker for Applications: Missing “App Pool/ix-apps/docker, App Pool/ix-apps/app_mounts, App Pool/ix-apps/truenas_catalog, App Pool/ix-apps/app_configs” dataset(s) required for starting docker."

I double-checked via the shell and ran zfs list and it shows the correct new pool name “AppPool” in both the name and in the mountpoint.

NAME                                                                                          USED  AVAIL  REFER  MOUNTPOINT
AppPool                                                                                      46.5G   430G   120K  /mnt/AppPool
AppPool/.system                                                                              1.85G   430G  1.10G  /mnt/AppPool/.system
AppPool/.system/configs-ae32c386e13840b2bf9c0083275e7941                                     1.07M   430G   884K  /mnt/AppPool/.system/configs-ae32c386e13840b2bf9c0083275e7941
AppPool/.system/cores                                                                         208K  1024M    96K  /mnt/AppPool/.system/cores
AppPool/.system/netdata-ae32c386e13840b2bf9c0083275e7941                                      759M   430G   706M  /mnt/AppPool/.system/netdata-ae32c386e13840b2bf9c0083275e7941
AppPool/.system/nfs                                                                           248K   430G   104K  /mnt/AppPool/.system/nfs
AppPool/.system/samba4                                                                       1.39M   430G   324K  /mnt/AppPool/.system/samba4
AppPool/DockerCompose                                                                        18.5G   430G  18.5G  /mnt/AppPool/DockerCompose
AppPool/dockge                                                                               17.6G   430G   116K  /mnt/AppPool/dockge
AppPool/dockge/SWAGCERTS                                                                      452K   430G   356K  /mnt/AppPool/dockge/SWAGCERTS
AppPool/dockge/dockgedata                                                                     308K   430G   108K  /mnt/AppPool/dockge/dockgedata
AppPool/dockge/dockgestacks                                                                  17.6G   430G  17.0G  /mnt/AppPool/dockge/dockgestacks
AppPool/ix-applications                                                                      1.97G   430G   256K  /mnt/AppPool/ix-applications
AppPool/ix-applications/catalogs                                                             47.7M   430G  45.8M  /mnt/AppPool/ix-applications/catalogs
AppPool/ix-applications/default_volumes                                                       168K   430G    96K  /mnt/AppPool/ix-applications/default_volumes
AppPool/ix-applications/k3s                                                                  1.91G   430G  1.90G  /mnt/AppPool/ix-applications/k3s
AppPool/ix-applications/k3s/kubelet                                                           408K   430G   408K  legacy
AppPool/ix-applications/releases                                                             13.7M   430G   104K  /mnt/AppPool/ix-applications/releases
AppPool/ix-applications/releases/jellyfin                                                    13.5M   430G   104K  /mnt/AppPool/ix-applications/releases/jellyfin
AppPool/ix-applications/releases/jellyfin/charts                                              304K   430G   224K  /mnt/AppPool/ix-applications/releases/jellyfin/charts
AppPool/ix-applications/releases/jellyfin/volumes                                            13.0M   430G    96K  /mnt/AppPool/ix-applications/releases/jellyfin/volumes
AppPool/ix-applications/releases/jellyfin/volumes/ix_volumes                                 12.9M   430G   112K  /mnt/AppPool/ix-applications/releases/jellyfin/volumes/ix_volumes
AppPool/ix-applications/releases/jellyfin/volumes/ix_volumes/cache                           2.19M   430G  45.9M  /mnt/AppPool/ix-applications/releases/jellyfin/volumes/ix_volumes/cache
AppPool/ix-applications/releases/jellyfin/volumes/ix_volumes/config                          10.4M   430G  4.55G  /mnt/AppPool/ix-applications/releases/jellyfin/volumes/ix_volumes/config
AppPool/ix-applications/releases/jellyfin/volumes/ix_volumes/transcodes                        72K   430G    96K  /mnt/AppPool/ix-applications/releases/jellyfin/volumes/ix_volumes/transcodes
AppPool/ix-apps                                                                              6.52G   430G   112K  /mnt/.ix-apps
AppPool/ix-apps/app_configs                                                                   796K   430G    96K  /mnt/.ix-apps/app_configs
AppPool/ix-apps/app_mounts                                                                   4.66G   430G    96K  /mnt/.ix-apps/app_mounts
AppPool/ix-apps/app_mounts/jellyfin                                                          4.66G   430G   112K  /mnt/.ix-apps/app_mounts/jellyfin
AppPool/ix-apps/app_mounts/jellyfin/cache                                                    46.2M   430G  45.9M  /mnt/.ix-apps/app_mounts/jellyfin/cache
AppPool/ix-apps/app_mounts/jellyfin/config                                                   4.62G   430G  4.52G  /mnt/.ix-apps/app_mounts/jellyfin/config
AppPool/ix-apps/app_mounts/jellyfin/transcodes                                                 96K   430G    96K  /mnt/.ix-apps/app_mounts/jellyfin/transcodes
AppPool/ix-apps/docker                                                                       1.76G   430G   860M  /mnt/.ix-apps/docker
AppPool/ix-apps/truenas_catalog                                                               103M   430G  40.2M  /mnt/.ix-apps/truenas_catalog

When I try to choose pool in the Apps > App Setings I get the following error (TL:DR, multiple references to “App Pool”)

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 77, in umount
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 534, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 78, in umount
    dataset = zfs.get_dataset(name)
              ^^^^^^^^^^^^^^^^^^^^^
  File "libzfs.pyx", line 1463, in libzfs.ZFS.get_dataset
libzfs.ZFSException: Dataset App Pool/ix-apps not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 82, in umount
    raise CallError(f'Failed to umount dataset: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to umount dataset: Dataset App Pool/ix-apps not found
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/update.py", line 71, in do_update
    await self.middleware.call('service.stop', 'docker')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1603, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1447, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service.py", line 267, in stop
    await service_object.after_stop()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/docker.py", line 72, in after_stop
    await self.mount_umount_ix_apps(False)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/docker.py", line 19, in mount_umount_ix_apps
    await self.middleware.call('zfs.dataset.umount', docker_ds, {'force': True})
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1603, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1455, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1461, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1369, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1353, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed to umount dataset: Dataset App Pool/ix-apps not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 511, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/update.py", line 73, in do_update
    raise CallError(f'Failed to stop docker service: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to stop docker service: [EFAULT] Failed to umount dataset: Dataset App Pool/ix-apps not found

When I attempt to unset the pool via Apps > Settings, I get the following error

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 77, in umount
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 534, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 78, in umount
    dataset = zfs.get_dataset(name)
              ^^^^^^^^^^^^^^^^^^^^^
  File "libzfs.pyx", line 1463, in libzfs.ZFS.get_dataset
libzfs.ZFSException: Dataset App Pool/ix-apps not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 82, in umount
    raise CallError(f'Failed to umount dataset: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to umount dataset: Dataset App Pool/ix-apps not found
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/update.py", line 71, in do_update
    await self.middleware.call('service.stop', 'docker')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1603, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1447, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service.py", line 267, in stop
    await service_object.after_stop()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/docker.py", line 72, in after_stop
    await self.mount_umount_ix_apps(False)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/docker.py", line 19, in mount_umount_ix_apps
    await self.middleware.call('zfs.dataset.umount', docker_ds, {'force': True})
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1603, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1455, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1461, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1369, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1353, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed to umount dataset: Dataset App Pool/ix-apps not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 511, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/update.py", line 73, in do_update
    raise CallError(f'Failed to stop docker service: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to stop docker service: [EFAULT] Failed to umount dataset: Dataset App Pool/ix-apps not found

Is there an easy fix to this, or am I better off replicating that pool to my DataPool, blowing it up, recreating the pool and then moving the data back over?

Stux · September 25, 2024, 2:42am

I think there’s a fair case for a bug report here, after all, just because the app dataset can’t be found doesn’t mean that you shouldn’t be able to unset it…

Meanwhile, if you rename the pool back to what it was, then unset the app pool, then blow away the app pool’s app dataset (you’ll lose your ixvolumes etc), which you may need to do in the shell as the ix-apps data set is hidden, then you should be able to rename the pool, and then recreate the ix-apps dataset by re-selecting the pool.

Regarding the issue… it seems that the pool name is being stored in places perhaps it shouldn’t be, ie the app dataset is relative to the pool…

Tyrell_Midland · September 25, 2024, 11:59am

What I did to resolve this is unmounted the zpool in the UI, used the CLI to run

zpool import AppPool "App Pool"

It successfully imported the pool with the new name, but I also got the following error:

cannot mount '/App Pool': failed to create mountpoint: Read-only file system
Import was successful, but unable to mount some datasets

When I went into the UI, it was neither present or under the “storage” nor was it available for import in the GUI. As such, I then ran

zpool export "App Pool"

Then from the UI, I was able to import “App Pool”. When I went to the “Applications” section, there were no more errors. I was able to unset the “App Pool” from the applications settings. I then Chose my “DataPool” for apps, exported the “App Pool” from Storage section in the UI and ran the following command to rename:

zpool import "App Pool" AppPool

Predictably, I received the same error as before, and predictably, it was not eligible for import in the UI.

cannot mount '/AppPool': failed to create mountpoint: Read-only file system
Import was successful, but unable to mount some datasets

So I ran the export command

zpool export AppPool

Then I was able to import AppPool in the UI again. I was able to successfully use the “Choose Pool” option under Applications > Settings > Choose Pool to move the apps back tot he App Pool.

After reboot, the docker deamon automatically started, but the containers didn’t. Upon running “sudo docker compose up -d”, I got “context canceled” errors. When I tried to do a pull, I got

sudo docker compose pull
unable to get image 'certbot/dns-cloudflare:latest': Error response from daemon: layer does not exist

After a bit of guess & check, I restarted the docker service and it was able to successfully pull the images and start the containers. I rebooted the TrueNAS server again and everything worked as expected… which was strange, because I had rebooted the entire TrueNAS server, which should have also included a fresh start of the docker service.

I rebooted the server again and got the exact same behavior… again…

admin@truenas[~]$ cd /mnt/AppPool/DockerCompose
admin@truenas[/mnt/AppPool/DockerCompose]$ ls
SWAGCERTS  adguardhome1  adguardhome2  cloudflared.env  docker-compose.yaml  duckdns.env  homeassistant  jellyfin  watchtower.env
admin@truenas[/mnt/AppPool/DockerCompose]$ sudo docker ps
[sudo] password for admin:
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
admin@truenas[/mnt/AppPool/DockerCompose]$ sudo docker compose up -d
[+] Running 8/8
 ✘ homeassistant Error context canceled                                                                                                                                                                                                                                                 0.6s
 ✘ certbot Error       context canceled                                                                                                                                                                                                                                                 0.6s
 ✘ watchtower Error    context canceled                                                                                                                                                                                                                                                 0.6s
 ✘ adguardhome2 Error  context canceled                                                                                                                                                                                                                                                 0.6s
 ✘ jellyfin Error      context canceled                                                                                                                                                                                                                                                 0.6s
 ✘ cloudflare Error    context canceled                                                                                                                                                                                                                                                 0.6s
 ✘ duckdns Error       context canceled                                                                                                                                                                                                                                                 0.6s
 ✔ adguardhome1 Pulled                                                                                                                                                                                                                                                                  0.6s
Error response from daemon: layer does not exist
admin@truenas[/mnt/AppPool/DockerCompose]$ sudo systemctl restart docker
admin@truenas[/mnt/AppPool/DockerCompose]$ sudo docker compose up -d
[+] Running 6/6
 ✔ watchtower Pulled                                                                                                                                                                                                                                                                    0.6s
 ✔ homeassistant Pulled                                                                                                                                                                                                                                                                 0.5s
 ✔ cloudflare Pulled                                                                                                                                                                                                                                                                    0.5s
 ✔ duckdns Pulled                                                                                                                                                                                                                                                                       1.2s
 ✔ jellyfin Pulled                                                                                                                                                                                                                                                                      1.2s
 ✔ certbot Pulled                                                                                                                                                                                                                                                                       0.6s
[+] Running 8/8
 ✔ Container duckdns        Running                                                                                                                                                                                                                                                     0.0s
 ✔ Container certbot        Started                                                                                                                                                                                                                                                     0.7s
 ✔ Container watchtower     Running                                                                                                                                                                                                                                                     0.0s
 ✔ Container adguardhome2   Running                                                                                                                                                                                                                                                     0.0s
 ✔ Container homeassistant  Running                                                                                                                                                                                                                                                     0.0s
 ✔ Container cloudflared    Running                                                                                                                                                                                                                                                     0.0s
 ✔ Container jellyfin       Running                                                                                                                                                                                                                                                     0.0s
 ✔ Container adguardhome1   Running

I checked out the docker.service file and everything but 1 line was commented out. Google-Fu suggests this may be an issue of the docker daemon service starting before the network is initialized, which is why restarting the service after it’s fully booted is resolving the issue.

I uncommented out the following lines in the docker.service file:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service containerd.service time-set.target
Wants=network-online.target containerd.service
Requires=docker.socket

After reboot, I experienced the exact same issues again.

admin@truenas[~]$ sudo su
[sudo] password for admin: 
root@truenas[/home/admin]# cd /mnt/AppPool/DockerCompose 
root@truenas[/mnt/AppPool/DockerCompose]# sudo docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
root@truenas[/mnt/AppPool/DockerCompose]# docker compose up -d
[+] Running 5/8
 ✔ adguardhome1 Pulled                                                                                                                                                                                                                                                                                                                                                                            0.6s 
 ⠴ watchtower Pulling                                                                                                                                                                                                                                                                                                                                                                             0.6s 
 ✘ certbot Error        context canceled                                                                                                                                                                                                                                                                                                                                                          0.6s 
 ✘ duckdns Error        context canceled                                                                                                                                                                                                                                                                                                                                                          0.6s 
 ⠴ adguardhome2 Pulling                                                                                                                                                                                                                                                                                                                                                                           0.6s 
 ✘ jellyfin Error       context canceled                                                                                                                                                                                                                                                                                                                                                          0.6s 
 ✘ homeassistant Error  context canceled                                                                                                                                                                                                                                                                                                                                                          0.6s 
 ⠴ cloudflare Pulling                                                                                                                                                                                                                                                                                                                                                                             0.6s 
Error response from daemon: layer does not exist

root@truenas[/mnt/AppPool/DockerCompose]# docker compose up -d
[+] Running 4/4
 ✔ homeassistant Pulled                                                                                                                                                                                                                                                                                                                                                                           0.6s 
 ✔ jellyfin Pulled                                                                                                                                                                                                                                                                                                                                                                                1.2s 
 ✔ duckdns Pulled                                                                                                                                                                                                                                                                                                                                                                                 1.2s 
 ✔ certbot Pulled                                                                                                                                                                                                                                                                                                                                                                                 0.6s 
[+] Running 8/8
 ✔ Container adguardhome1   Running                                                                                                                                                                                                                                                                                                                                                               0.0s 
 ✔ Container jellyfin       Running                                                                                                                                                                                                                                                                                                                                                               0.0s 
 ✔ Container certbot        Started                                                                                                                                                                                                                                                                                                                                                               0.8s 
 ✔ Container duckdns        Running                                                                                                                                                                                                                                                                                                                                                               0.0s 
 ✔ Container homeassistant  Running                                                                                                                                                                                                                                                                                                                                                               0.0s 
 ✔ Container adguardhome2   Running                                                                                                                                                                                                                                                                                                                                                               0.0s 
 ✔ Container watchtower     Running                                                                                                                                                                                                                                                                                                                                                               0.0s 
 ✔ Container cloudflared    Running

I checked the service.docker file again and all of those lines were commented out again, which tells me that likely TrueNAS is overriding that file somewhere else… but I’m wading pretty far into “WTF are you doing?! This is completely unsupported” territory. Since I’m unable to use apt-get to uninstall/reinstall the docker app/service, I’m kind of at a loss as to where to start now

I’m not a fan of Bandaid solutions, so doing a cron job that will run the following seems undesirable.

sudo systemctl restart docker
sudo docker compose -f /mnt/AppPool/DockerCompose/docker-compose.yaml up -d

Should I just backup my configuration, blow up my boot pool, reinstall TrueNAS, restore my configuration and profit?

Tyrell_Midland · September 25, 2024, 12:32pm

Well… FML. I got a wild idea and I ran

sudo systemctl enable docker.service
sudo systemctl enable containerd.service

Rebooted, experienced the same issues, and upon undoing that change by running

sudo systemctl disable docker.service
sudo systemctl disable containerd.service

and rebooting, seems to have resolved the issue… 3 reboots in a row and the docker service is starting properly, the containers are auto-starting, etc. I don’t know what caused the issue, but trying to force the auto-start, then disabling the auto-start resolved it. It’s unclear why… but it works. Which is both a relief and terrifying because not knowing why it’s fixed means I also don’t know why it was broken