Apps: Validation Error / Datasets: Failed to load datasets - 500 Server Error

ProMG · September 22, 2026, 7:09pm

On TrueNAS 25.04.2.6

Today, I updated Nextcloud-App and nginx to the latest versions. Everything looked fine.
I did not do any update for TrueNAS.

Then, I attached my external HDD via USB, imported the pool, started a replication task onto this pool. This took quite a while and was successfully finished.

When I logged in for the TrueNAS-UI to export the pool, I faced several error messages:

Opening the “Apps” leads to an “Validation Error”:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/docker/api/client.py”, line 275, in _raise_for_status
response.raise_for_status()
File “/usr/lib/python3/dist-packages/requests/models.py”, line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.47/containers/1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py”, line 323, in process_method_call
result = await method.call(app, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py”, line 52, in call
result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 911, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 731, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 624, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py”, line 101, in wrapped
result = func(*args)
^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py”, line 72, in query
apps = list_apps(available_apps_mapping, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/query.py”, line 83, in list_apps
for app_name, app_resources in list_resources_by_project(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/docker/query.py”, line 13, in list_resources_by_project
return list_resources_by_project_internal(project_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/docker/query.py”, line 23, in list_resources_by_project_internal
containers = client.containers.list(all=True, filters=label_filter, sparse=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/models/containers.py”, line 1018, in list
containers.append(self.get(r[‘Id’]))
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/models/containers.py”, line 954, in get
resp = self.client.api.inspect_container(container_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/utils/decorators.py”, line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/api/container.py”, line 793, in inspect_container
return self._result(
^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/api/client.py”, line 281, in _result
self._raise_for_status(response)
File “/usr/lib/python3/dist-packages/docker/api/client.py”, line 277, in _raise_for_status
raise create_api_error_from_http_exception(e) from e
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/errors.py”, line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.47/containers/1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6/json: Internal Server Error (“stat /mnt/.ix-apps/docker/overlay2/2a18a5b79fdbda3aec58207273fd3e630a1f1f3d0797cb69b2d8b8511f1ddc7d: no such file or directory”)

Same message appears three times and needs to be confirmed by the “close” button. The list of installed apps is then shown empty!

Clicking on “Datasets” leads to the error “Failed to load datasets”:

500 Server Error for http+docker://localhost/v1.47/containers/1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6/json: Internal Server Error (“stat /mnt/.ix-apps/docker/overlay2/2a18a5b79fdbda3aec58207273fd3e630a1f1f3d0797cb69b2d8b8511f1ddc7d: no such file or directory”)

Hitting the “Retry” button keeps the same message coming up again and again.

Clicking the “Export/Disconnect” button for the external HDD drive in the “Storage” section ends up in a “Validation Error”

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/docker/api/client.py”, line 275, in _raise_for_status
response.raise_for_status()
File “/usr/lib/python3/dist-packages/requests/models.py”, line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.47/containers/1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py”, line 323, in process_method_call
result = await method.call(app, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py”, line 52, in call
result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 911, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 720, in call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 174, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 48, in nf
res = await f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool/info.py", line 55, in attachments
return await self.middleware.call(‘pool.dataset.attachments_with_path’, pool[‘path’])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1005, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 720, in call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool/dataset_attachments.py", line 48, in attachments_with_path
for attachment in await delegate.query(path, True, options):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/fs_attachments.py”, line 10, in query
for app in await self.middleware.call(‘app.query’):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1005, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 731, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 624, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py”, line 101, in wrapped
result = func(*args)
^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py”, line 72, in query
apps = list_apps(available_apps_mapping, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/query.py”, line 83, in list_apps
for app_name, app_resources in list_resources_by_project(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/docker/query.py”, line 13, in list_resources_by_project
return list_resources_by_project_internal(project_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/docker/query.py”, line 23, in list_resources_by_project_internal
containers = client.containers.list(all=True, filters=label_filter, sparse=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/models/containers.py”, line 1018, in list
containers.append(self.get(r[‘Id’]))
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/models/containers.py”, line 954, in get
resp = self.client.api.inspect_container(container_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/utils/decorators.py”, line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/api/container.py”, line 793, in inspect_container
return self._result(
^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/api/client.py”, line 281, in _result
self._raise_for_status(response)
File “/usr/lib/python3/dist-packages/docker/api/client.py”, line 277, in _raise_for_status
raise create_api_error_from_http_exception(e) from e
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/docker/errors.py”, line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.47/containers/1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6/json: Internal Server Error (“stat /mnt/.ix-apps/docker/overlay2/2a18a5b79fdbda3aec58207273fd3e630a1f1f3d0797cb69b2d8b8511f1ddc7d: no such file or directory”)

No alerts are present.

I did not find any helpful hint about what might have gone wrong here.

Who has an idea what I could do to repair my TrueNAS installation?

Thanks a lot in advance!

nite_route · September 22, 2026, 8:27pm

that traceback is docker inspecting a container whose overlay2 layer is gone under /mnt/.ix-apps/docker. apps and datasets both call app.query, so one broken container blocks the ui.

start with:
sudo docker ps -a
sudo ls /mnt/.ix-apps/docker/overlay2 | head
journalctl -u docker -b --no-pager | tail -80

if that container id 1583d9ca… is a ghost, remove it:
sudo docker rm -f 1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6

then restart docker and see if Apps loads again. dont export pools until that 500 stops. if overlay2 is still half missing after the usb pool work, stop apps, clean the orphaned container, start apps back up.

ProMG · September 23, 2026, 8:06pm

The container seems not to be a ghost. It seems to belong to the nextcloud cron job:

truenas_admin@truenas[~]$ sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1583d9ca802c 3c99f1e59422 “/bin/sh -c 'echo '*…” 28 hours ago Up 28 hours (unhealthy) 80/tcp ix-nextcloud-cron-1
abaac53ab148 3c99f1e59422 “/entrypoint.sh apac…” 28 hours ago Up 28 hours (unhealthy) 0.0.0.0:30027->80/tcp, [::]:30027->80/tcp ix-nextcloud-nextcloud-1
60f629dadc21 248efd5e58cd “docker-entrypoint.s…” 28 hours ago Up 28 hours (unhealthy) 5432/tcp ix-nextcloud-postgres-1
6e92a7ce6728 c2d199f6085c “/bin/bash -c /upgra…” 28 hours ago Exited (0) 28 hours ago ix-nextcloud-postgres_upgrade-1
225436049f88 726dea3590a8 “tini – docker-entr…” 28 hours ago Up 28 hours (unhealthy) 6379/tcp ix-nextcloud-redis-1
c2e0ffba2deb cae7b55f7b26 “python3 /script/per…” 28 hours ago Exited (0) 28 hours ago ix-nextcloud-permissions-1
a7741ef1d9f1 f44e23f5e4d7 “/init” 29 hours ago Up 29 hours (unhealthy) 0.0.0.0:30021->80/tcp, [::]:30021->80/tcp, 0.0.0.0:30020->81/tcp, [::]:30020->81/tcp, 0.0.0.0:30022->443/tcp, [::]:30022->443/tcp ix-nginx-proxy-manager-npm-1

Overlay2 layers look like this:

truenas_admin@truenas[~]$ sudo ls /mnt/.ix-apps/docker/overlay2 | head
00c9fc4b5463a7e816c6015157089b3623ad9e65f0794077c166ae56c98f24a0
026192a88175ee4d1787b8a13a35f5ae4b42f03934e086b0ddeb90e9619b783e
0275109b0f8e11dc294efde4d44127762dc75b085cbde16732e0153f0492522f
03bbee3db83b08357d11c355f21428737828c04da94f8fa9719b5a9237cb825c
043c974e3788e41c1df74206afab212955e47091feb401224be92965c9433889
0440bb864952a18d530bc122f6ed88160f99d96c6c6db0c706a2417647cb05fb
049f5ce31c98a61a81126cab1ef7efb40c23e1fd945c4ede94e0e28e88ccb089
04cf32ab80e1e8d32f8f2f51618b7cf668cf83c907ee59efe1aafc0f7a372e89
04f2ffbcff47caa5bfd8934ccd04570aee3e0022d2caf337ee84e8a62408a695
04f6d1398f9c2553fa066aac9fdd4d67e66c027bbbbb017486d5519c244ed0ff

Docker journal looks like this:

truenas_admin@truenas[~]$ sudo journalctl -u docker -b --no-pager | tail -80
Sep 23 21:27:24 truenas dockerd[6349]: time=“2026-09-23T21:27:24.666087674+02:00” level=warning msg=“Health check for container 1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:27:24 truenas dockerd[6349]: time=“2026-09-23T21:27:24.666196435+02:00” level=warning msg=“Health check for container 225436049f88482635e42a39f194f1f5dc8a99c3930e5202685d7c294e316e8f error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:27:24 truenas dockerd[6349]: time=“2026-09-23T21:27:24.666782175+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:24 truenas dockerd[6349]: time=“2026-09-23T21:27:24.667816113+02:00” level=warning msg=“Health check for container abaac53ab148f1e1c99b59bf82f5a0f86ed0ece95e5cd23c79e35a918390fdf1 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:27:24 truenas dockerd[6349]: time=“2026-09-23T21:27:24.668974331+02:00” level=warning msg=“Health check for container a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.746927618+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.747768540+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.747869270+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.748114559+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.749326798+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.749510709+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.752881918+02:00” level=warning msg=“Health check for container 225436049f88482635e42a39f194f1f5dc8a99c3930e5202685d7c294e316e8f error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.752957665+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.753032197+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.753283283+02:00” level=warning msg=“Health check for container 60f629dadc216a2840702712834a2053ea80b36ed99c429c81b9f58782699579 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.753941294+02:00” level=warning msg=“Health check for container 1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.755562624+02:00” level=warning msg=“Health check for container a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.757363057+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.757364708+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:27:54 truenas dockerd[6349]: time=“2026-09-23T21:27:54.758393813+02:00” level=warning msg=“Health check for container abaac53ab148f1e1c99b59bf82f5a0f86ed0ece95e5cd23c79e35a918390fdf1 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.832147432+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.832256119+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.836644859+02:00” level=warning msg=“Health check for container 225436049f88482635e42a39f194f1f5dc8a99c3930e5202685d7c294e316e8f error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.837736366+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.837767571+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.839579623+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.839675225+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.841429223+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.841924368+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.844315369+02:00” level=warning msg=“Health check for container 1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.845088547+02:00” level=warning msg=“Health check for container 60f629dadc216a2840702712834a2053ea80b36ed99c429c81b9f58782699579 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.846199073+02:00” level=warning msg=“Health check for container a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.846930538+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.846972770+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:24 truenas dockerd[6349]: time=“2026-09-23T21:28:24.848081499+02:00” level=warning msg=“Health check for container abaac53ab148f1e1c99b59bf82f5a0f86ed0ece95e5cd23c79e35a918390fdf1 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.923462247+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.927666428+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.933423701+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.933806832+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.934475074+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.935754314+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.935368683+02:00” level=warning msg=“Health check for container 225436049f88482635e42a39f194f1f5dc8a99c3930e5202685d7c294e316e8f error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.935589951+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.935598037+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.936074414+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.936206172+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.938704635+02:00” level=warning msg=“Health check for container abaac53ab148f1e1c99b59bf82f5a0f86ed0ece95e5cd23c79e35a918390fdf1 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.938704594+02:00” level=warning msg=“Health check for container 60f629dadc216a2840702712834a2053ea80b36ed99c429c81b9f58782699579 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.939158256+02:00” level=warning msg=“Health check for container a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:28:54 truenas dockerd[6349]: time=“2026-09-23T21:28:54.939550011+02:00” level=warning msg=“Health check for container 1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.962736066+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.962856013+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.966024870+02:00” level=warning msg=“Health check for container 225436049f88482635e42a39f194f1f5dc8a99c3930e5202685d7c294e316e8f error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.967357363+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.967415421+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.967752104+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.967760247+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.967795722+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.967764687+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.969175550+02:00” level=warning msg=“Health check for container 60f629dadc216a2840702712834a2053ea80b36ed99c429c81b9f58782699579 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.969847711+02:00” level=warning msg=“Health check for container a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.969970585+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.969854540+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.970134383+02:00” level=warning msg=“Health check for container 1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:29:24 truenas dockerd[6349]: time=“2026-09-23T21:29:24.971703755+02:00” level=warning msg=“Health check for container abaac53ab148f1e1c99b59bf82f5a0f86ed0ece95e5cd23c79e35a918390fdf1 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.021091144+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.021199478+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.025415053+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.025474493+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.026481180+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.026885074+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.032722802+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.034046155+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.034107936+02:00” level=warning msg=“Health check for container abaac53ab148f1e1c99b59bf82f5a0f86ed0ece95e5cd23c79e35a918390fdf1 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.035918354+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.036739056+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.037279439+02:00” level=warning msg=“Health check for container a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.037319674+02:00” level=warning msg=“Health check for container 1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.038972783+02:00” level=warning msg=“Health check for container 225436049f88482635e42a39f194f1f5dc8a99c3930e5202685d7c294e316e8f error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 23 21:29:55 truenas dockerd[6349]: time=“2026-09-23T21:29:55.041525115+02:00” level=warning msg=“Health check for container 60f629dadc216a2840702712834a2053ea80b36ed99c429c81b9f58782699579 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”

Any further idea?

Thanks again for your help!

nite_route · September 23, 2026, 8:32pm

yeah those arent ghosts. nextcloud is running but every health check dies with setns / no such file or directory. that usually means the docker runtime got half broken after the usb pool import, not that overlay2 is empty.

dont docker rm the nextcloud containers yet.

try this order:

sudo systemctl stop docker
sudo systemctl start docker
sudo docker ps -a

if the health checks still spam that setns line, reboot the nas once. apps ui often comes back after docker fully restarts when that exact journal line shows up.

after docker is clean, open Apps again before you export that usb pool. if Apps still throws Validation Error, paste:

sudo docker inspect 1583d9ca802c --format '{{.State.Status}} {{.State.Health.Status}}'

and one full setns line from journalctl.

ProMG · September 24, 2026, 9:51am

I stopped docker.
However it does not start up again:

truenas_admin@truenas[~]$ sudo systemctl start docker
Job for docker.service failed because the control process exited with error code.
See “systemctl status docker.service” and “journalctl -xeu docker.service” for details.

truenas_admin@truenas[~]$ systemctl status docker.service
× docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled; preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─override.conf
Active: failed (Result: exit-code) since Thu 2026-09-24 11:41:30 CEST; 4min 9s ago
Duration: 10month 1w 1d 12h 33min 24.579s
TriggeredBy: × docker.socket
Docs: https://docs.docker.com
Process: 892985 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 892985 (code=exited, status=1/FAILURE)
CPU: 219ms

truenas_admin@truenas[~]$ sudo journalctl -xeu docker.service
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit docker.service has entered the ‘failed’ state with result ‘exit-code’.
Sep 24 11:41:54 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ A start job for unit docker.service has finished with a failure.
░░
░░ The job identifier is 3253611 and the job result is failed.
Sep 24 11:42:34 truenas systemd[1]: docker.service: Start request repeated too quickly.
Sep 24 11:42:34 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit docker.service has entered the ‘failed’ state with result ‘exit-code’.
Sep 24 11:42:34 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ A start job for unit docker.service has finished with a failure.
░░
░░ The job identifier is 3253683 and the job result is failed.
Sep 24 11:43:55 truenas systemd[1]: docker.service: Start request repeated too quickly.
Sep 24 11:43:55 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit docker.service has entered the ‘failed’ state with result ‘exit-code’.
Sep 24 11:43:55 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ A start job for unit docker.service has finished with a failure.
░░
░░ The job identifier is 3253905 and the job result is failed.
Sep 24 11:45:00 truenas systemd[1]: docker.service: Start request repeated too quickly.
Sep 24 11:45:00 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit docker.service has entered the ‘failed’ state with result ‘exit-code’.
Sep 24 11:45:00 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ A start job for unit docker.service has finished with a failure.
░░
░░ The job identifier is 3253977 and the job result is failed.

With the docker stopped I can access tha Datasets without any error message and I see all datasets.

The Apps also could be accessed without any error message. However, the table with the installed apps is empty.

I did not restart the system yet.

Appreciate any further help! Thanks!

nite_route · September 24, 2026, 12:22pm

yeah the journal you pasted is just systemd rate-limit spam. it never shows why dockerd itself exited.

also check the usb pool is still imported. if that drive dropped after export, docker often dies looking under /mnt/.ix-apps.

paste those two outputs. dont reboot yet unless you have to.

ProMG · September 24, 2026, 1:21pm

Thanks. I did and here are the outputs:

truenas_admin@truenas[~]$ sudo journalctl -u docker -b --no-pager | grep -iE “error|fatal|failed|cannot” | tail -40
Sep 24 11:40:43 truenas dockerd[6349]: time=“2026-09-24T11:40:43.300598079+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 24 11:40:43 truenas dockerd[6349]: time=“2026-09-24T11:40:43.300598084+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 24 11:40:43 truenas dockerd[6349]: time=“2026-09-24T11:40:43.301675490+02:00” level=warning msg=“Health check for container 1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 error: OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: fork/exec /proc/self/fd/6: no such file or directory: unknown”
Sep 24 11:40:47 truenas dockerd[6349]: time=“2026-09-24T11:40:47.297922440+02:00” level=info msg=“Container failed to exit within 10s of signal 28 - using the force” container=1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6
Sep 24 11:40:47 truenas dockerd[6349]: time=“2026-09-24T11:40:47.300228858+02:00” level=info msg=“Container failed to exit within 10s of signal 15 - using the force” container=a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207
Sep 24 11:40:47 truenas dockerd[6349]: time=“2026-09-24T11:40:47.379644801+02:00” level=warning msg=“ShouldRestart failed, container will not be restarted” container=1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 daemonShuttingDown=true error=“restart canceled” execDuration=42h12m0.414293029s exitStatus=“{137 2026-09-24 09:40:47.311335384 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Sep 24 11:40:47 truenas dockerd[6349]: time=“2026-09-24T11:40:47.379656794+02:00” level=warning msg=“ShouldRestart failed, container will not be restarted” container=a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 daemonShuttingDown=true error=“restart canceled” execDuration=43h21m9.449734615s exitStatus=“{137 2026-09-24 09:40:47.325684907 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Sep 24 11:40:47 truenas dockerd[6349]: time=“2026-09-24T11:40:47.497238327+02:00” level=warning msg=“Failed deleting service host entries to the running container: open /mnt/.ix-apps/docker/containers/1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6/hosts: no such file or directory”
Sep 24 11:40:47 truenas dockerd[6349]: time=“2026-09-24T11:40:47.600226116+02:00” level=error msg=“failed to process event” container=1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 error=“open /mnt/.ix-apps/docker/containers/1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6/.tmp-config.v2.json4287683994: no such file or directory” event=exit event-info=“{1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 482663 137 2026-09-24 09:40:47.311335384 +0000 UTC }” module=libcontainerd namespace=moby
Sep 24 11:40:47 truenas dockerd[6349]: time=“2026-09-24T11:40:47.601972642+02:00” level=error msg=“umount /mnt/.ix-apps/docker/overlay2/2a18a5b79fdbda3aec58207273fd3e630a1f1f3d0797cb69b2d8b8511f1ddc7d/merged, flags: 0x2: no such file or directory”
Sep 24 11:40:48 truenas dockerd[6349]: time=“2026-09-24T11:40:48.073278335+02:00” level=warning msg=“Failed deleting service host entries to the running container: open /mnt/.ix-apps/docker/containers/a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207/hosts: no such file or directory”
Sep 24 11:40:48 truenas dockerd[6349]: time=“2026-09-24T11:40:48.213665493+02:00” level=error msg=“failed to process event” container=a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 error=“open /mnt/.ix-apps/docker/containers/a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207/.tmp-config.v2.json1744285933: no such file or directory” event=exit event-info=“{a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 415967 137 2026-09-24 09:40:47.325684907 +0000 UTC }” module=libcontainerd namespace=moby
Sep 24 11:40:48 truenas dockerd[6349]: time=“2026-09-24T11:40:48.215810010+02:00” level=error msg=“umount /mnt/.ix-apps/docker/overlay2/4d711df813d3ee2e5b117a3091114c93eaef393a8386582b8fa8198082806c9b/merged, flags: 0x2: no such file or directory”
Sep 24 11:41:02 truenas dockerd[6349]: time=“2026-09-24T11:41:02.476088473+02:00” level=error msg=“error during layerStore.Cleanup(): umount /mnt/.ix-apps/docker/overlay2/2a18a5b79fdbda3aec58207273fd3e630a1f1f3d0797cb69b2d8b8511f1ddc7d/merged, flags: 0x2: no such file or directory (possible cause: umount /mnt/.ix-apps/docker/overlay2/4d711df813d3ee2e5b117a3091114c93eaef393a8386582b8fa8198082806c9b/merged, flags: 0x2: no such file or directory)”
Sep 24 11:41:02 truenas dockerd[6349]: time=“2026-09-24T11:41:02.500674101+02:00” level=info msg=“stopping event stream following graceful shutdown” error=“” module=libcontainerd namespace=moby
Sep 24 11:41:02 truenas dockerd[6349]: time=“2026-09-24T11:41:02.501470644+02:00” level=error msg=“umount /mnt/.ix-apps/docker/overlay2/4d711df813d3ee2e5b117a3091114c93eaef393a8386582b8fa8198082806c9b/merged, flags: 0x2: no such file or directory”
Sep 24 11:41:02 truenas dockerd[6349]: time=“2026-09-24T11:41:02.501586095+02:00” level=error msg=“umount /mnt/.ix-apps/docker/overlay2/820d421278a8b03ae470b011b9573e407e8bb49663b1ca2e9829eaaa8e733833/merged, flags: 0x2: no such file or directory”
Sep 24 11:41:02 truenas dockerd[6349]: time=“2026-09-24T11:41:02.501864746+02:00” level=error msg=“umount /mnt/.ix-apps/docker/overlay2/64c65169b5a246a057c341c766b38130fae5159a13007dd0ecde9de465b54cae/merged, flags: 0x2: no such file or directory”
Sep 24 11:41:02 truenas dockerd[6349]: time=“2026-09-24T11:41:02.501954154+02:00” level=error msg=“umount /mnt/.ix-apps/docker/overlay2/aa51765b14c999c404f6b624069a9f438427023e22cc32904f1f3f84b3d2b66d/merged, flags: 0x2: no such file or directory”
Sep 24 11:41:02 truenas dockerd[6349]: time=“2026-09-24T11:41:02.502219039+02:00” level=error msg=“umount /mnt/.ix-apps/docker/overlay2/2a18a5b79fdbda3aec58207273fd3e630a1f1f3d0797cb69b2d8b8511f1ddc7d/merged, flags: 0x2: no such file or directory”
Sep 24 11:41:02 truenas dockerd[6349]: time=“2026-09-24T11:41:02.506600690+02:00” level=info msg=“stopping event stream following graceful shutdown” error=“context canceled” module=libcontainerd namespace=plugins.moby
Sep 24 11:41:07 truenas dockerd[6349]: time=“2026-09-24T11:41:07.508648102+02:00” level=error msg=“Error shutting down http server” error=“context canceled”
Sep 24 11:41:28 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
Sep 24 11:41:28 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.

truenas_admin@truenas[~]$ sudo dockerd --debug 2>&1 | head -80
time=“2026-09-24T15:18:32.482362147+02:00” level=info msg=“Starting up”
chmod /mnt/.ix-apps/docker: read-only file system

The USB pool is still imported.

nite_route · September 24, 2026, 2:38pm

yeah there it is.

chmod /mnt/.ix-apps/docker: read-only file system

dockerd cant write its own dir, so start keeps dying. the missing hosts/overlay2 paths line up with that.

check:
zpool status
mount | grep -E ‘ix-apps|/mnt/’
zfs list -o name,mounted,readonly | grep -i ix
ls -ld /mnt/.ix-apps /mnt/.ix-apps/docker

if the dataset is readonly=on:
zfs set readonly=off DATASET_NAME
(use the real name from that zfs list)

if the mount itself is ro:
mount -o remount,rw /mnt/POOL

then:
sudo systemctl reset-failed docker.socket docker.service
sudo systemctl start containerd
sudo systemctl start docker
sudo docker ps -a

paste the zfs/mount lines if it still fails.

ProMG · September 25, 2026, 8:05am

Yeeeaaaah, you are my hero!

The datasets on the extermal pool were readonly=on:

truenas_admin@truenas[~]$ sudo zfs list -o name,mounted,readonly | grep -i ix
[sudo] password for truenas_admin:
Backup_extern/ix-apps yes on
Backup_extern/ix-apps/app_configs yes on
Backup_extern/ix-apps/app_mounts yes on
Backup_extern/ix-apps/docker yes on
Backup_extern/ix-apps/truenas_catalog yes on
Backup_intern/ix-apps no on
Backup_intern/ix-apps/app_configs no on
Backup_intern/ix-apps/app_mounts no on
Backup_intern/ix-apps/docker no on
Backup_intern/ix-apps/truenas_catalog no on
Gsaenger-Pool/ix-apps yes off
Gsaenger-Pool/ix-apps/app_configs yes off
Gsaenger-Pool/ix-apps/app_mounts yes off
Gsaenger-Pool/ix-apps/docker yes off
Gsaenger-Pool/ix-apps/truenas_catalog yes off

I changed them to off and everything is now as it should be. Thanks for your great support!!! I would have been lost without your hints.

ProMG · September 25, 2026, 8:50am

Arrrrghhhh, it broke up again.

Everything was running. What I did the is to start a manual replication to the external USB drive. This was successful.

However, now all apps are gone again and docker ist not coming up. Hmpffff.

I tried to use the same solution, but this time changing readonly from on to off did not help to recover the system.

Docker status is now as follows:

truenas_admin@truenas[~]$ sudo journalctl -u docker -b --no-pager | grep -iE “error|fatal|failed|cannot” | tail -40
Sep 25 10:08:59 truenas dockerd[908834]: time=“2026-09-25T10:08:59.067024241+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 25 10:08:59 truenas dockerd[908834]: time=“2026-09-25T10:08:59.069243594+02:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 25 10:08:59 truenas dockerd[908834]: time=“2026-09-25T10:08:59.073272659+02:00” level=warning msg=“Health check for container 0ed72cb4f354c8de564c1e1d7675e2da92618656e00b9cd3ee361a040bbae66a error: OCI runtime exec failed: exec failed: unable to start container process: chdir to cwd (”/app") set in config.json failed: permission denied: unknown"
Sep 25 10:08:59 truenas dockerd[908834]: time=“2026-09-25T10:08:59.361181662+02:00” level=error msg=“Handler for GET /v1.47/containers/json returned error: write unix /run/docker.sock->@: write: broken pipe”
Sep 25 10:08:59 truenas dockerd[908834]: time=“2026-09-25T10:08:59.361385092+02:00” level=error msg=“Handler for GET /v1.47/containers/json returned error: write unix /run/docker.sock->@: write: broken pipe”
Sep 25 10:09:00 truenas dockerd[908834]: time=“2026-09-25T10:09:00.287006091+02:00” level=error msg=“Error writing log message” driver=json-file error=“error writing log entry: write /mnt/.ix-apps/docker/containers/9b724a2fc11d2d46c38d65b2a4fce959035012f1232daa84cc3d58af8cdefa72/9b724a2fc11d2d46c38d65b2a4fce959035012f1232daa84cc3d58af8cdefa72-json.log: input/output error” message=
Sep 25 10:09:01 truenas dockerd[908834]: time=“2026-09-25T10:09:01.385421279+02:00” level=warning msg=“driver failed revoking external connectivity on endpoint ix-nextcloud-cron-1 (4eb619d998b8e6e6c5994e844db9b35e005021d9f7fc52fed2265e72954eb9bd): failed to update bridge endpoint 4eb619d to store: failed to update bridge store for object type *bridge.bridgeEndpoint: Key not found in store”
Sep 25 10:09:01 truenas dockerd[908834]: time=“2026-09-25T10:09:01.425360956+02:00” level=warning msg=“Failed detaching sandbox 64c0e544065a2fc0c9da0411860c152a662b0efa4f6058c3c3769cd78fb9fb29 from endpoint 4eb619d998b8e6e6c5994e844db9b35e005021d9f7fc52fed2265e72954eb9bd: failed to update store for object type *libnetwork.Endpoint: Key not found in store\n”
Sep 25 10:09:01 truenas dockerd[908834]: time=“2026-09-25T10:09:01.426380988+02:00” level=warning msg=“Failed deleting endpoint 4eb619d998b8e6e6c5994e844db9b35e005021d9f7fc52fed2265e72954eb9bd: endpoint with name ix-nextcloud-cron-1 id 4eb619d998b8e6e6c5994e844db9b35e005021d9f7fc52fed2265e72954eb9bd has active containers\n”
Sep 25 10:09:01 truenas dockerd[908834]: time=“2026-09-25T10:09:01.429650169+02:00” level=warning msg=“Failed to delete sandbox 64c0e544065a2fc0c9da0411860c152a662b0efa4f6058c3c3769cd78fb9fb29 from store: Key not found in store”
Sep 25 10:09:01 truenas dockerd[908834]: time=“2026-09-25T10:09:01.532523435+02:00” level=error msg=“failed to process event” container=9b724a2fc11d2d46c38d65b2a4fce959035012f1232daa84cc3d58af8cdefa72 error=“open /mnt/.ix-apps/docker/containers/9b724a2fc11d2d46c38d65b2a4fce959035012f1232daa84cc3d58af8cdefa72/.tmp-config.v2.json2919025517: no such file or directory” event=exit event-info=“{9b724a2fc11d2d46c38d65b2a4fce959035012f1232daa84cc3d58af8cdefa72 9b724a2fc11d2d46c38d65b2a4fce959035012f1232daa84cc3d58af8cdefa72 967716 135 2026-09-25 08:09:01.302405908 +0000 UTC }” module=libcontainerd namespace=moby
Sep 25 10:09:01 truenas dockerd[908834]: time=“2026-09-25T10:09:01.538960416+02:00” level=error msg=“9b724a2fc11d2d46c38d65b2a4fce959035012f1232daa84cc3d58af8cdefa72: failed saving state on start failure: open /mnt/.ix-apps/docker/containers/9b724a2fc11d2d46c38d65b2a4fce959035012f1232daa84cc3d58af8cdefa72/.tmp-config.v2.json800348054: no such file or directory”
Sep 25 10:09:01 truenas dockerd[908834]: time=“2026-09-25T10:09:01.554914957+02:00” level=error msg=“9b724a2fc11d2d46c38d65b2a4fce959035012f1232daa84cc3d58af8cdefa72 cleanup: failed to delete container from containerd: NotFound: container “9b724a2fc11d2d46c38d65b2a4fce959035012f1232daa84cc3d58af8cdefa72” in namespace “moby”: not found”
Sep 25 10:09:01 truenas dockerd[908834]: time=“2026-09-25T10:09:01.558500639+02:00” level=error msg=“restartmanger wait error: stat /mnt/.ix-apps/docker/overlay2/e023f3a99c7c858cbd1297eed1e3634a862cff2814007ad12dd46f8624d29cfa: no such file or directory”
Sep 25 10:09:05 truenas dockerd[908834]: time=“2026-09-25T10:09:05.656636277+02:00” level=warning msg=“driver failed revoking external connectivity on endpoint ix-nginx-proxy-manager-npm-1 (24ec99c4626da4eb888c63bf2e5963876687523a41748972167c5a0b5980af0a): failed to update bridge endpoint 24ec99c to store: failed to update bridge store for object type *bridge.bridgeEndpoint: Key not found in store”
Sep 25 10:09:05 truenas dockerd[908834]: time=“2026-09-25T10:09:05.699720714+02:00” level=warning msg=“Failed detaching sandbox f38880f1ac81401a77b170adfbcba5112d2414af3da9bbad340d17d630d2b5c1 from endpoint 24ec99c4626da4eb888c63bf2e5963876687523a41748972167c5a0b5980af0a: failed to update store for object type *libnetwork.Endpoint: write /mnt/.ix-apps/docker/network/files/local-kv.db: read-only file system\n”
Sep 25 10:09:05 truenas dockerd[908834]: time=“2026-09-25T10:09:05.699944136+02:00” level=warning msg=“Failed deleting endpoint 24ec99c4626da4eb888c63bf2e5963876687523a41748972167c5a0b5980af0a: endpoint with name ix-nginx-proxy-manager-npm-1 id 24ec99c4626da4eb888c63bf2e5963876687523a41748972167c5a0b5980af0a has active containers\n”
Sep 25 10:09:05 truenas dockerd[908834]: time=“2026-09-25T10:09:05.701093612+02:00” level=warning msg=“Failed to delete sandbox f38880f1ac81401a77b170adfbcba5112d2414af3da9bbad340d17d630d2b5c1 from store: Key not found in store”
Sep 25 10:09:05 truenas dockerd[908834]: time=“2026-09-25T10:09:05.704948794+02:00” level=error msg=“failed to process event” container=0ed72cb4f354c8de564c1e1d7675e2da92618656e00b9cd3ee361a040bbae66a error=“open /mnt/.ix-apps/docker/containers/0ed72cb4f354c8de564c1e1d7675e2da92618656e00b9cd3ee361a040bbae66a/.tmp-config.v2.json3937021031: no such file or directory” event=exit event-info=“{0ed72cb4f354c8de564c1e1d7675e2da92618656e00b9cd3ee361a040bbae66a 0ed72cb4f354c8de564c1e1d7675e2da92618656e00b9cd3ee361a040bbae66a 928621 135 2026-09-25 08:09:05.029494371 +0000 UTC }” module=libcontainerd namespace=moby
Sep 25 10:09:05 truenas dockerd[908834]: time=“2026-09-25T10:09:05.705415884+02:00” level=error msg=“0ed72cb4f354c8de564c1e1d7675e2da92618656e00b9cd3ee361a040bbae66a: failed saving state on start failure: open /mnt/.ix-apps/docker/containers/0ed72cb4f354c8de564c1e1d7675e2da92618656e00b9cd3ee361a040bbae66a/.tmp-config.v2.json4162898827: no such file or directory”
Sep 25 10:09:05 truenas dockerd[908834]: time=“2026-09-25T10:09:05.708896918+02:00” level=error msg=“0ed72cb4f354c8de564c1e1d7675e2da92618656e00b9cd3ee361a040bbae66a cleanup: failed to delete container from containerd: NotFound: container “0ed72cb4f354c8de564c1e1d7675e2da92618656e00b9cd3ee361a040bbae66a” in namespace “moby”: not found”
Sep 25 10:09:05 truenas dockerd[908834]: time=“2026-09-25T10:09:05.711792239+02:00” level=error msg=“restartmanger wait error: stat /mnt/.ix-apps/docker/overlay2/85b6a616d6045749490882fe91211a439fff220175e4cb6c2790f42f0ff706ed: no such file or directory”
Sep 25 10:09:11 truenas dockerd[908834]: time=“2026-09-25T10:09:11.945476262+02:00” level=warning msg=“driver failed revoking external connectivity on endpoint ix-nextcloud-postgres-1 (568e10c4e207f2f10b824d5f497a08ee5d8ace708466126f0682187fd7f331b6): failed to update bridge endpoint 568e10c to store: failed to update bridge store for object type *bridge.bridgeEndpoint: Key not found in store”
Sep 25 10:09:11 truenas dockerd[908834]: time=“2026-09-25T10:09:11.996502160+02:00” level=warning msg=“Failed detaching sandbox b904a8e0c890649f746e0dc6a71a7ca3fba9ec6427fbf1843010a9394578238a from endpoint 568e10c4e207f2f10b824d5f497a08ee5d8ace708466126f0682187fd7f331b6: failed to update store for object type *libnetwork.Endpoint: Key not found in store\n”
Sep 25 10:09:11 truenas dockerd[908834]: time=“2026-09-25T10:09:11.996877958+02:00” level=warning msg=“Failed deleting endpoint 568e10c4e207f2f10b824d5f497a08ee5d8ace708466126f0682187fd7f331b6: endpoint with name ix-nextcloud-postgres-1 id 568e10c4e207f2f10b824d5f497a08ee5d8ace708466126f0682187fd7f331b6 has active containers\n”
Sep 25 10:09:11 truenas dockerd[908834]: time=“2026-09-25T10:09:11.999061897+02:00” level=warning msg=“Failed to delete sandbox b904a8e0c890649f746e0dc6a71a7ca3fba9ec6427fbf1843010a9394578238a from store: Key not found in store”
Sep 25 10:09:12 truenas dockerd[908834]: time=“2026-09-25T10:09:12.004585500+02:00” level=error msg=“failed to process event” container=7c3cabc7337db26866bffebb237753787d1d166c444529fddc8a15342213561a error=“open /mnt/.ix-apps/docker/containers/7c3cabc7337db26866bffebb237753787d1d166c444529fddc8a15342213561a/.tmp-config.v2.json334343417: no such file or directory” event=exit event-info=“{7c3cabc7337db26866bffebb237753787d1d166c444529fddc8a15342213561a 7c3cabc7337db26866bffebb237753787d1d166c444529fddc8a15342213561a 967108 135 2026-09-25 08:09:11.914661997 +0000 UTC }” module=libcontainerd namespace=moby
Sep 25 10:09:12 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
Sep 25 10:09:14 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
Sep 25 10:09:14 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
Sep 25 10:09:16 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
Sep 25 10:09:16 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
Sep 25 10:33:34 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
Sep 25 10:33:34 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
Sep 25 10:33:36 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
Sep 25 10:33:36 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
Sep 25 10:37:27 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
Sep 25 10:37:27 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
Sep 25 10:37:29 truenas systemd[1]: docker.service: Failed with result ‘exit-code’.
Sep 25 10:37:29 truenas systemd[1]: Failed to start docker.service - Docker Application Container Engine.

truenas_admin@truenas[~]$ sudo dockerd --debug 2>&1 | head -80
time=“2026-09-25T10:47:38.070066306+02:00” level=info msg=“Starting up”
time=“2026-09-25T10:47:38.071257082+02:00” level=debug msg=“Listener created for HTTP on unix (/var/run/docker.sock)”
time=“2026-09-25T10:47:38.071567373+02:00” level=info msg=“OTEL tracing is not configured, using no-op tracer provider”
time=“2026-09-25T10:47:38.082419650+02:00” level=debug msg=“Golang’s threads limit set to 110340”
time=“2026-09-25T10:47:38.083073589+02:00” level=debug msg=“metrics API listening on /var/run/docker/metrics.sock”
time=“2026-09-25T10:47:38.085921729+02:00” level=debug msg=“Using default logging driver json-file”
time=“2026-09-25T10:47:38.086015404+02:00” level=debug msg=“processing event stream” module=libcontainerd namespace=plugins.moby
time=“2026-09-25T10:47:38.086452252+02:00” level=debug msg=“No quota support for local volumes in /mnt/.ix-apps/docker/volumes: Filesystem does not support, or has not enabled quotas”
time=“2026-09-25T10:47:38.171908969+02:00” level=info msg=“[graphdriver] trying configured driver: overlay2”
time=“2026-09-25T10:47:38.314801487+02:00” level=debug msg=“successfully detected metacopy status” storage-driver=overlay2 usingMetacopy=false
time=“2026-09-25T10:47:38.340767476+02:00” level=debug msg=“backingFs=zfs, projectQuotaSupported=false, usingMetacopy=false, indexOff=“index=off,”, userxattr=”“” storage-driver=overlay2
time=“2026-09-25T10:47:38.340872633+02:00” level=debug msg=“Initialized graph driver overlay2”
time=“2026-09-25T10:47:38.445352522+02:00” level=debug msg=“Max Concurrent Downloads: 3”
time=“2026-09-25T10:47:38.445372395+02:00” level=debug msg=“Max Concurrent Uploads: 5”
time=“2026-09-25T10:47:38.445377901+02:00” level=debug msg=“Max Download Attempts: 5”
time=“2026-09-25T10:47:38.445402167+02:00” level=info msg=“Loading containers: start.”
time=“2026-09-25T10:47:38.446410212+02:00” level=debug msg=“loaded container” container=a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 paused=false running=false
time=“2026-09-25T10:47:38.446430345+02:00” level=debug msg=“loaded container” container=c2e0ffba2deb1ef25c11d43e7513b43b55e9dd5dc5a855e0f75127ad294b38a1 paused=false running=false
time=“2026-09-25T10:47:38.446485844+02:00” level=debug msg=“loaded container” container=225436049f88482635e42a39f194f1f5dc8a99c3930e5202685d7c294e316e8f paused=false running=false
time=“2026-09-25T10:47:38.446493562+02:00” level=debug msg=“processing event stream” module=libcontainerd namespace=moby
time=“2026-09-25T10:47:38.446536820+02:00” level=debug msg=“loaded container” container=abaac53ab148f1e1c99b59bf82f5a0f86ed0ece95e5cd23c79e35a918390fdf1 paused=false running=false
time=“2026-09-25T10:47:38.446611527+02:00” level=debug msg=“loaded container” container=1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 paused=false running=false
time=“2026-09-25T10:47:38.446819093+02:00” level=debug msg=“loaded container” container=60f629dadc216a2840702712834a2053ea80b36ed99c429c81b9f58782699579 paused=false running=false
time=“2026-09-25T10:47:38.446841096+02:00” level=debug msg=“loaded container” container=6e92a7ce67287d1b30833aa235cecaea8d2ba38af70a92b8dc20a406a21c06a0 paused=false running=false
time=“2026-09-25T10:47:38.538290265+02:00” level=debug msg=“restoring container” container=60f629dadc216a2840702712834a2053ea80b36ed99c429c81b9f58782699579 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.538366132+02:00” level=debug msg=“restoring container” container=c2e0ffba2deb1ef25c11d43e7513b43b55e9dd5dc5a855e0f75127ad294b38a1 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.538439056+02:00” level=debug msg=“restoring container” container=1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.540636588+02:00” level=debug msg=“done restoring container” container=1583d9ca802c93ca6e1ec75123d415de4bf21956c582b9fe45c8f779a999fdf6 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.540674966+02:00” level=debug msg=“done restoring container” container=60f629dadc216a2840702712834a2053ea80b36ed99c429c81b9f58782699579 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.540820217+02:00” level=debug msg=“done restoring container” container=c2e0ffba2deb1ef25c11d43e7513b43b55e9dd5dc5a855e0f75127ad294b38a1 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.546982754+02:00” level=debug msg=“restoring container” container=abaac53ab148f1e1c99b59bf82f5a0f86ed0ece95e5cd23c79e35a918390fdf1 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.547152159+02:00” level=debug msg=“restoring container” container=a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.547362135+02:00” level=debug msg=“restoring container” container=6e92a7ce67287d1b30833aa235cecaea8d2ba38af70a92b8dc20a406a21c06a0 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.547518713+02:00” level=debug msg=“restoring container” container=225436049f88482635e42a39f194f1f5dc8a99c3930e5202685d7c294e316e8f paused=false restarting=false running=false
time=“2026-09-25T10:47:38.549085285+02:00” level=debug msg=“done restoring container” container=225436049f88482635e42a39f194f1f5dc8a99c3930e5202685d7c294e316e8f paused=false restarting=false running=false
time=“2026-09-25T10:47:38.549089045+02:00” level=debug msg=“done restoring container” container=abaac53ab148f1e1c99b59bf82f5a0f86ed0ece95e5cd23c79e35a918390fdf1 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.549086710+02:00” level=debug msg=“done restoring container” container=6e92a7ce67287d1b30833aa235cecaea8d2ba38af70a92b8dc20a406a21c06a0 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.549477294+02:00” level=debug msg=“done restoring container” container=a7741ef1d9f154becf82284d243d6d63c1c6a78a3dc277d969624de4654fd207 paused=false restarting=false running=false
time=“2026-09-25T10:47:38.549686285+02:00” level=debug msg=“Option DefaultDriver: bridge”
time=“2026-09-25T10:47:38.549736213+02:00” level=debug msg=“Option DefaultNetwork: bridge”
time=“2026-09-25T10:47:38.549763120+02:00” level=debug msg=“Network Control Plane MTU: 1500”
time=“2026-09-25T10:47:38.564771991+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -C FORWARD -j DOCKER-ISOLATION]”
time=“2026-09-25T10:47:38.571343865+02:00” level=debug msg=“/sbin/iptables, [–wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]”
time=“2026-09-25T10:47:38.578476766+02:00” level=debug msg=“/sbin/iptables, [–wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]”
time=“2026-09-25T10:47:38.585419089+02:00” level=debug msg=“/sbin/iptables, [–wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]”
time=“2026-09-25T10:47:38.591878300+02:00” level=debug msg=“/sbin/iptables, [–wait -t nat -D PREROUTING]”
time=“2026-09-25T10:47:38.597436814+02:00” level=debug msg=“/sbin/iptables, [–wait -t nat -D OUTPUT]”
time=“2026-09-25T10:47:38.602903879+02:00” level=debug msg=“/sbin/iptables, [–wait -t nat -F DOCKER]”
time=“2026-09-25T10:47:38.607864657+02:00” level=debug msg=“/sbin/iptables, [–wait -t nat -X DOCKER]”
time=“2026-09-25T10:47:38.629124105+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -F DOCKER]”
time=“2026-09-25T10:47:38.635040507+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -X DOCKER]”
time=“2026-09-25T10:47:38.661513289+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -F DOCKER-ISOLATION-STAGE-1]”
time=“2026-09-25T10:47:38.685191285+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -X DOCKER-ISOLATION-STAGE-1]”
time=“2026-09-25T10:47:38.705513658+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -F DOCKER-ISOLATION-STAGE-2]”
time=“2026-09-25T10:47:38.725183214+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -X DOCKER-ISOLATION-STAGE-2]”
time=“2026-09-25T10:47:38.749384854+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -F DOCKER-ISOLATION]”
time=“2026-09-25T10:47:38.755881613+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -X DOCKER-ISOLATION]”
time=“2026-09-25T10:47:38.761862885+02:00” level=debug msg=“/sbin/iptables, [–wait -t nat -n -L DOCKER]”
time=“2026-09-25T10:47:38.767903839+02:00” level=debug msg=“/sbin/iptables, [–wait -t nat -N DOCKER]”
time=“2026-09-25T10:47:38.774554694+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -n -L DOCKER]”
time=“2026-09-25T10:47:38.781315963+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -n -L DOCKER-ISOLATION-STAGE-1]”
time=“2026-09-25T10:47:38.787676353+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -n -L DOCKER-ISOLATION-STAGE-2]”
time=“2026-09-25T10:47:38.794287023+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -N DOCKER-ISOLATION-STAGE-2]”
time=“2026-09-25T10:47:38.800110358+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -C DOCKER-ISOLATION-STAGE-1 -j RETURN]”
time=“2026-09-25T10:47:38.806160173+02:00” level=debug msg=“/sbin/iptables, [–wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN]”
time=“2026-09-25T10:47:38.810439381+02:00” level=debug msg=“/sbin/iptables, [–wait -t filter -C DOCKER-ISOLATION-STAGE-2 -j RETURN]”
time=“2026-09-25T10:47:38.816565599+02:00” level=debug msg=“/sbin/iptables, [–wait -A DOCKER-ISOLATION-STAGE-2 -j RETURN]”
time=“2026-09-25T10:47:38.822875409+02:00” level=debug msg=“/sbin/iptables, [–wait -t nat -C DOCKER -i loopback0 -d 127.0.0.0/8 -j RETURN]”
time=“2026-09-25T10:47:38.830403505+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t filter -n -L FORWARD]”
time=“2026-09-25T10:47:38.837686532+02:00” level=debug msg=“Modules already loaded” modules=“[ip6_tables]”
time=“2026-09-25T10:47:38.837805708+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t filter -C FORWARD -j DOCKER-ISOLATION]”
time=“2026-09-25T10:47:38.844434773+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]”
time=“2026-09-25T10:47:38.850093612+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst ::1/128 -j DOCKER]”
time=“2026-09-25T10:47:38.853918922+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]”
time=“2026-09-25T10:47:38.857393385+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t nat -D PREROUTING]”
time=“2026-09-25T10:47:38.860885463+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t nat -D OUTPUT]”
time=“2026-09-25T10:47:38.864180166+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t nat -F DOCKER]”
time=“2026-09-25T10:47:38.867607066+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t nat -X DOCKER]”
time=“2026-09-25T10:47:38.897165431+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t filter -F DOCKER]”
time=“2026-09-25T10:47:38.903363868+02:00” level=debug msg=“/sbin/ip6tables, [–wait -t filter -X DOCKER]”

Seems to be something different now. Any idea what to do next?

nite_route · September 25, 2026, 10:09am

yeah the replication is flipping those Backup_extern/ix-apps datasets back to readonly=on. zfs receive does that by default.

your live apps shouldnt be sitting on the usb backup pool tho. from the earlier list Gsaenger-Pool/ix-apps is the real one. Backup_extern/ix-apps is a copy that keeps biting docker when receive remounts it ro.

try doin this:

check what /mnt/.ix-apps actually is

readlink -f /mnt/.ix-apps
df -h /mnt/.ix-apps
zfs list -o name,mountpoint,readonly | grep -i ix-apps

exclude ix-apps from that usb replication task (or destroy Backup_extern/ix-apps if its just a leftover receive). dont replicate the live apps dataset onto the usb box again
make sure Gsaenger-Pool/ix-apps stays readonly=off, then

sudo systemctl reset-failed docker.socket docker.service
sudo systemctl start containerd
sudo systemctl start docker

if dockerd still dies, paste

zfs get readonly Backup_extern/ix-apps Gsaenger-Pool/ix-apps
mount | grep ix-apps
sudo journalctl -u docker -b --no-pager | grep -iE 'read-only|chmod|permission|fatal' | tail -30

until the usb receive stops touching ix-apps this will keep coming back after every run.

ProMG · September 25, 2026, 10:57am

I excluded ix-apps from that usb replication task. I did not destroy Backup_extern/ix-apps so far.

Docker is now up again, nextcloud and nginx are working as expected.

However, I think the apps are now sitting on the Backup_extern pool:

truenas_admin@truenas[~]$ readlink -f /mnt/.ix-apps
/mnt/.ix-apps

truenas_admin@truenas[~]$ df -h /mnt/.ix-apps
Filesystem Size Used Avail Use% Mounted on
Backup_extern/ix-apps 3.9T 128K 3.9T 1% /mnt/.ix-apps

truenas_admin@truenas[~]$ sudo zfs list -o name,mountpoint,readonly | grep -i ix-apps
Backup_extern/ix-apps /mnt/.ix-apps off
Backup_extern/ix-apps/app_configs /mnt/.ix-apps/app_configs off
Backup_extern/ix-apps/app_mounts /mnt/.ix-apps/app_mounts off
Backup_extern/ix-apps/docker /mnt/.ix-apps/docker off
Backup_extern/ix-apps/truenas_catalog /mnt/.ix-apps/truenas_catalog off
Backup_intern/ix-apps /mnt/Backup_intern/ix-apps on
Backup_intern/ix-apps/app_configs /mnt/Backup_intern/ix-apps/app_configs on
Backup_intern/ix-apps/app_mounts /mnt/Backup_intern/ix-apps/app_mounts on
Backup_intern/ix-apps/docker /mnt/Backup_intern/ix-apps/docker on
Backup_intern/ix-apps/truenas_catalog /mnt/Backup_intern/ix-apps/truenas_catalog on
Gsaenger-Pool/ix-apps /mnt/.ix-apps off
Gsaenger-Pool/ix-apps/app_configs /mnt/.ix-apps/app_configs off
Gsaenger-Pool/ix-apps/app_mounts /mnt/.ix-apps/app_mounts off
Gsaenger-Pool/ix-apps/docker /mnt/.ix-apps/docker off
Gsaenger-Pool/ix-apps/truenas_catalog /mnt/.ix-apps/truenas_catalog off

Any chance to get them back to Gsaenger-Pool?

nite_route · September 25, 2026, 12:14pm

yeah /mnt/.ix-apps is currently Backup_extern, so docker is living on the usb pool. Gsaenger-Pool/ix-apps wants the same mountpoint but lost the race.

try doin this:

stop all apps in the ui
sudo systemctl stop docker
move the usb copy out of the way so it stops claiming /mnt/.ix-apps:
sudo zfs set mountpoint=/mnt/Backup_extern/ix-apps-leftover Backup_extern/ix-apps
make sure the real one is mounted:
sudo zfs mount Gsaenger-Pool/ix-apps
df -h /mnt/.ix-apps
should show Gsaenger-Pool/ix-apps
sudo systemctl start docker
Apps → Settings → Choose Pool → Gsaenger-Pool if the ui asks

if step 3 complains about busy mounts, unmount kids first:
sudo zfs unmount -f Backup_extern/ix-apps/docker
sudo zfs unmount -f Backup_extern/ix-apps
then set the mountpoint.

once Gsaenger-Pool owns /mnt/.ix-apps again you can destroy Backup_extern/ix-apps if you dont need that leftover receive.

ProMG · September 25, 2026, 2:04pm

Setting mountpoint yields the same error as for unmounting the kids:

truenas_admin@truenas[~]$ sudo zfs set mountpoint=/mnt/Backup_extern/ix-apps-leftover Backup_extern/ix-apps
cannot unmount ‘/mnt/.ix-apps/docker’: pool or dataset is busy

truenas_admin@truenas[~]$ sudo zfs unmount -f Backup_extern/ix-apps/docker
cannot unmount ‘/mnt/.ix-apps/docker’: pool or dataset is busy

truenas_admin@truenas[~]$ sudo zfs unmount -f Backup_extern/ix-apps
cannot unmount ‘/mnt/.ix-apps/docker’: pool or dataset is busy

nite_route · September 25, 2026, 2:12pm

docker still has the overlay open on that usb dataset so zfs wont let go.

try doin this:

stop apps in the ui
sudo systemctl stop docker containerd
check whats holding the path:
sudo lsof +D /mnt/.ix-apps 2>/dev/null | head
sudo fuser -mv /mnt/.ix-apps/docker
kill any leftover pids from that, then:
sudo zfs unmount -f Backup_extern/ix-apps/docker
sudo zfs unmount -f Backup_extern/ix-apps
sudo zfs set mountpoint=/mnt/Backup_extern/ix-apps-leftover Backup_extern/ix-apps
sudo zfs mount Gsaenger-Pool/ix-apps
df -h /mnt/.ix-apps
should show Gsaenger-Pool/ix-apps
sudo systemctl start containerd docker

if unmount still says busy after docker is fully stopped, reboot once with the usb pool exported (or the disk unplugged), then do steps 5-8 before you import Backup_extern again.

ProMG · September 26, 2026, 8:02am

I will be on vacation for two weeks from today. I will not touch the system before leaving for the airport, as the NAS is operational for the moment and I can access my data.

I will work on moving the ix-apps mount to the local pool after my return on October 11th. I will share success or failure within this thread.

Thanks for the moment for your great support!

ProMG · October 13, 2026, 10:16am

Finally, I could solve the issue.

Unmount still said beeing busy after docker fully stopped. Export of the USB pool was not possible. Thus, I restarted the system with the disk unplugged.

df -h /mnt/.ix-apps
showed Gsaenger-Pool/ix-apps without need for any further action.

Apps came up immediately from the right place.

Thanks for solving this issue!

Topic		Replies	Views
Apps service refuses to start no matter what after update TrueNAS General SCALE	42	358	July 6, 2026
Needs to restart docker after startup Apps and Virtualization	6	1087	February 1, 2026
Nextcloud gets stuck in state deploying / "no destination available" in syslog Apps and Virtualization	9	1254	May 10, 2024
After 25.10 upgrade, "Critical Failed to start docker for Applications: Docker service could not be started" TrueNAS General	13	1608	November 3, 2025
Truenas Scale 24.10.1 -- pool fails to load TrueNAS General	8	737	January 25, 2025

Apps: Validation Error / Datasets: Failed to load datasets - 500 Server Error

Related topics