Hello everyone,
I am trying to migrate my existing Apps from the TrueNAS catalog located in the iXVolume dataset on my HDD pool (stockage) to my SSD dataset (apps). The new dataset is completely empty because the SSD was formatted just before.
I am using the “Choose a pool for Apps” migration button in the GUI. However, I am encountering the following issues:
-
After a few seconds, the UI freezes and the system is no longer reachable via ping.
-
I have run several tests, and even after waiting multiple hours, the network remains unreachable. I am forced to do a hard reboot every time.
My Analysis: It seems the error logs don’t explain why the migration fails initially, but they do explain why the network remains unreachable. After the migration fails, Docker tries to perform a DockerService.restore_backup. This restoration fails because there are snapshots on the new dataset preventing the deletion of the folder (filesystem has children).
Because of this failure, the script does not restart the Docker service, and the network never comes back up. What is confusing is that the migration goes from stockage to apps, and apps was empty in the first place, so I cannot delete these snapshots beforehand.
After the hard reboot, it looks like the replication actually worked because I can find my folders on the SSD pool. However, ix-apps is still mapped to stockage in ZFS, while the GUI shows the opposite (indicating the new pool is being used).
Here are the raw middlewared logs I was able to find:
[2026/04/10 14:58:55\] (WARNING) middlewared.\_loop_monitor_thread():1354 - Task seems blocked:\\n File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1555, in main\\n middleware.run()\\n File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1389, in run\\n self.loop.run_forever()\\n File "/usr/lib/python3.11/asyncio/base_events.py", line 608, in run_forever\\n self.\_run_once()\\n File "/usr/lib/python3.11/asyncio/base_events.py", line 1936, in \_run_once\\n handle.\_run()\\n File "/usr/lib/python3.11/asyncio/events.py", line 84, in \_run\\n self.\_context.run(self.\_callback, \*self.\_args)\\n File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 398, in process_method_call\\n app.send({\\n File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 65, in send\\n asyncio.run_coroutine_threadsafe(self.ws.send_str(data\_), self.middleware.loop)\\n File "/usr/lib/python3.11/asyncio/tasks.py", line 936, in run_coroutine_threadsafe\\n loop.call_soon_threadsafe(callback)\\n File "/usr/lib/python3.11/asyncio/base_events.py", line 813, in call_soon_threadsafe\\n self.\_write_to_self()\\n File "/usr/lib/python3.11/asyncio/selector_events.py", line 139, in \_write_to_self\\n csock.send(b'\\0')\\n
\[2026/04/10 15:08:56\] (ERROR) middlewared.reader():137 - Error in ShellWorkerThread.reader @cee:{"TNLOG": {"exception": "Traceback (most recent call last):\\n File \\"/usr/lib/python3/dist-packages/middlewared/apps/webshell_app.py\\", line 135, in reader\\n ).result()\\n ^^^^^^^^\\n File \\"/usr/lib/python3.11/concurrent/futures/\_base.py\\", line 456, in result\\n return self.\__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File \\"/usr/lib/python3.11/concurrent/futures/\_base.py\\", line 401, in \__get_result\\n raise self.\_exception\\n File \\"/usr/lib/python3/dist-packages/aiohttp/web_ws.py\\", line 426, in send_bytes\\n await self.\_writer.send_frame(data, WSMsgType.BINARY, compress=compress)\\n File \\"/usr/lib/python3/dist-packages/aiohttp/\_websocket/writer.py\\", line 72, in send_frame\\n raise ClientConnectionResetError(\\"Cannot write to closing transport\\")\\naiohttp.client_exceptions.ClientConnectionResetError: Cannot write to closing transport", "type": "PYTHON_EXCEPTION", "time": "2026-04-10 13:08:56.400575"}}
\[2026/04/10 15:16:48\] (ERROR) middlewared.reader():137 - Error in ShellWorkerThread.reader @cee:{"TNLOG": {"exception": "Traceback (most recent call last):\\n File \\"/usr/lib/python3/dist-packages/middlewared/apps/webshell_app.py\\", line 135, in reader\\n ).result()\\n ^^^^^^^^\\n File \\"/usr/lib/python3.11/concurrent/futures/\_base.py\\", line 456, in result\\n return self.\__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File \\"/usr/lib/python3.11/concurrent/futures/\_base.py\\", line 401, in \__get_result\\n raise self.\_exception\\n File \\"/usr/lib/python3/dist-packages/aiohttp/web_ws.py\\", line 426, in send_bytes\\n await self.\_writer.send_frame(data, WSMsgType.BINARY, compress=compress)\\n File \\"/usr/lib/python3/dist-packages/aiohttp/\_websocket/writer.py\\", line 72, in send_frame\\n raise ClientConnectionResetError(\\"Cannot write to closing transport\\")\\naiohttp.client_exceptions.ClientConnectionResetError: Cannot write to closing transport", "type": "PYTHON_EXCEPTION", "time": "2026-04-10 13:16:48.165716"}}
\[2026/04/10 15:59:43\] (INFO) ZFSSnapshotService.create():195 - Snapshot taken: stockage/ix-apps@ix-apps-migrate-2026-04-10_15-59
\[2026/04/10 15:59:52\] (DEBUG) middlewared.plugins.zettarepl.\_process_command_queue():254 - Running task <Replication Task 'job_848'>
\[2026/04/10 16:15:28\] (ERROR) middlewared.job.run():540 - Job <bound method DockerService.restore_backup of <middlewared.plugins.docker.restore_backup.DockerService object at 0x7f51e42cf290>> failed: CallError("Failed to delete dataset: cannot destroy 'apps/ix-apps/docker': filesystem has children\\nuse '-r' to destroy the following datasets:\\napps/ix-apps/docker@auto-2026-03-15_04-00\\napps/ix-apps/docker@auto-2026-03-16_04-00\\napps/ix-apps/docker@auto-2026-03-17_04-00\\napps/ix-apps/docker@auto-2026-03-18_04-00\\napps/ix-apps/docker@auto-2026-03-19_04-00\\napps/ix-apps/docker@auto-2026-03-20_04-00\\napps/ix-apps/docker@auto-2026-03-21_04-00\\napps/ix-apps/docker@auto-2026-03-22_04-00\\napps/ix-apps/docker@auto-2026-03-23_04-00\\napps/ix-apps/docker@auto-2026-03-24_04-00\\napps/ix-apps/docker@auto-2026-03-25_04-00\\napps/ix-apps/docker@auto-2026-03-26_04-00\\napps/ix-apps/docker@auto-2026-03-27_04-00\\napps/ix-apps/docker@auto-2026-03-28_04-00\\napps/ix-apps/docker@auto-2026-03-29_04-00\\napps/ix-apps/docker@auto-2026-03-30_04-00\\napps/ix-apps/docker@auto-2026-03-31_04-00\\napps/ix-apps/docker@auto-2026-04-01_04-00\\napps/ix-apps/docker@auto-2026-04-02_04-00\\napps/ix-apps/docker@auto-2026-04-03_04-00\\napps/ix-apps/docker@auto-2026-04-04_04-00\\napps/ix-apps/docker@auto-2026-04-05_04-00\\napps/ix-apps/docker@auto-2026-04-06_04-00\\napps/ix-apps/docker@auto-2026-04-07_04-00\\napps/ix-apps/docker@auto-2026-04-08_04-00\\napps/ix-apps/docker@auto-2026-04-09_04-00\\napps/ix-apps/docker@auto-2026-04-10_04-00", 14)
\[2026/04/10 16:15:28\] (ERROR) middlewared.job.run():540 - Job <bound method DockerService.do_update of <middlewared.plugins.docker.update.DockerService object at 0x7f51b6d07d90>> failed: CallError('Docker service is not running')
\[2026/04/10 16:41:59\] (INFO) middlewared.\__init_\_():109 - Starting TrueNAS-25.10.1 middleware
Are there other logs I can provide to help identify the issue?
Thank you.
