Problem after power failure

I had a power outage, and when I started it, the apps wouldn’t start and stuck in a loop. After some research, I was told that unsetting the pool would solve the problem. This worked for a while. But today, after another power outage, it stopped working. When I tried to unset the pool, I got this message…
I don’t understand English very well, my language is Brazilian Portuguese. I use Google Translate to understand the language.

Error: concurrent.futures.process.RemoteTraceback:
“”"
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs
/dataset_actions.py", line 70, in umount
with libzfs.ZFS() as zfs:
File “libzfs.pyx”, line 534, in libzfs.ZFS.exit
File “/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py”, line 72, in umount
dataset.umount(force=options[‘force’])
File “libzfs.pyx”, line 4321, in libzfs.ZFSDataset.umount
libzfs.ZFSException: cannot unmount ‘/mnt/.ix-apps/docker’: pool or dataset is busy

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.11/concurrent/futures/process.py”, line 261, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 116, in main_worker
res = MIDDLEWARE._run(*call_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 47, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 41, in call
return methodobj(*params)
^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 178, in nf
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs
/dataset_actions.py", line 76, in umount
raise CallError(f’Failed to umount dataset: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to umount dataset: cannot unmount ‘/mnt/.ix-apps/docker’: pool or dataset is busy
“”"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 515, in run
await self.future
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 560, in __run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py”, line 93, in wrapped
result = await func(*args)
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/docker/update.py”, line 117, in do_update
catalog_sync_job = await self.middleware.call(‘docker.fs_manage.umount’)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1000, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 715, in _call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/docker/fs_manage.py”, line 33, in umount
return await self.common_func(False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/docker/fs_manage.py”, line 20, in common_func
await self.middleware.call(‘zfs.dataset.umount’, docker_ds, {‘force’: True})
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1000, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 723, in _call
return await self._call_worker(name, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 729, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 635, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 619, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed to umount dataset: cannot unmount ‘/mnt/.ix-apps/docker’: pool or dataset is busy

UPSes are cheap…

1 Like

Adding an UPS now will not solve the existing problem.

@LeinadSOFT We need a better description of your system, and of the problem.
What is the server? Which version of TrueNAS? How many drives? How many pools?
If you have error messages from the middleware, the NAS has not completely stopped working. I understand that apps are not working. Is it still working as a NAS and sharing files?

1 Like

True but a Truenas system definitely needs a UPS. Linux and Truenas are not made for pull plug type of power failures. I can testify…

1 Like

ZFS was specifically designed to work WITHOUT a UPS:

That said, we have started to see a lot more ZFS pools that won’t import after power loss than we did years ago. Perhaps Linux is doing enough things different from FreeBSD to cause ZFS problems.

2 Likes