Hello,
I had a raidz1 pool with 3 8tb drives that I was expanding and was taking longer than normal (over a week). At some point the system suffered a power loss. After powering the system back on the expansion continued but stalled saying it would take several more days. At some point during this one of the disks entered a faulted state with too many READ and CKSUM errors. So I added another disk to start resilvering the bad one, however that also stalled after around 3 days. I rebooted it on the advice given to others online, which allowed it to continue with no errors, it stalled again after about a day, rebooted one more time and it made a lot of progress, but the next morning I noticed multiple errors in the console, unfortunately I forgot to screenshot them but it was something with mpt2sas_cm0 IO. When I noticed those I also had more drives faulted at this point with a lot of data errors.
At this point I started to think it wasn’t a drive fault but a HBA or backplane issue, so I decided to connect all the drives directly to the motherboard, however I’m unable to import the array and get the error cannot import 'Tank': I/O error Destroy and re-create the pool from a backup source. .
Once I saw that I started trying to import the pool by manually specifying individual disks to see if I could get it to import at all, but that didn’t work. I then ran zdb -l /dev/disk/by-id/ata-<disk> to verify everything looked ok on all the disks and it looked good to me. At this point I’m out of ideas of where to go from here. Any direction or tips of what to do would be greatly appreciated!
here is the full error.
concurrent.futures.process._RemoteTraceback:
“”"
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py”, line 231, in import_pool
zfs.import_pool(found, pool_name, properties, missing_log=missing_log, any_host=any_host)
File “libzfs.pyx”, line 1374, in libzfs.ZFS.import_pool
File “libzfs.pyx”, line 1402, in libzfs.ZFS.__import_pool
libzfs.ZFSException: cannot import ‘Tank’ as ‘Tank’: I/O error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/lib/python3.11/concurrent/futures/process.py”, line 261, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 112, in main_worker
res = MIDDLEWARE._run(*call_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 46, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 34, in _call
with Client(f’ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock’, py_exceptions=True) as c:
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 40, in _call
return methodobj(*params)
^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 183, in nf
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py”, line 211, in import_pool
with libzfs.ZFS() as zfs:
File “libzfs.pyx”, line 534, in libzfs.ZFS.exit
File “/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py”, line 235, in import_pool
raise CallError(f’Failed to import {pool_name!r} pool: {e}', e.code)
middlewared.service_exception.CallError: [EZFS_IO] Failed to import ‘Tank’ pool: cannot import ‘Tank’ as ‘Tank’: I/O error
“”"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 509, in run
await self.future
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 554, in __run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 179, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 49, in nf
res = await f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/pool_/import_pool.py”, line 114, in import_pool
await self.middleware.call(‘zfs.pool.import_pool’, guid, opts, any_host, use_cachefile, new_name)
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1629, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1468, in _call
return await self._call_worker(name, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1474, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1380, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1364, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EZFS_IO] Failed to import ‘Tank’ pool: cannot import ‘Tank’ as ‘Tank’: I/O error