TrueNAS system dropped vdevs

[Parsivan]

hello everyone

I was watching movies from my plex server where the movie paused and started loading then it said no connection to server. when i looked the pool had dropped vdevs. then i exported the pool and tried importing see if that helps but i get an error.

Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py", line 227, in import_pool
    zfs.import_pool(found, pool_name, properties, missing_log=missing_log, any_host=any_host)
  File "libzfs.pyx", line 1369, in libzfs.ZFS.import_pool
  File "libzfs.pyx", line 1397, in libzfs.ZFS.__import_pool
libzfs.ZFSException: cannot import 'Alexius_Repository' as 'Alexius_Repository': I/O error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 191, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py", line 207, in import_pool
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 529, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py", line 231, in import_pool
    raise CallError(f'Failed to import {pool_name!r} pool: {e}', e.code)
middlewared.service_exception.CallError: [EZFS_IO] Failed to import 'Alexius_Repository' pool: cannot import 'Alexius_Repository' as 'Alexius_Repository': I/O error
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 511, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 47, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/import_pool.py", line 113, in import_pool
    await self.middleware.call('zfs.pool.import_pool', guid, opts, any_host, use_cachefile, new_name)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1425, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1431, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1337, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EZFS_IO] Failed to import 'Alexius_Repository' pool: cannot import 'Alexius_Repository' as 'Alexius_Repository': I/O error

i had another pool that had problems that I could import read-only but it also spits out similar errors like processor.py, main.py etc.(i have some Minecraft world here that i would copy the maps from)

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1428, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 191, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 53, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/filesystem.py", line 430, in stat
    raise CallError(f'Path {_path} not found', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Path /MC_Damien not found

running Truenas scale dragonfish 24.04 (the pools were not upgraded to use the zfs tags)

My NAS has commodity hardware

CPU: i9900k

RAM:16GB 3200Mhz corsair

MB: ASUS

BOOT: 2 * 1TB M.2

HDD: 3 * 4TBs in raidz1

I have a feeling that it’s not the pools that have a problem. I even changed the sata cables and tried different ports. I ran tests on the HDDs and never had a problem. the pools just dropped randomly. is there a way to fix it? I’m not that experienced with TrueNAS and linux

[Protopia]

1. 3x 4TB in RAIDZ2? You can do that in EE but AFAIK in Dragonfish and earlier you need at least 4 drives to create a RAIDZ2 pool.

2. I have no idea what you mean by “dropped” vDevs. Please provide screenshots of UI or copy & paste of a sudo zpool status -v and lsblk both before and after the pool has “dropped” vDevs.

[Parsivan]

Thank you for pointing that out yes I wrote wrong it is in raidz1 (i fixed the text)

well it said that the pool didn’t have any vdevs and it allowed me to make a new vdev from the 3 HDDs I had but that would destroy the data on them. so I exported the pool and when i try to import i get the first error message.

so rn the pool is exported. i sadly don’t have any data taken from before the vdevs dropped. this happened a few months back I’ve been busy before. there’s a chance i might not remember everything correctly.

the marked drives are the HDDs for the mentioned pool(Alexius). Damien was used for Minecraft and apps before. the rest are not important. running smart test on the hard drives rn

[Protopia]

Well, I am not a ZFS expert, but I suspect that if you cannot import the pool through either the UI or the command line, then the pool is beyond recovery and your only option is to clear the disks, build the RAIDZ1 pool afresh and restore your data from backup (hopefully you have one).

If you want to try to get it back online one final time before you rebuild it, then I think you need to switch to using the command line. (If we can get it working using the command line, then you should be able to export it using the command line and import it using the UI and be back to where you were before. But that is a big “if”…)

Please try the following shell commands and copy and paste the results here: (They probably won’t work - but hopefully the command line results will tell us something extra that we don’t currently know.)

• sudo zpool import
• sudo zpool import Alexius_Repository

Note: Please do NOT take any other actions that might make a potentially recoverable pool more difficult or impossible to recover, or make it more difficult to get diagnostics.

[Parsivan]

i guess i don’t have any other option! but one thing that i don’t understand is why the other pool “MC_Damien” doesn’t mount!? and has similar error. The pool got corrupted so i only could mount it read-only now it does get imported but it can’t mount !!

[dan]

What about the output of zpool import?

[Parsivan]

@Parsivan

Here is the output. I’m going to recreate all my pools again!

[dan]

@Parsivan

Here is the output.

No, it isn’t. zpool import is not the same command as zpool import Alexius_Repository, which is why I asked (again, after Protopia had already asked once) for the output of the former command. Isn’t this obvious?

[Protopia]

@Parsivan

I’m going to recreate all my pools again!

If you have a backup, and you want it up again as quickly as possible, this is probably the best action to achieve that. HOWEVER…

If you don’t know why this pool went offline, and why your other pool went offline, then IMO it is highly likely to happen again.

NOTE: These things don’t happen on most hardware - if they have happened to you not only once BUT TWICE, it is indicative of something badly wrong with your system. But it is your NAS and your data - all we can do is give advice, which you are free to adopt or ignore and to take your own decisions (and live with any consequences). If you bypass diagnostics, recreate your pool and restore from backup, but then it happens again, that is something that you will be living with.

This is why I am suggesting that you take the time and work with us to diagnose the cause. And this is why I cautioned you NOT to take any actions that might make things worse and / or prevent diagnostics.