Pool shows unavailable after update; need help to save data!

,

Hello, I have a TrueNas configuration with 4 x 4TB HDD in ZFS “raid” to 12 TB capacity. I’m not entirely sure what happened, but my SMB shares were disconnecting, and when I looked into the system console/ portal it showed the pool was Offline. Looking at the disks I can see all 4 disks but for Pool it shows NA. The only option I could see for the pool was to Disconnect/Export it, so I did export it but did not delete data or configuration data.

Now when I go to import the pool it shows up, but I get the following error when I import it:

'''
Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 985, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 352, in import_pool
    self.logger.error(
  File "libzfs.pyx", line 402, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 346, in import_pool
    zfs.import_pool(found, new_name or found.name, options, any_host=any_host)
  File "libzfs.pyx", line 1175, in libzfs.ZFS.import_pool
  File "libzfs.pyx", line 1203, in libzfs.ZFS.__import_pool
libzfs.ZFSException: one or more devices is currently unavailable
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run
    await self.future
  File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 391, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 981, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1462, in import_pool
    await self.middleware.call('zfs.pool.import_pool', pool['guid'], {
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1283, in call
    return await self._call(
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1248, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1254, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1173, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1156, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
libzfs.ZFSException: ('one or more devices is currently unavailable',)
'''

I have a knot in my stomach right now because I have so many family photos and I would really hate if I lost them all. Please any assistance would be helpful, or any way to recover the data on another TrueNas system.

I did apply the most recent TrueNas update last night.

Whats your hardware and how are your 4 HDDs connected ?
What does zpool status show when you enter that in the shell ?

The output of “zpool import” is

root@Ying[~]# zpool import
   pool: Warehouse
     id: 11438372291247230538
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C
 config:

        Warehouse                                       UNAVAIL  insufficient re
plicas
          raidz1-0                                      UNAVAIL  insufficient re
plicas
            gptid

the output of zpool status is

root@Ying[~]# zpool status
  pool: Prisons
 state: ONLINE
  scan: scrub repaired 0B in 00:02:38 with 0 errors on Sun Mar 17 00:02:38 2024
config:

        NAME                                          STATE     READ WRITE CKSUM
        Prisons                                       ONLINE       0     0     0
          gptid/502b49a0-54c4-11ee-a2fa-d050998691a3  ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:06 with 0 errors on Sun Jun 30 03:45:06 2024
config:
            [--rewind-to-checkpoint] <pool | id> [newpool]
root@Ying[~]# zpool import
   pool: Warehouse
     id: 11438372291247230538
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C
 config:

        Warehouse                                       UNAVAIL  insufficient replicas

Your outputs are not very easy to read but that could be my phone. Your Warehouse pool appears to have only one drive as a member and currently the system can’t see it hence the issue. The hope is that the drive hasn’t failed and it’s just a connection issue but you will need to investigate that. Worst case is that this drive has indeed died and as you have no fault tolerance the pool would be lost. Any particular reason you created two pools comprising of one disk in each? Wouldn’t a mirror of the two been better?

These are WD RED NAS Drives, I hope its just a failed connector , after running import - f several times, I see the output varies, sometimes 3 drives show online, sometimes two.


The drives are connected directly to the Sata connectors on the MOBO.

Ah ok that’s easier to read so you have four drives in a Z1. It may well be a connection error so you will need to investigate that.

Nope, not just your phone. They’re hard to decipher, and also (until just a moment ago) incomplete.

@Kemraj_Sobhai, you have several problems, starting with the fact that you put critical data on a RAIDZ1 pool, followed by the apparent fact that you don’t have a backup. After that is the fact that you seem to have one dead drive and another dodgy one, along with some corrupted data somewhere in your pool. Can you identify which drives belong to which gptids?

1 Like

I realize this is of no technical help to your current situation, however I wonder why would you get that far in using ZFS and TN without apparently caring to learn the basics of ZFS: 4 x 4TB HDD in ZFS “raid” means nothing… is that RAIDZ1, maybe RAIDZ2?

If you want to use something I would argue it’s common sense to learn how to use said thing. :slight_smile:

My point in this specific situation is to use proper wording to help us help you. For the same reason, why did you not post your FULL hardware configuration in your first message? Not attacking you here, but it seems to be a trend recently.

@kris @HoneyBadger pinging you as the most active iX’s employees here, in the old forum we had a few lines in the rules about how to make a proper post asking for help… could we have something similar, maybe in the FAQ - TrueNAS Community Forums or in a more visible place? Also, sorry if I pinged the wrong people but I don’t really know who is in charge of the decisions regarding the forum’s layout anymore.

You really need to post your hardware. Expand the info in Dan or Davvo’s signature to get an idea of what they are looking for.

write down a list of the drives serial numbers to gptid, etc.

I would suggest this info gets put in an initial email or notification to all users. I know I got a Greetings! from TrueNAS-Bot

1 Like

As an aside, please note, whatever type of pool you have configured, it is never ok to not do a backup if you want your data such as those photos. Never! You must do backups if you care that much about it.