Help recovering pool (if possible)

Jafian · April 26, 2025, 9:14pm

Hi there, I was playing around with TrueNAS for a while and a hard drive cratered on me. I have replaced the drive, and my storage dashboard was telling me that I had 7 disks to import and all disks had my pool name and (exported) beside them. I tried to import the pool but the pool was not listed in the drop box.

Knowing just enough to be dangerous, I hit the export button to clear the settings (I did not select destroy data). The pool is no longer listed in my storage manager and now shows up as an option under the import pool menu.

Unfortunately, it runs into the following error:

File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1356, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1267, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1251, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EZFS_IO] Failed to import ‘Diskof1337’ pool: cannot import ‘Diskof1337’ as ‘Diskof1337’: I/O error

Anyways, I am sure I left out a bunch of pertinent info here, but if anyone has any suggestions I would appreciate it!

Protopia · April 26, 2025, 10:53pm

Please run the following shell commands and post the output here with the output from each command in a separate </> box:

lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID
sudo zpool status -v
sudo zpool import
lspci
sudo sas2flash -list
sudo sas3flash -list
sudo storcli show all

These will give us a starting point for diagnosis. In the mean time please do NOT make any further changes or reboot your system.

Jafian · April 26, 2025, 11:38pm

Thanks Protopia, I appreciate you taking the time to have a look into this for me.

lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID

NAME     MODEL                ROTA PTTYPE TYPE     START          SIZE PARTTYPENAME             PARTUUID
sda      ST3000DM001-1ER166      1 gpt    disk           3000592982016                          
└─sda1                           1 gpt    part      4096 3000590402048 Solaris /usr & Apple ZFS 2105541b-ec97-4100-b6fc-3bdae4fac837
sdb      ST3000DM001-9YN166      1 gpt    disk           3000592982016                          
└─sdb1                           1 gpt    part      4096 3000590402048 Solaris /usr & Apple ZFS 3cab4e4a-2cd9-40e6-847f-a7960b18c910
sdc      ST3000DM001-1ER166      1 gpt    disk           3000592982016                          
└─sdc1                           1 gpt    part      4096 3000590402048 Solaris /usr & Apple ZFS c98caffe-a6c1-4e2e-a318-a58efa7aacd7
sdd      OCZ-AGILITY3            0 gpt    disk            240057409536                          
├─sdd1                           0 gpt    part      4096       1048576 BIOS boot                7909da88-5309-4141-a091-05355f43b38b
├─sdd2                           0 gpt    part      6144     536870912 EFI System               770e3454-9448-45f8-b92f-5efbbce2fec9
├─sdd3                           0 gpt    part  34609152  222337506816 Solaris /usr & Apple ZFS 67c3c283-48f1-42ce-83c9-9ff58671387b
└─sdd4                           0 gpt    part   1054720   17179869184 Linux swap               2e741593-7902-4bd4-bc93-761573d09b0a
  └─sdd4                         0        crypt            17179869184                          
sde      WDC WD40PURX-64GVNY0    1 gpt    disk           4000787030016                          
├─sde1                           1 gpt    part        34     134217728 Microsoft reserved       62eed4f9-5d5e-4929-a407-dd2a31b0d5f9
└─sde2                           1 gpt    part    264192 4000650887168 Microsoft basic data     b3314700-6625-446b-9287-c3e96da7010e
sdf      ST3000DM001-1CH166      1 gpt    disk           3000592982016                          
└─sdf1                           1 gpt    part      4096 3000590402048 Solaris /usr & Apple ZFS 1a8ffb7f-40fe-48b5-a6c7-f5c426edb100
sdg      WDC WD30EFRX-68EUZN0    1 gpt    disk           3000592982016                          
└─sdg1                           1 gpt    part      4096 3000590402048 Solaris /usr & Apple ZFS ec63cdf7-16d7-4035-b1b6-63002dba1f42
sdh      WDC WD30EFRX-68EUZN0    1 gpt    disk           3000592982016                          
└─sdh1                           1 gpt    part      4096 3000590402048 Solaris /usr & Apple ZFS 7cbeea3b-3497-40e0-9070-e74d7e139e3e
sdi      SD/MMC                  1        disk                       0                          
sdj      Compact Flash           1        disk                       0                          
sdk      SM/xD-Picture           1        disk                       0                          
sdl      MS/MS-Pro               1        disk                       0

sudo zpool status -v

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:13 with 0 errors on Sat Apr 26 03:45:15 2025
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sdd3      ONLINE       0     0     0

errors: No known data errors

sudo zpool import

   pool: Diskof1337
     id: 1284357822605068817
  state: DEGRADED
status: One or more devices contains corrupted data.
 action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
 config:

        Diskof1337                                DEGRADED
          raidz2-0                                DEGRADED
            c98caffe-a6c1-4e2e-a318-a58efa7aacd7  ONLINE
            3cab4e4a-2cd9-40e6-847f-a7960b18c910  ONLINE
            e30b985b-ea3b-4b46-a941-1283c3a65e85  UNAVAIL
            2105541b-ec97-4100-b6fc-3bdae4fac837  ONLINE
            1a8ffb7f-40fe-48b5-a6c7-f5c426edb100  ONLINE
            ec63cdf7-16d7-4035-b1b6-63002dba1f42  ONLINE
            7cbeea3b-3497-40e0-9070-e74d7e139e3e  ONLINE

lspci

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0)
00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d0)
00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0)
00:1c.7 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 8 (rev d0)
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
00:1f.0 ISA bridge: Intel Corporation H97 Chipset LPC Controller
00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11)
04:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04)
05:00.0 Network controller: Ralink corp. RT5362 PCI 802.11n Wireless Network Adapter
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

sudo sas2flash -list

LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18) 
Copyright (c) 2008-2014 LSI Corporation. All rights reserved 

        No LSI SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.

sudo sas3flash -list

Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02) 
Copyright 2008-2017 Avago Technologies. All rights reserved.

        No Avago SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.

sudo storcli show all

CLI Version = 007.1504.0000.0000 June 22, 2020
Operating system = Linux 6.1.74-production+truenas
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = truenas
Operating System  = Linux 6.1.74-production+truenas

Protopia · April 27, 2025, 7:04am

Ok - some weird devices. /dev/sde which has (or had) Windows data on it. /dev/sdi-/dev/sdl???

And I see 6x 3TB disks whereas the pool should have 7 - the disk with partuuid e30b985b-ea3b-4b46-a941-1283c3a65e85 is missing (presumably failed).

So, you need to investigate whether that drive is spinning or not, and if it is spinning why it is not appearing in Linux (the power or SATA cables could have become loose for example).

In the mean time we can try to import the pool in a degraded state so you can keep using it whilst you resolve the issue with the broken hard drive. Try the following and post the output…

sudo zpool import -R /mnt Diskof1337

This will (hopefully) import it in Linux - but it will not appear in the UI - once you have fixed the degraded disk issue (using the CLI) we can export it and try to re-import it into TrueNAS. (Next time, don’t try random actions (like exporting) when you don’t know the consequences because a lot of times (like this one) it just complicates things.)

Jafian · April 27, 2025, 1:15pm

don’t try random actions (like exporting) when you don’t know the consequences because a lot of times (like this one) it just complicates things.)

Yep. Heard loud and clear.

/dev/sde which has (or had) Windows data on it.

This was just the drive I was hoping to replace the failed drive with.

/dev/sdi-/dev/sdl

USB SD/CF/etc reader in the case

investigate whether that drive is spinning or not, and if it is spinning why it is not appearing in Linux

Drive spins, the odd mechanical “chirp”, system will not boot with it plugged in. Don’t recall the exact step during boot that the system wouldn’t progress past.

sudo zpool import -R /mnt Diskof1337

cannot import 'Diskof1337': I/O error
        Destroy and re-create the pool from
        a backup source.

etorix · April 27, 2025, 1:43pm

This is a low end two-port controller.
How are the drives attached? Motherboard or add-in card?

A raidz2 pool not being imported with only one drive missing is a very bad sign.

You may try and report on (these won’t import anything)
sudo zpool import -R /mnt -fn Diskof1337
sudo zpool import -R /mnt -fFn Diskof1337
sudo zpool import -R /mnt -fFXn Diskof1337
but then it would probably be best to hear from @HoneyBadger before proceeding to any actual recovery attempt.

neofusion · April 27, 2025, 1:51pm

How would a port multiplier show up in lspci? Could this be going through one?

etorix · April 27, 2025, 1:57pm

A SATA port multiplier is not a PCI device, so I think it would not show up at all in lspci. If this controller, from an add-in card or built into the unspecified motherboard, just provides two ports it might be OK. But if it feeds more than two ports, “Houston, we’ve had a problem”…

neofusion · April 27, 2025, 1:59pm

That’s what I was thinking, so the port multiplier is essentially invisible in lspci and it could be that the this device has more than two ports.

Jafian · April 27, 2025, 2:38pm

sudo zpool import -R /mnt -fn Diskof1337

-n or -X only meaningful with -F

sudo zpool import -R /mnt -fFn Diskof1337
There was no output from this command, just back to the prompt.

etorix · April 27, 2025, 3:35pm

No output means that the pool may be imported with -fF. But wait for more expert opinion before proceeding further.

Protopia · April 27, 2025, 4:28pm

I agree that no output is a good sign - however the error message for -fn means that this was not actually attempted - so whilst we think that -fF may import we have no idea whether -f will import or not.

Personally I would suggest an actual import attempt with -f i.e.

sudo zpool import -R /mnt -f Diskof1337

It will either work or it won’t - as far as I know it won’t cause further issues and if it fails we should get some diagnostic info.

HoneyBadger · April 28, 2025, 2:47pm

No output in this case means no error, but it doesn’t mean that re-running without the n will be successful. The pool itself shows DEGRADED as expected but with only one disk UNAVAIL a raidz2 should import.

Try again with sudo zpool import -R /mnt -fF Diskof1337

That said, reviewing the thread, you’re using the somewhat-infamous Seagate Barracuda 3TB drives that don’t have a stellar reputation for reliabililty.

A sound described as a “chirp” sounds like mechanical failure to me, if it sounds like it’s repeatedly trying and failing to ramp the spindle up to full speed.

Jafian · April 28, 2025, 5:58pm

sudo zpool import -R /mnt -fF Diskof1337

admin@truenas[~]$ sudo zpool import -R /mnt -fF Diskof1337
[sudo] password for admin: 
cannot import 'Diskof1337': I/O error
        Destroy and re-create the pool from
        a backup source.

etorix · April 28, 2025, 6:10pm

Another case of label corruption?

HoneyBadger · April 28, 2025, 6:56pm

Try rewinding further with the added -X in there:

sudo zpool import -R /mnt -fFX Diskof1337

Jafian · April 28, 2025, 7:13pm

cannot import 'Diskof1337': one or more devices is currently unavailable

HoneyBadger · April 28, 2025, 7:22pm

I’d like to see

for disk in $(ls /dev/sd?1); sudo blkid --probe $disk

Jafian · April 28, 2025, 7:29pm

/dev/sda1: VERSION="5000" LABEL="Diskof1337" UUID="1284357822605068817" UUID_SUB="12672947030987444519" BLOCK_SIZE="4096" TYPE="zfs_member" USAGE="filesystem" PART_ENTRY_SCHEME="gpt" PART_ENTRY_UUID="2105541b-ec97-4100-b6fc-3bdae4fac837" PART_ENTRY_TYPE="6a898cc3-1dd2-11b2-99a6-080020736631" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="4096" PART_ENTRY_SIZE="5860528129" PART_ENTRY_DISK="8:0"
/dev/sdb1: VERSION="5000" LABEL="Diskof1337" UUID="1284357822605068817" UUID_SUB="6753057509992160284" BLOCK_SIZE="4096" TYPE="zfs_member" USAGE="filesystem" PART_ENTRY_SCHEME="gpt" PART_ENTRY_UUID="3cab4e4a-2cd9-40e6-847f-a7960b18c910" PART_ENTRY_TYPE="6a898cc3-1dd2-11b2-99a6-080020736631" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="4096" PART_ENTRY_SIZE="5860528129" PART_ENTRY_DISK="8:16"
/dev/sdc1: VERSION="5000" LABEL="Diskof1337" UUID="1284357822605068817" UUID_SUB="11498854745790703038" BLOCK_SIZE="4096" TYPE="zfs_member" USAGE="filesystem" PART_ENTRY_SCHEME="gpt" PART_ENTRY_UUID="c98caffe-a6c1-4e2e-a318-a58efa7aacd7" PART_ENTRY_TYPE="6a898cc3-1dd2-11b2-99a6-080020736631" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="4096" PART_ENTRY_SIZE="5860528129" PART_ENTRY_DISK="8:32"
/dev/sdd1: PART_ENTRY_SCHEME="gpt" PART_ENTRY_UUID="7909da88-5309-4141-a091-05355f43b38b" PART_ENTRY_TYPE="21686148-6449-6e6f-744e-656564454649" PART_ENTRY_FLAGS="0x4" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="4096" PART_ENTRY_SIZE="2048" PART_ENTRY_DISK="8:48"
/dev/sde1: PART_ENTRY_SCHEME="gpt" PART_ENTRY_NAME="Microsoft reserved partition" PART_ENTRY_UUID="62eed4f9-5d5e-4929-a407-dd2a31b0d5f9" PART_ENTRY_TYPE="e3c9e316-0b5c-4db8-817d-f92df00215ae" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="34" PART_ENTRY_SIZE="262144" PART_ENTRY_DISK="8:64"
/dev/sdf1: VERSION="5000" LABEL="Diskof1337" UUID="1284357822605068817" UUID_SUB="16023864051599864718" BLOCK_SIZE="4096" TYPE="zfs_member" USAGE="filesystem" PART_ENTRY_SCHEME="gpt" PART_ENTRY_UUID="1a8ffb7f-40fe-48b5-a6c7-f5c426edb100" PART_ENTRY_TYPE="6a898cc3-1dd2-11b2-99a6-080020736631" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="4096" PART_ENTRY_SIZE="5860528129" PART_ENTRY_DISK="8:80"
/dev/sdg1: VERSION="5000" LABEL="Diskof1337" UUID="1284357822605068817" UUID_SUB="13126095211415388040" BLOCK_SIZE="4096" TYPE="zfs_member" USAGE="filesystem" PART_ENTRY_SCHEME="gpt" PART_ENTRY_UUID="ec63cdf7-16d7-4035-b1b6-63002dba1f42" PART_ENTRY_TYPE="6a898cc3-1dd2-11b2-99a6-080020736631" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="4096" PART_ENTRY_SIZE="5860528129" PART_ENTRY_DISK="8:96"
/dev/sdh1: VERSION="5000" LABEL="Diskof1337" UUID="1284357822605068817" UUID_SUB="15166309430042581167" BLOCK_SIZE="4096" TYPE="zfs_member" USAGE="filesystem" PART_ENTRY_SCHEME="gpt" PART_ENTRY_UUID="7cbeea3b-3497-40e0-9070-e74d7e139e3e" PART_ENTRY_TYPE="6a898cc3-1dd2-11b2-99a6-080020736631" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="4096" PART_ENTRY_SIZE="5860528129" PART_ENTRY_DISK="8:112"

HoneyBadger · April 28, 2025, 7:37pm

So no ambiguous labels on the devices that would make it think they are missing and report the UNAVAIL status

This feels like an old TrueNAS installation, can you cat /etc/version for me please?