TrueNAS ZFS-2 Pool Offline After Drive Replacement – Help!

TrueNAS General
, ,
1

Hello everyone,

I recently encountered a issue while replacing a failing drive on my TrueNAS server, and I need your help, hopefully I did not lose all my data.

Steps I Took:

  1. Shut down the system, physically removed the failing drive, installed a new one, then powered it back on. I did not click “Detach” before removing the failed drive.
  2. In the TrueNAS software, I went to my vdev and selected “Replace” for the removed drive.
  3. The replacement process kicked off and seemed to be running fine, so I left it for a few days.

When I checked back, I found that all my drives were unassigned, the vdev is offline, and I have no useful error messages in my notifications to explain what happened.

Is there a way to recover my pool and get my drives reassigned

2

Also, not sure why it says 6 unassigned drives, when I have 5 drives.
I have 1 SATA SSD for OS, 1 Optane 16GB SLOG , and 5x HDDs 18TB.

3

The 6th drive is the failed drive you physically removed. Since the replacement has not completed the system still knows about it.

Please describe your hardware, which version of TrueNAS you are currently running, and which version (if different) you were running when you created this zpool. If you started with a different version, which one did you start with and what upgrades in place did you do or did you do fresh installations and imports.

I am asking about versions because we have seen a number of issues with zpools when they were created under an older version.

In terms of figuring out why your 5 HDD are being seen as available, please run the folloiwng in a System → Shell and copy/paste the results here (using the preformatted text style. You can get the device names from the Storage → Disks page.

sudo /sbin/zdb -l /dev/sd<n> and sudo /sbin/zdb -l /dev/sd<n>1

Note that I am looking for ZFS labels on both the entire disk and partition 1.

4

I am currently running TrueNAS-23.10.1. I’m unsure which version I used to create the pool originally, but I built the system and pool in mid-December 2023 by downloading the latest available version of TrueNAS Scale from the website.

Automatic updates are enabled, and I have never manually updated the system.

When running the following commands for each of sda, sdb, sdc, sdd, and sde:
sudo /sbin/zdb -l /dev/sd<n> and sudo /sbin/zdb -l /dev/sd<n>1

I receive the following errors:

failed to unpack label 1
failed to unpack label 2
failed to unpack label 3
failed to unpack label 4
5

Some screenshots from Dashboard and Disks
ibb.co/8nFf9jZd
ibb.co/rG85N4N5

6

Wow, that is an old version. I do not recommend updating until we get your zpool problem fixed, but as soon as we do you should upgrade to the latest release (25.04.1), but you may have to do that as a set of steps, you may not be able to jump straight to 25.04.1.

Disk partitioning changed over time and the commands I gave you assumed a recent release.

Please try the following, and post the full command and result together, unless you get another failed to unpack message, that just means there is no ZFS label on that partition.

/sbin/zdb -l /dev/sda1
/sbin/zdb -l /dev/sda2
/sbin/zdb -l /dev/sda3

One of those three should return a ZFS label, when you find out which, run against your other 4 devices as well (note the partition number at the end).

7

Shows the zpool as exported. But …

Shows the zpool as imported but with no devices.

I think the safest path is to EXPORT the zpool, but do not destroy the data.

Then you can try to import the zpool.

8

do I keep this option checked during export:
Delete saved configurations from TrueNAS?

9

I did export without selecting these:

  • Destroy data on this pool?
  • Delete saved configurations from TrueNAS?

I got and Error:

Error Importing pool
[EZFS_BADDEV] Failed to import 'Murphy' pool: cannot import 'Murphy' as 'Murphy': one or more devices is currently unavailable

More info:

Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py", line 227, in import_pool
    zfs.import_pool(found, pool_name, properties, missing_log=missing_log, any_host=any_host)
  File "libzfs.pyx", line 1369, in libzfs.ZFS.import_pool
  File "libzfs.pyx", line 1397, in libzfs.ZFS.__import_pool
libzfs.ZFSException: cannot import 'Murphy' as 'Murphy': one or more devices is currently unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 181, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py", line 207, in import_pool
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 529, in libzfs.ZFS._exit_
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py", line 231, in import_pool
    raise CallError(f'Failed to import {pool_name!r} pool: {e}', e.code)
middlewared.service_exception.CallError: [EZFS_BADDEV] Failed to import 'Murphy' pool: cannot import 'Murphy' as 'Murphy': one or more devices is currently unavailable
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 427, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 465, in __run_body
    rv = await self.method(*([self] + args))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/import_pool.py", line 113, in import_pool
    await self.middleware.call('zfs.pool.import_pool', guid, opts, any_host, use_cachefile, new_name)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1399, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1350, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1356, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1267, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EZFS_BADDEV] Failed to import 'Murphy' pool: cannot import 'Murphy' as 'Murphy': one or more devices is currently unavailable
10

Please run the following commands to give us detailed diagnostic information and post the results of each ion a separate </> box:

  • lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME
  • sudo zpool status -v
  • sudo zpool import
  • lspci
  • sudo sas2flash -list
  • sudo sas3flash -list
  • sudo storcli show all
11
  • lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME
lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME
NAME LABEL MAJ:MIN TRAN   ROTA ZONED VENDOR MODEL SERIAL PARTUUID                                START           SIZE PARTTYPENAME
sda          8:0   sata      1 none  ATA    WDC W 4YHZ03                                               18000207937536 
└─sda1
     Murphy
             8:1             1 none                      355b7bd2-2d89-4789-b13b-81470e7011f4     4096 18000204792320 Solaris /usr & Apple ZFS
sdb          8:16  sata      1 none  ATA    WDC W 4BKTJY                                               18000207937536 
└─sdb1
     Murphy
             8:17            1 none                      8821a923-8f37-403e-b398-7b142675292e     4096 18000204792320 Solaris /usr & Apple ZFS
sdc          8:32  sata      1 none  ATA    WDC W 6RGGDR                                               18000207937536 
└─sdc1
     Murphy
             8:33            1 none                      fe2af012-d601-45b8-a2ee-69807e123191     4096 18000204792320 Solaris /usr & Apple ZFS
sdd          8:48  sata      1 none  ATA    WDC W 2JJWLJ                                               18000207937536 
└─sdd1
     Murphy
             8:49            1 none                      c6f49e05-23f1-44e7-85d9-2992698893a5     4096 18000204792320 Solaris /usr & Apple ZFS
sde          8:64  sata      1 none  ATA    WDC W 2JJZGP                                               18000207937536 
└─sde1
     Murphy
             8:65            1 none                      3435f367-3b33-4f01-8840-1290d5fc86da     4096 18000204792320 Solaris /usr & Apple ZFS
nvme1n1
           259:0   nvme      0 none         Patri P300ED                                                 128035676160 
├─nvme1n1p1
│          259:1   nvme      0 none                      6ba577a1-c84d-44a1-bdb6-63a94cd35b66     4096        1048576 BIOS boot
├─nvme1n1p2
│    EFI   259:2   nvme      0 none                      f7fcfa97-bdb0-44b4-bdbe-9dd5241b135d     6144      536870912 EFI System
├─nvme1n1p3
│    boot-pool
│          259:3   nvme      0 none                      a210b147-e4e8-4410-9cfb-68cda6b9459e 34609152   110315773440 Solaris /usr & Apple ZFS
└─nvme1n1p4
           259:4   nvme      0 none                      47fa58db-0b43-48de-b7e7-e563ac24a7d5  1054720    17179869184 Linux swap
  └─nvme1n1p4
           253:0             0 none                                                                       17179869184 
nvme0n1
           259:5   nvme      0 none         INTEL PHBT82                                                  14403239936 
└─nvme0n1p1
     Murphy
           259:6   nvme      0 none                      a479fe8b-ff6b-418e-82c4-a05f9c0da709     4096    14400094720 Solaris /usr & Apple ZFS
  • sudo zpool status -v
admin@truenas[~]$ sudo zpool status -v
  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:31 with 0 errors on Mon Jun  9 03:45:33 2025
config:

        NAME         STATE     READ WRITE CKSUM
        boot-pool    ONLINE       0     0     0
          nvme1n1p3  ONLINE       0     0     0

errors: No known data errors
  • sudo zpool import
admin@truenas[~]$ sudo zpool import   
   pool: Murphy
     id: 5933952339070335385
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        Murphy                                    ONLINE
          raidz2-0                                ONLINE
            3435f367-3b33-4f01-8840-1290d5fc86da  ONLINE
            fe2af012-d601-45b8-a2ee-69807e123191  ONLINE
            8821a923-8f37-403e-b398-7b142675292e  ONLINE
            c6f49e05-23f1-44e7-85d9-2992698893a5  ONLINE
            355b7bd2-2d89-4789-b13b-81470e7011f4  ONLINE
        logs
          a479fe8b-ff6b-418e-82c4-a05f9c0da709    ONLINE
admin@truenas[~]$
  • lspci
admin@truenas[~]$ lspci
00:00.0 Host bridge: Intel Corporation 10th Gen Core Processor Host Bridge/DRAM Registers (rev 03)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 03)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:14.0 USB controller: Intel Corporation Comet Lake PCH-V USB Controller
00:14.2 Signal processing controller: Intel Corporation Comet Lake PCH-V Thermal Subsystem
00:16.0 Communication controller: Intel Corporation Comet Lake PCH-V HECI Controller
00:17.0 SATA controller: Intel Corporation 400 Series Chipset Family SATA AHCI Controller
00:1b.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #21 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Device a393 (rev f0)
00:1c.4 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #05 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port 9 (rev f0)
00:1d.2 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port 11 (rev f0)
00:1f.0 ISA bridge: Intel Corporation B460 Chipset LPC/eSPI Controller
00:1f.2 Memory controller: Intel Corporation Cannon Lake PCH Power Management Controller
00:1f.3 Audio device: Intel Corporation Comet Lake PCH-V cAVS
00:1f.4 SMBus: Intel Corporation Comet Lake PCH-V SMBus Host Controller
01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P1000] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
02:00.0 Non-Volatile memory controller: Intel Corporation NVMe Optane Memory Series
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 04)
04:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (rev 01)
06:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller
admin@truenas[~]$
  • sudo sas2flash -list
admin@truenas[~]$ sudo sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18) 
Copyright (c) 2008-2014 LSI Corporation. All rights reserved 

        No LSI SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.
admin@truenas[~]$
  • sudo sas3flash -list
admin@truenas[~]$ sudo sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02) 
Copyright 2008-2017 Avago Technologies. All rights reserved.

        No Avago SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.
admin@truenas[~]$
  • sudo storcli show all
admin@truenas[~]$ sudo storcli show all
CLI Version = 007.1504.0000.0000 June 22, 2020
Operating system = Linux 6.1.63-production+truenas
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = truenas
Operating System  = Linux 6.1.63-production+truenas


admin@truenas[~]$
12

OK, I suspected we would have this issue.

Try to import via the GUI and let us know what happens. It may complain about duplicate names.

13

Thanks for these results which all seem fine.

Your partitions all exist with the correct partuuids, and your controllers are all AHCI.

It also tells us that that ZFS partitions are /dev/sda1 etc. so it is kind of worrying that sudo zdb -l /dev/sda1 etc. results in failed to unpack label N.

Can you confirm that you are running TrueNAS bare metal and not virtualised under e.g. Proxmox?

Before suggesting potential further actions, I suggest we wait to see what the result of @PK1048 Paul’s suggestion is?

14

that error I posted is when I imported through the GUI after exporting as you instructed, I think if I read it correctly, it did complain about duplicate names

15

Yes it’s running as vanilla bare metal as it gets.

16

Sorry, that was unlcear as you said you did the export but never mentioned trying the import.

I do not think the GUI will permit you to import by UUID, which is the fix for duplicate names.

Since the system is in an inconsistent state (part of this thinks the zpool is imported and part does not), I think the best next step may be to reboot and see if that clears the inconsistency. @Protopia , what do think about that approach?

17

Np! yeah forgot to mention sorry. Also, I double-checked not sure if it complained about duplicate names, just that “one or more devices is currently unavailable”. I ran some zpool commands to see if I can get more info on which device is not available, but nothing showed up.

Restarting the system did not work, I just tried it a couple of times.

Also if this is useful here’s more commands:

  • sudo dmesg | grep ZFS
tadmin@truenas[~]$ sudo dmesg | grep ZFS
[    0.000000] Command line: BOOT_IMAGE=/ROOT/23.10.1@/boot/vmlinuz-6.1.63-production+truenas root=ZFS=boot-pool/ROOT/23.10.1 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
[    0.020732] Kernel command line: BOOT_IMAGE=/ROOT/23.10.1@/boot/vmlinuz-6.1.63-production+truenas root=ZFS=boot-pool/ROOT/23.10.1 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
[    3.959177] ZFS: Loaded module v2.2.2-1, ZFS pool version 5000, ZFS filesystem version 5
admin@truenas[~]$
  • ls /dev/disk/by-id/
admin@truenas[~]$ ls /dev/disk/by-id/
ata-WDC_WD181KFGX-68AFPN0_2JJWLJXL
ata-WDC_WD181KFGX-68AFPN0_2JJWLJXL-part1
ata-WDC_WD181KFGX-68AFPN0_2JJZGPTL
ata-WDC_WD181KFGX-68AFPN0_2JJZGPTL-part1
ata-WDC_WD181KFGX-68AFPN0_4BKTJYRZ
ata-WDC_WD181KFGX-68AFPN0_4BKTJYRZ-part1
ata-WDC_WD181KFGX-68AFPN0_4YHZ036P
ata-WDC_WD181KFGX-68AFPN0_4YHZ036P-part1
ata-WDC_WD181KFGX-68AFPN0_6RGGDRJU
ata-WDC_WD181KFGX-68AFPN0_6RGGDRJU-part1
dm-name-nvme1n1p4
dm-uuid-CRYPT-PLAIN-nvme1n1p4
nvme-INTEL_MEMPEK1J016GA_PHBT8255037U016N
nvme-INTEL_MEMPEK1J016GA_PHBT8255037U016N-part1
nvme-Patriot_M.2_P300_128GB_P300EDBB23091303757
nvme-Patriot_M.2_P300_128GB_P300EDBB23091303757-part1
nvme-Patriot_M.2_P300_128GB_P300EDBB23091303757-part2
nvme-Patriot_M.2_P300_128GB_P300EDBB23091303757-part3
nvme-Patriot_M.2_P300_128GB_P300EDBB23091303757-part4
nvme-eui.5cd2e4321a940100
nvme-eui.5cd2e4321a940100-part1
nvme-nvme.1e4b-50333030454442423233303931333033373537-50617472696f74204d2e322050333030203132384742-00000001
nvme-nvme.1e4b-50333030454442423233303931333033373537-50617472696f74204d2e322050333030203132384742-00000001-part1
nvme-nvme.1e4b-50333030454442423233303931333033373537-50617472696f74204d2e322050333030203132384742-00000001-part2
nvme-nvme.1e4b-50333030454442423233303931333033373537-50617472696f74204d2e322050333030203132384742-00000001-part3
nvme-nvme.1e4b-50333030454442423233303931333033373537-50617472696f74204d2e322050333030203132384742-00000001-part4
wwn-0x5000cca2abe8baa5
wwn-0x5000cca2abe8baa5-part1
wwn-0x5000cca2abea0955
wwn-0x5000cca2abea0955-part1
wwn-0x5000cca2b6f56f14
wwn-0x5000cca2b6f56f14-part1
wwn-0x5000cca2bbdbbb06
wwn-0x5000cca2bbdbbb06-part1
wwn-0x5000cca2ebc68cf8
wwn-0x5000cca2ebc68cf8-part1
admin@truenas[~]$
18

@PK1048

Since you are an iX employee, I will defer to your more expert knowledge, but it seems to me that there are two options:

  1. Reboot and see if either: A) the pool comes back imported in both Linux and the UI; or B) the pool is exported in both Linux and the UI. @NightWatcher says he has rebooted several times on their own initiative (which they probably should not do as a possible change of device names may complicate things) but hasn’t said whether either of these states is achieved or whether it remains with the pool exported in Linux and imported in the UI.

  2. Try to import the pool from the command line.

I would personally have gone for option 2. but as I say, you probably have more expert knowledge than I do.

So at this point, I think OP needs to advise what the status is and then we can advise what actions to take next.

19

Previously using GUI, I exported the pool.
Then after I rebooted the pool was not imported automatically neither in GUI, nor checking using the Shell.
I tried importing the pool in GUI and Shell after using the following commands:

sudo zpool import 5933952339070335385
sudo zpool import -d /dev/disk/by-id 5933952339070335385

and I got:
cannot import 'Murphy': one or more devices is currently unavailable

I did not try a force import nor clearing cache as those are risky I think.

20

I believe @NightWatcher said that a reboot did not change the situation, but I await confirmation of that.

The command line zpool status did not see the bad zpool as imported, so I think I agree that the next step would be a command line zpool import -f 5933952339070335385 but I think I would like @HoneyBadger to weigh in.

If that works I would then zpool export and try to import from the GUI to get the TrueNAS config aligned.

@NightWatcher Please confirm that after reboot the GUI thinks the zpool is imported but faulted but the disks show as an exported zpool. Please do not take any action until after confirming this and we hear from @HoneyBadger .