Pool state is DEGRADED : needing to id drive to replace

TrueNAS General
,
1

Dear TrueNas Fellowship,

I am in need of your assistance to help me bring my server back to life.

My true nas server is running in a degraded state at present, and I get this error

Critical

Pool {ServerName} state is DEGRADED: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state.

The following devices are not healthy:

  • Disk 7831626758070777019 is UNAVAIL

Here I have included a snap shot of my zpool status showing that all my drives are online, and the one id that is unavailable is the issue.

admin@{ServerName}[~]$ sudo zpool status
[sudo] password for admin:
pool: {ServerName}
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using ‘zpool replace’.
see:
scan: scrub repaired 0B in 06:51:25 with 0 errors on Sun May 4 06:51:32 2025
config:

    NAME                                      STATE     READ WRITE CKSUM
    {ServerName}                             DEGRADED     0     0     0
      raidz1-0                                DEGRADED     0     0     0
        89281c87-59cd-4165-bb86-ce8a6664551a  ONLINE       0     0     0
        7831626758070777019                   UNAVAIL      0     0     0  wa                                                                                     s /dev/disk/by-partuuid/91c95d3b-6ba2-43fc-a56a-5d52aa418854
        57093934-5f88-4fcb-b1b7-d7892c179c65  ONLINE       0     0     0
        3c32f442-34ee-4aba-ae6d-b1b554bc08cf  ONLINE       0     0     0
        269dd20f-7756-44a0-9a76-42e029d1a91a  ONLINE       0     0     0
    dedup
      7f7a12d1-a01a-4283-aac7-8a6282886a93    ONLINE       0     0     0
    cache
      5ee090ff-e43b-4b64-b5f5-eda75b330cc5    ONLINE       0     0     0

errors: No known data errors

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:01:01 with 0 errors on Sat Jul 26 03:46:02 2025
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      sdc3      ONLINE       0     0     0

errors: No known data errors

I have read a few forums saying it could be an issue when one of the drives re-silvered, or possible bad sata connections.

If I could understand what drive is bad I can replace it as I have a spare drive for that purpose, but I get a feeling this issue is related to something else.

If there is anyone out there that has experienced this issue and has any advice on my next steps, I would hugely appreciate it.

thanks

2

You should be able to use the following commands to get disk serial number. You can use that to rule out the working ones from your list.

Do you have a spare hard drive port to to an in place replacement of the unavail drive? You would add the new disk physically, without removing the unavail and then choose the unavail in the GUI and then Replace option and point to new drive.
Otherwise, you just have to shuffle bad and good drives and then do a replace procedure.

lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME

sudo ZPOOL_SCRIPTS_AS_ROOT=1 zpool status -vLtsc lsblk,serial,smartx,smart
3

FYI, this configuration means that your deduplication drive is a single point of failure. Do you have a second identical/larger drive you can use to ATTACH to this device and provide redundancy?

4

In addition to the two commands that @SmallBarky has come up with, please let us know why you think you need dedup and how much memory your system has?

Once we have the output from these commands we will probably want you to run a couple of zdb commands to see what the labels say, and also likely some smartctl commands to check the drive diagnostic status.

In the mean time I would refrain from rebooting because the pool is currently working degraded and after a reboot it may have difficulties being imported.

5

Hello Small Barky,

Thank you for your reply. I am posting here the result from the first command line you sent me. I will now run the second one.

admin@{server name}[~]$ lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME
NAME LABEL MAJ:MIN TRAN ROTA ZONED VENDOR MODEL SERIAL PARTUUID START SIZE PA RTTYPENAME
sda 8:0 sata 0 none ATA SAMSU S206NX 128035676160
└─sda1
8:1 0 none 5ee090ff-e43b-4b64-b5f5-eda75b330cc5 2048 128034275328 So laris /usr & Apple ZFS
sdb 8:16 sata 1 none ATA WDC W 5MG1GB 14000519643136
└─sdb1
{server name}
8:17 1 none 57093934-5f88-4fcb-b1b7-d7892c179c65 4096 14000517529088 So laris /usr & Apple ZFS
sdc 8:32 sata 0 none ATA Samsu S39KNW 256060514304
├─sdc1
│ 8:33 0 none 2769aa24-912e-4096-9fd6-5521a1b3b2b6 4096 1048576 BI OS boot
├─sdc2
│ EFI 8:34 0 none a8591b9f-f687-4674-b288-39d8817c5d7f 6144 536870912 EF I System
├─sdc3
│ boot-pool
│ 8:35 0 none 201caab2-3aa6-458e-b423-73db31b84504 34609152 238340611584 So laris /usr & Apple ZFS
└─sdc4
8:36 0 none e81fb459-24e5-4998-a8a2-65b44a53da9e 1054720 17179869184 Li nux swap
sdd 8:48 sata 1 none ATA WDC W 6AGB36 14000519643136
└─sdd1
{server name}
8:49 1 none 3c32f442-34ee-4aba-ae6d-b1b554bc08cf 4096 14000517529088 So laris /usr & Apple ZFS
sde 8:64 sata 1 none ATA WDC W 5MG1E6 14000519643136
└─sde1
{server name}
8:65 1 none 269dd20f-7756-44a0-9a76-42e029d1a91a 4096 14000517529088 So laris /usr & Apple ZFS
sdf 8:80 sata 1 none ATA WDC W 5MG1EN 14000519643136
└─sdf1
{server name}
8:81 1 none 7f7a12d1-a01a-4283-aac7-8a6282886a93 2048 14000518577664 So laris /usr & Apple ZFS
sdg 8:96 sata 1 none ATA WDC W 6AGB7R 14000519643136
└─sdg1
{server name}
8:97 1 none 89281c87-59cd-4165-bb86-ce8a6664551a 4096 14000517529088 So

6

here is the results form the second command line.

pool: {ServerName}
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using ‘zpool replace’.
see:
scan: scrub repaired 0B in 06:51:25 with 0 errors on Sun May 4 06:51:32 2025
config:

    NAME                     STATE     READ WRITE CKSUM  SLOW    size  vendor                       model                                                                                                      serial  hours_on  pwr_cyc  health  realloc  cmd_to  temp  off_ucor  ata_err  rep_ucor  pend_sec
    {ServerName}             DEGRADED     0     0     0     -
      raidz1-0               DEGRADED     0     0     0     -
        sdg1                 ONLINE       0     0     0     0   12.7T     ATA       WDC WD142KFGX-68AFPN0         6A                                                                                           GB7R7U     10193       27  PASSED        0       -    46         0        -         -         0  (trim unsupported)
        7831626758070777019  UNAVAIL      0     0     0     0  was /dev/disk/by-partuuid/91c95d3b-6ba2-43fc-a56a-5d5                                                                                           2aa418854       -       -                           -                -         -        -       -        -       -     -                                                                                                    -        -         -         -  (trim unsupported)
        sdb1                 ONLINE       0     0     0     0   12.7T     ATA       WDC WD142KFGX-68AFPN0         5M                                                                                           G1GBVK      8635       24  PASSED        0       -    43         0        -         -         0  (trim unsupported)
        sdd1                 ONLINE       0     0     0     0   12.7T     ATA       WDC WD142KFGX-68AFPN0         6A                                                                                           GB365U     10314       27  PASSED        0       -    46         0        -         -         0  (trim unsupported)
        sde1                 ONLINE       0     0     0     0   12.7T     ATA       WDC WD142KFGX-68AFPN0         5M                                                                                           G1E6YK      8611       23  PASSED        0       -    47         0        -         -         0  (trim unsupported)
    dedup
      sdf1                   ONLINE       0     0     0     0   12.7T     ATA       WDC WD142KFGX-68AFPN0         5M                                                                                           G1EN9K      8168       22  PASSED        0       -    43         0       36         -         0  (trim unsupported)
    cache
      sda1                   ONLINE       0     0     0     0  119.2G     ATA  SAMSUNG MZNLN128HCGR-000H1   S206NXAG                                                                                           A06786     11582     3606  PASSED        0      16    36         0        -         -         -  (untrimmed)

errors: No known data errors

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:01:01 with 0 errors on Sat Jul 26 03:46:02 2025
config:

    NAME        STATE     READ WRITE CKSUM  SLOW    size  vendor                       model           serial  hours                                                                                           _on  pwr_cyc  health  realloc  cmd_to  temp  off_ucor  ata_err  rep_ucor  pend_sec
    boot-pool   ONLINE       0     0     0     -
      sdc3      ONLINE       0     0     0     0  238.5G     ATA   Samsung SSD 850 PRO 256GB  S39KNWAJ303212A     25                                                                                           536     3420  PASSED        0       -    29         -        -         -         -  (untrimmed)

errors: No known data errors

7

to add, I do not have a spare port unfortunatly, I can only take out and replace as it stands at the moment.

8

hey Protopia, To be honest, I had no idea what the dedup drive was, I don;t remember ever putting it into this setup. Should I replace that drive?

9

I also wanted to add. the system has 64gb of ram and 128gb cache drive. :slight_smile:

I have rebooted the system a couple of times since I saw the error, and the system has actually been in a degraded state for some months now. I will avoid rebooting though from this point as you have instructed.

10

Don’t touch the dedupe drive. It is critical to the pool once there. Is the dedup and L2ARC cache drive the same size? The L2ARC can be removed from a pool without damage, the dedupe would destroy the pool.

You need to copy and paste any CLI commands and results back to the forum using Preformatted Text </> on the toolbar or Ctrl-+e. I think you are just copy/pasting everything into the reply box and it is autoformatting. It’s hard to read. Take a look at the results for the second command and the box of text describing the pool setup. As you scroll the window to the right, it looks like a huge blank space is inserted before you see the results on the far right of the box.

11

AFAIK you cannot remove a dedup vDev - but as I understand it if you haven’t got a use case then it is likely to have a significant detrimental effect on the performance of your NAS.

So as a long term strategy you should aim to move your data off, destroy and recreate your pool and move your data back again. In the mean time, if this fails you will lose the entire pool’s worth of data.

You would also be advised to create the new pool as RAIDZ2 because a 5x 14TB RAIDZ1 is recommended to be RAIDZ2 because of the risk of a resilver (which you now need to do) causing a 2nd drive to fail causing you to lose all your data.

I see that your current drives have between 8,600 and 10,200 power-on hours = so they are only between just under 1 year and 1.25 years old and thus should still be under warranty.

It looks like the failing drive can no longer be seen by Linux as those listed by lsblk are all accounted for:

  • sda - cache
  • sdb - data
  • sdc - boot
  • sdd - data
  • sde - data
  • sdf - dedup
  • sdg - data

I have three possible theories that could have happened to this drive:

  1. It has failed completely - you will need a physical replacement and will need to resilver - and you can likely return the broken drive under warranty;
  2. It has lost connectivity with the MB (e.g. sata or power cable came off) - but you can bring this back to life by reconnecting it and then resilver;
  3. The dedup drive was until recently the missing data drive and somehow it has been reassigned as a dedup drive. You cannot remove this and you don’t have any spare slots to resilver so backup, recreate and restore is your only solution.

What you do next will depend on which of these is the case?

How many 14TB drives do you physically have in your system? And do you have a record of the serial numbers for them (without having to physically withdraw the drives to look)?

12

Hello Protopia, and Small Barky,

Apologize for not understanding the correct formatting for the results, I’m a little new to the forum, I will improve on that moving forward.

I think I will move the data off and rebuild the Nas. it is still relatively new, and can hopefully make all data safe within a few days. I will then rebuild the nas based on RAIDZ2 for the future. I might also order some new connectors just to be safe.

Thank you so much for your answers, I really appreciate it.

13

Copying your data elsewhere is a good safety action anyway, however I would advise that you work out what happened and why before you destroy the existing pool and recreate it.

How many 14TB drives do you physically have in your system? And did you take any actions before the pool got into this state that might explain it?