Pool degraded, 2 drives in removed state

TrueNAS General
1

While I was on vacation, my NAS went down, showing “CAM status: CCB request completed with an error. Retrying” on the console repeatedly. When I got back I rebooted it, and it’s still showing those errors but the NAS is up and functional. However, the pool is showing “degraded,” and I got this error:

< Pool deadpool state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.
The following devices are not healthy:

  • Disk WDC WD30EFZX-68AWUN0 WD-WX22D61KD0U5 is REMOVED
  • Disk WDC WD30EFZX-68AWUN0 WD-WX42D51NKLAL is REMOVED

2024-09-26 10:18:33 PM (America/New_York)/>

FWIW I have never seen a SMART test fail, and I run them regularly. I am running TrueNAS Core 13-U6.2 with six WD Red Plus 3TB drives. It’s been up and running for maybe a year? I have a hard time believing two drives have a hardware failure, although I suppose it’s possible.

Not sure it’s related, but a few months ago I added a 12TB external USB backup drive, and while it’s “working,” it has reported it as Degraded since day 1 and says there are data errors now. I’ve been thinking of just removing that, wiping it and starting over, but then the issue with the two main data drives popped up.

Should I bring them back online and see what happens?

2

Sh!t happens, and multiple drive failures are not unheard of, especially if the first failure was not addressed on time. SMART tests are better than nothing, but are not guaranteed to catch all possible issues.
The good point here is that the pool is “only” degraded. Raidz2? Raidz3? Stripe of mirrors?

The backup drive may have developped issues due to unreliable USB connection.

You should definitively investigate the condition of the removed drives (still on? SMART report?), consider backing up the pool anew, and check the backup. In no particular order, as I do not know what is most critical.

3

List you hardware please.

Also how are the disks connected to the mainboard?

4

The output of zpool status in triple backticks ``` would be helpful.

Your priority should be to restore redundancy, and then you can determine what happened to your drives.

Perhaps they did fail. Or maybe the plugs just jiggled loose.

Refresh your backup.

And double check device serials before removing the wrong disk!

5

The AI seems right…

—-

It sounds like your NAS is experiencing issues with one of its disks, which is causing the pool to be in a degraded state. The error message “CAM status: CCB request completed with an error” typically indicates a problem with the communication between the system and a disk.

To address this issue, you should:

  1. ** Check the Pool Status**: Use the TrueNAS web interface to check the status of your pool. Navigate to the Storage Dashboard by clicking on Storage in the main navigation menu. This will show you which disk is causing the degraded state.
  2. Identify the Faulty Disk: The dashboard will indicate which disk is faulted or degraded. If a disk shows a faulted state, it means TrueNAS has detected an issue with that disk.
  3. Replace the Faulty Disk: If a disk is indeed faulted, you should replace it. You can find detailed instructions on how to replace a failed disk in the TrueNAS Documentation.
  4. Resilver the Pool: After replacing the disk, TrueNAS will automatically start the resilvering process to restore redundancy to your pool.

If you continue to experience issues after replacing the disk, it may be worth checking other hardware components such as cables or the disk controller.

Your feedback helps make this bot better! Click the thumbs-up or thumbs-down icons to rate responses. Thanks

6

To answer some of the questions:

I have a raidz2 configuration. CPU is an Intel i7-960 w/ 12GB RAM. I have six 3TB Western Digital Red Plus drives, connected to the motherboard via onboard SATA but I can’t recall which version…I think SATA II?

I will try to back everything up (maybe I will just remove the degraded backup and try adding it back fresh) and then try to get more info on the removed data drives.

Thanks for the info.

7

Update…finally got my backup to complete, but of course I don’t think I can trust it. My external hard drive has reported as degraded from day 1. Maybe there’s a connection issue, I don’t know. But I had to reboot because we lost power, and the main pool has all six drives back and running. I did get an error that a device has had an unrecoverable error, but data is still fine, and it’s not even showing me which drive had the issue. So confusing.

What is the best way to check the drives for problems? The SMART tests always say everything is fine, is there a way to do more rigorous check? It turns out the drives are under warranty until next month, so if there’s an actual drive problem, it would be nice to be able to prove it.