The drive repeatedly:disappeared from the SATA bus and reappeared during resilver - How I fixed it

TrueNAS General
,
1

The new drive repeatedly:
disappeared from the SATA bus
reappeared
disappeared again
reappeared

It was doing my head in, what fixed it for me was installing 0.5M Sata 6Gbps Speed and Double Group Shielding Anti Interference cables

  • Anti-interference: double group shielding made of aluminum foil anti-interference, double group aluminum foil shielding, only for more professional anti-interference and effective improvement of transmission performance. It is used to connect multi-channel removable hard drives in computer cases or other SATA 7 pin devices.

41IZQ7Cs39L
41IZQ7Cs39L500Ɨ500 22.8 KB

After running monitoring command:-

tail -f /var/log/messages | egrep -i ā€œada|ahci|cam|timeout|reset|I/O|diskā€ or:-

tail -f /var/log/messages

had non of these errors appear:-

detached
Periph destroyed
CAM status
RESET
lost device
detached
SETFEATURES SET TRANSFER MODE
ATA status: 51 (DRDY SERV ERR)
error: 04 (ABRT)
Retries exhausted

The resilver worked flawlessly

1 Like
2

Cables can be bad, yes. That is why changing them is a common recommendation when troubleshooting.

Did your drive show CRC errors in its SMART report as well?

3

I edited the title of the thread so that people know you were not asking for help. Iā€™m not saying they will not ask you questions but some forum members skip this and look for threads asking for help.

I too am curious what the SMART data shows. Typically a SATA cable failure results in a high number count for UDMA_CRC_ERRORS. These errors ā€œneverā€ clear, but if they are incrementing, then the problem is not fixed. But it does sound like you fixed it.

4

The most important clues

1. AHCI port 4 is repeatedly timing out



ahcich4: Timeout on slot...
(aprobe0:ahcich4:0:0:0): ATA_IDENTIFY
CAM status: Command timeout

Later:



(ada1:ahcich4:0:0:0): READ_DMA
CAM status: Command timeout
Retrying command

Then:



READ_FPDMA_QUEUED
CAM status: Command timeout

Those are communication failures, not media errors.

A failing hard drive usually reports things like:

  • UNC (uncorrectable read)

  • Reallocated sectors

  • Pending sectors

  • SMART failures

Instead, your drive is simply not responding over SATA.

5

Thanks for that. It would be quite helpful for anyone experiencing this issue, it took me few days to work it out, The shielded sata cables fixed the problem 100%

6

one of the logs attached, I donā€™t see UDMA_CRC_ERRORS, Iā€™m pretty sure that was never seen

Alot of troubleshooting was done before I nailed it with the sata cables, example

1). Powered down one of the mirror pools thinking maybe this has something to do with not enough power for resilver. The power supply is 650watt and i have 10 drives.

2). upgraded the OS, from 13.0 to 13.3.

3). Removed a power cable splitter that i was using on resilver drive and put it on a proper power connector.

  1. changed sata ports etc etc

Jul 18 122639 freenas ahcich29 AHCI.txt (143.7 KB)

7

SMART value for UDMA CRC Errors is a communications failure, not necessarily a drive hardware failure. This is the typical indication we see when a SATA cable or HBA has gone bad, I have never seen this value increment due to a failed drive.

It is often faster to identify it this way than going through the TrueNAS logs.

8

Sorry, I did not read your posting close enough. You would not see the SMART UDMA_CRC_ERROR in the TrueNAS logs, it would be in the SMART Log data smartctl -a /dev/sd? where ?= the drive letter.

Cheers

9

For the record, that quote about Chatgpt attributed to me, is not what I wrote.
It looks like a [quote] tag ended up somewhere where it shouldnā€™t have been.

Further, I would appreciate it if you donā€™t filter all your replies through ChatGPT having it decide what to share and what not to. The full output of the SMART report would have been much more useful.

2 Likes
10

All good, It would be in the smartctl output. This is my original drive which I resilvered after replacing sata cables

smartctl -a devada.txt (20.6 KB)

11

I knew was going to get flamed for using chatgpt, I was only trying to help, Iā€™m not versed in all this stuff, so wanted to get any help i can get. but you are right full command output would be better

1 Like
12

@robert65 If this is not your primary failure reason, Iā€™d be shocked. You have an SMR drive! This can cause all kinds of problems. Imagine thisā€¦ You update/write 200GB of data to your pool, then your drive must reorganize the pool of data and this takes a lot of time. It is happening on the drive automatically and has a very low priority. Now you run a scrub, or you write more data. These actions combine and create chaos with the drive while it still tries to reorganize your Shingled data. That is just an idea why SMR is a very poor choice as a ZFS Pool drive. I donā€™t think you had any physical SATA connectivity issues as UDMA_CRC_ERRORS = 0. But yes, thanks for sharing the data, things make more sense to me now.

As for ChatGPT, unless it is a mathematical solution, IA is very unforgiving about providing bad data, regardless of what your specs are. I generalizes things and I have yet to find an AI that can tell me correct answers. I try to not use them unless I need something that I cannot figure out. But I am wide enough to spot when they are generalizing and likely to be giving me bad information.

So my advice with respect to AI, use it for math, if you use it for much else, ask it clarifying questions like ā€œAre you sure that is for TrueNAS version 26ā€. If it tells you that ā€œyou are correct to push back, Iā€™m sorryā€¦ā€. It infuriates me when it passes off data with such confidence that it seems like factual data, when actually it isnā€™t.

1 Like
13

Yes I know I have the wrong drives, regardless though I got it working solid with those shielded sata cables, and I know about chatgpt does not get it right, but it gave me enough that it wasnā€™t totally wrong. Do not think just because you think you know better that you do, sometimes youā€™d be surprised, enough said, I am not here to argue I am here to say I definitely fixed the flaky issue with sata connectivity bu changing to those cables I mentioned.

14

I didnā€™t think I was arguing, I only provide the best advice I can based on the data provided and I am fallible. We can agree to disagree, Iā€™m perfectly fine with that.

However, if the problem returns, and the UDMA_CRC_ERROR_COUNT is still a zero value, you might just try the following steps to rule out it being an SMR issue:

  1. If a scrub is in progress, stop it.
  2. If a SMART Long test is in progress, stop it.
  3. Listen to the drives, do they still sound active? If yes, unplug the network cable to ensure data is not being read or written.
  4. Listen to the drives, do the heads sound like they are moving around?
    a. If yes:
    1. The drive is likely rearranging the Shingled Data.
    2. Let it perform the shingling rewrite and complete the operation. Upon completion the drive should be quiet, or at least not very active. How long will this take? It depends on how much data the drive needs to reshingle. It could be 10 minutes or 10 hours.
    3. If you have Snapshots setup. Then disable your snapshots for the duration of this troubleshooting.
    4. Eventually the drive(s) will stop chattering (head armature movement).
    5. Run a zpool clear on the affected pool and then start a scrub.
    b. If no:
    1. Sorry I wasted your time with this silliness.

I hope the cables truly did fix the issue. No one enjoys constant troubleshooting.

But would you be so kind as to update this thread in a month or two with an update. If the problem remains gone then it really was the cables, that would be a great data point for everyone here. After all, these forums are about sharing information and helping one another out.

15

I yet have to see an actual solution to a Truenas problem coming from an LLM on this forum. And I dont consider random rm -fr AI suggestions as a solution.

I know its Off topicā€¦

1 Like
16

Hi Joe, the drives are good atm, thanks for the info, it is not silly and definitely not a waste of time.

That is why I am here because I spent so much time on it and wanted to share what I did to fix it.

I could not have cared at all to post what I went through, but that is not me, I like to help.

I do apologize if it seems sporadic with logs, but a lot was going on when I was trying to fix the issue.

I started another resilver but this time on 10TB ironwolf mirror set and it is running much better than the SMR drives, (not using those shielded sata cables) attached are some information, so the SMR drives were the real issue, but still to get that working right, installing those cables did the trick.

Iā€™ll take another snapshot in few hours, itā€™s over 12% resilver now and no issues.

ironwolf.txt (11.6 KB)

17

Thanks for the kind words. I think most of us like to share our experiences.

Your Ironwolf SMART data looks good. The only think I will comment on is Iā€™m a believer in running a SMART Long/Extended test Weekly or Monthly. Short tests are nice but it is very basic. The Long test will read the entire surface of your drive, where a SCRUB only reads the active data.

Anyway, good luck.

18

Thank you. uploaded another snapshot, resilver on ironwolf mirror going strong, and no errors thrown

I have changed the smart test to ā€˜longā€™ monthly..

ironwolf2.txt (9.4 KB)

19

What would be useful is if someone pointed out some links to these cables, or the ones they think are best. Iā€™ve had a lot of issues like this and gave up on them a while back. I tend to blame backplanes more than cables and try direct attach things now, but still, good quality breakout cables - sas to sata or sas to sas etc, Iā€™d love to know what everyone thinks.

1 Like
20

Iā€™ve had good results with Supermicro cables.

1 Like