Another set of eyes on a smartctl output please

TrueNAS General
, ,
1

Fairly new user here in the process of moving around 20TiB from my old data storage consisting of a hodgepodge o’disks merged with mergerFS and (kinda) protected by snapraid to a brand new new server running TrueNAS SCALE with 6 x 18 TiB IronWolf drives formatted with RAIDZ2.

During that process I had a warning thrown that the pool had been degraded due to persistent errors. Digging around previous posts on the board helped me find which drive was showing the problem and found it had had 34 read errors logged against it. Other posts also suggested running a long offline test, which I did which passed successfully and then collect the SMART data and check that, which I’ve done, and to my (limited knowledge) I can’t see any errors reported by the drive itself, so can only assume the errors were reported by the controller chip.

Here’s the full SMART output if anyone else would please either confirm, or shoot down, my assumptions, as I suspect the backplane the drives are attached to could be the cause of the errors.

admin@truenas[~]$ sudo /sbin/smartctl -a -v 1,raw48:54 -v 7,raw48:54 /dev/sdd
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.32-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST18000NT001-3LU101
Serial Number:    ZR5E7VT8
LU WWN Device Id: 5 000c50 0e6901a71
Firmware Version: EN01
User Capacity:    18,000,207,937,536 bytes [18.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database 7.3/5528
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Aug 29 17:23:48 2024 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (1574) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   075   064   044    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0003   090   088   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       210
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   083   060   045    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       1469
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       209
 18 Unknown_Attribute       0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   097   000    Old_age   Always       -       919137026275
190 Airflow_Temperature_Cel 0x0022   058   049   000    Old_age   Always       -       42 (Min/Max 27/51)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       103
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       1584
194 Temperature_Celsius     0x0022   042   051   000    Old_age   Always       -       42 (0 22 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       52269
200 Multi_Zone_Error_Rate   0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       1230 (96 6 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       301927602314
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       320386278770

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      1443         -
# 2  Extended offline    Completed without error       00%        50         -
# 3  Conveyance offline  Completed without error       00%        26         -
# 4  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The above only provides legacy SMART information - try 'smartctl -x' for more

admin@truenas[~]$

Cheers, and thanks in advance.

2

Bit warm and has been too warm
You probably aren’t running enough smart tests

3

This is indicative of a cabling or power issue affecting communication between the cpu and drive, leading to io errors.

And this is a very high value.

2 Likes
4

UDMA CRC Error Count - 52,269, could be issues with power, cabling, or the controller. If this is happening on multiple disks I’d scrutinize the controller.

Command Timeout - 919,137,026,275, probably caused by the above, would keep an eye on this to see if it continues to increase after you figure out & fix whatever is causing the CRC Errors.

5

Thanks guys, this is the kind of information I was looking for. This is a NAS type device with a backplane for the drives, so no separate data and power cables, plus it also hosts the SATA chip, so I suspect that’s where all the issues are originating from. I’m guessing as the drive itself is OK, I’m good to go ahead and clear the “Faulted” from the drive and put it back into service while I’m pursuing a warranty claim on the backplane.

As a newbie user, which tests would you suggest at what intervals.

Cheers, and again thanks for the help.

6

What is the hardware?

There are still cables going to the backplane for power/data. You may want to try shutting it down and pulling/inserting the affected drive again.

7

“Regularly”.

A daily short cannot hurt.
As for the long (26 hours!), anything between weekly and monthly.

8

It’s a new ZimaCube that I picked up through KickStarter. They’ve had a few teething problems but are being very proactive working through them to their credit.

Yeah, they’re kinda weird, more like connector cables you’d find in a laptop. But hey, I’m not a hardware designer.

Thanks, I’ll try that as my next step after the current rsync finishes.

Noted. Thank you for the suggestions and I’ll make sure they don’t clash with the scrub, as the default settings there seem to be the same schedule, which I’m guessing I should avoid.

Cheers.

9

Thanks for the link. Those are interesting. Good to see more people putting out storage-focused hardware at a lower price than the well-known brands. U-Green also released some options recently.

Not much detail on the backplane. It’s probably a ‘pass through’ backplane that provides power and relies on the motherboard’s SATA chipset. Nothing wrong with that.

Try re-inserting the drive to see if the errors reappear. If they do, swap two of the drives so you have a different drive in the problematic slot. TrueNAS won’ mind. If the errors continue with the drive in a different slot – you know the drive is the issue. If the errors appear on whatever drive was moved to the slot – you know it’s the NAS. At that point I’d open it up and see if there are any internal cables I can re-seat. If it’s STILL occurring, I’d look into an RMA.

Good luck!

2 Likes
10

Just for some clarity, the UDMA errors will never clear (return to zero or decrease in value). Watch the numbers and if they continue to climb, you have a few options in order to figure out what is causing the issue.

  1. Pull the hard drive, inspect the electrical connectors for damage on both the drive and cage.

  2. Connect the drive directly to a SATA data connection (no backplane) and watch. If the UDMA values continue to go up then I would suspect the drive as the failing piece.

  3. Swap the drive into a different location where the original drive in that new location has no UDMA errors. Then watch.

The goal is to figure out if the drive has the issue or the cage does. Proper alignment is critical in drive cages and if not careful, easy to damage the electrical connectors on either side. Also, never hot swap a drive if you don’t have to, regardless of if the cage is said to support it. It is an unnecessary risk if you can power down the system and then replace the drive. I about shit my pants one day when replacing a drive in a Supermicro system. Burned up the power connector on the hard drive and on the drive cage. Thankfully it was a work system and we were evaluating it. Well it failed for hot swap. Now we never hot swap that system model. You learn and move forward.

1 Like
11

No, the SATA chip does actually live on the backplane.

Now, here’s the issue with that, and why I think the backplane is most likely the problem. Sometimes it takes 2 or 3 reboots combined with re-seating a couple of drives for all 6 to be recognised.

Already been done.

Understood.

Unfortunately the board doesn’t have any native SATA connectors.

Will be pulling the UDMA numbers for all the drives, swapping what I can (see reply above) and watching very carefully.

Nope, all swaps are done with system shut down and power cable pulled.

Cheers, and again, thanks to all.

12

The data about having to reboot multiple times or reseat drives for them all to be recognized is either new information, or I skimmed over it.

In that case, I would just move to RMA or refund. That’s not normal.

13

You can use my little Multi-Report script to monitor and it will flag you when the value increases. When you install the script you will have the option to scan the drives for an automatic compensation, this will read the drive values and for example, UDMA = 1 (or greater) then the report will say 0 (1) on a yellow background. If that value changes to 1 (2) in a orange background, then the value increased. in your specific case the value would start at 0 (52269) on a yellow background. It makes the error very obvious. And of course, if you have a value of 0 then it would be a normal background, nothing highlighted. You would run the script once a day via CRON Job. See link in my signature if you would like to give it a try. Here is the link to the forum page for some light reading.

14

Errr, yes I know it’s not normal. I’m currently working with their support team, but it’s difficult when they’re half way around the world from me.

15

Many thanks for that, I will definitely check it out.

Cheers.

16

I did check this out, but I’m not getting anything emailed to me. I tried 2 internal machines that have email servers running on them and also my ISP email, but nothing.

I’m not really sure where you want issues/questions reported, but I’m guessing it’s not really here on the TrueNAS board, so where.

Cheers.

17

Multi-report thread

2 Likes