Device: /dev/ada1, ATA error count increased from 1 to 2

Hi all,

what shall I make out of this alert in the UI (and also in TrueCommand)?

Device: /dev/ada1, ATA error count increased from 1 to 2.

In Scrutiny there is no indication of any error counter increasing (they are all at 0, actually).

The full smartctl -x tells a different story, though:

root@freenas[~]# smartctl -x /dev/ada1
smartctl 7.4 2023-08-01 r5530 [FreeBSD 13.3-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD60EFPX-68C5ZN0
Serial Number:    WD-*****************
LU WWN Device Id: 5 0014ee 21617f15d
Firmware Version: 81.00A81
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database 7.3/5528
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Oct 10 16:51:01 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(63540) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 659) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x3039)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   100   253   021    -    0
  4 Start_Stop_Count        -O--CK   100   100   000    -    2
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   096   096   000    -    3165
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    1
192 Power-Off_Retract_Count -O--CK   200   200   000    -    0
193 Load_Cycle_Count        -O--CK   200   200   000    -    2
194 Temperature_Celsius     -O---K   116   110   000    -    34
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x04       GPL     R/O    256  Device Statistics log
0x04       SL      R/O    255  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x0c       GPL     R/O   2048  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x24       GPL     R/O    307  Current Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      78  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 3
	CR     = Command Register
	FEATR  = Features Register
	COUNT  = Count (was: Sector Count) Register
	LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
	LH     = LBA High (was: Cylinder High) Register    ]   LBA
	LM     = LBA Mid (was: Cylinder Low) Register      ] Register
	LL     = LBA Low (was: Sector Number) Register     ]
	DV     = Device (was: Device/Head) Register
	DC     = Device Control Register
	ER     = Error register
	ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 3 [2] occurred at disk power-on lifetime: 3148 hours (131 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 51 00 0b 00 00 00 00 00 00 40 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  b0 00 d5 00 01 00 00 00 c2 4f e1 40 08 31d+18:17:52.974  SMART READ LOG
  b0 00 d6 00 01 00 00 00 c2 4f e0 40 08 31d+18:17:52.974  SMART WRITE LOG
  b0 00 d6 00 01 00 00 00 c2 4f e0 40 08 31d+18:17:52.973  SMART WRITE LOG
  b0 00 d5 00 01 00 00 00 c2 4f e0 40 08 31d+18:17:52.972  SMART READ LOG
  61 00 20 00 d0 00 02 2d 36 3c 38 40 08 31d+18:17:52.969  WRITE FPDMA QUEUED

Error 2 [1] occurred at disk power-on lifetime: 2908 hours (121 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 51 00 0b 00 00 00 00 00 00 40 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  b0 00 d5 00 01 00 00 00 c2 4f e1 40 08 21d+18:32:06.300  SMART READ LOG
  b0 00 d6 00 01 00 00 00 c2 4f e0 40 08 21d+18:32:06.300  SMART WRITE LOG
  b0 00 d6 00 01 00 00 00 c2 4f e0 40 08 21d+18:32:06.299  SMART WRITE LOG
  b0 00 d5 00 01 00 00 00 c2 4f e0 40 08 21d+18:32:06.299  SMART READ LOG
  61 00 08 00 a0 00 02 1e 04 ac 88 40 08 21d+18:32:06.286  WRITE FPDMA QUEUED

Error 1 [0] occurred at disk power-on lifetime: 439 hours (18 days + 7 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 51 00 0b 00 00 00 00 00 00 40 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  b0 00 d5 00 01 00 00 00 c2 4f e1 40 08 18d+07:04:08.527  SMART READ LOG
  b0 00 d6 00 01 00 00 00 c2 4f e0 40 08 18d+07:04:08.527  SMART WRITE LOG
  b0 00 d6 00 01 00 00 00 c2 4f e0 40 08 18d+07:04:08.526  SMART WRITE LOG
  b0 00 d5 00 01 00 00 00 c2 4f e0 40 08 18d+07:04:08.526  SMART READ LOG
  b0 00 d5 00 01 00 00 00 c2 4f e0 40 08 18d+07:04:08.526  SMART READ LOG

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3154         -
# 2  Short offline       Completed without error       00%      3130         -
# 3  Short offline       Completed without error       00%      3106         -
# 4  Short offline       Completed without error       00%      3082         -
# 5  Short offline       Completed without error       00%      3058         -
# 6  Extended offline    Completed without error       00%      3046         -
# 7  Short offline       Completed without error       00%      3010         -
# 8  Short offline       Completed without error       00%      2986         -
# 9  Short offline       Completed without error       00%      2962         -
#10  Short offline       Completed without error       00%      2938         -
#11  Short offline       Completed without error       00%      2915         -
#12  Short offline       Completed without error       00%      2891         -
#13  Extended offline    Completed without error       00%      2878         -
#14  Short offline       Completed without error       00%      2843         -
#15  Short offline       Completed without error       00%      2819         -
#16  Short offline       Completed without error       00%      2795         -
#17  Short offline       Completed without error       00%      2771         -
#18  Short offline       Completed without error       00%      2747         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
Device State:                        Active (0)
Current Temperature:                    34 Celsius
Power Cycle Min/Max Temperature:     20/40 Celsius
Lifetime    Min/Max Temperature:     20/40 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/65 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (36)

Index    Estimated Time   Temperature Celsius
  37    2024-10-10 08:54    34  ***************
 ...    ..( 98 skipped).    ..  ***************
 136    2024-10-10 10:33    34  ***************
 137    2024-10-10 10:34    30  ***********
 ...    ..(  3 skipped).    ..  ***********
 141    2024-10-10 10:38    30  ***********
 142    2024-10-10 10:39    31  ************
 ...    ..(  4 skipped).    ..  ************
 147    2024-10-10 10:44    31  ************
 148    2024-10-10 10:45    32  *************
 ...    ..(  7 skipped).    ..  *************
 156    2024-10-10 10:53    32  *************
 157    2024-10-10 10:54    33  **************
 ...    ..( 30 skipped).    ..  **************
 188    2024-10-10 11:25    33  **************
 189    2024-10-10 11:26    34  ***************
 ...    ..(324 skipped).    ..  ***************
  36    2024-10-10 16:51    34  ***************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 3) ==
0x01  0x008  4               1  ---  Lifetime Power-On Resets
0x01  0x010  4            3165  ---  Power-on Hours
0x01  0x018  6      6949254854  ---  Logical Sectors Written
0x01  0x020  6        87412998  ---  Number of Write Commands
0x01  0x028  6     71904826733  ---  Logical Sectors Read
0x01  0x030  6        42791946  ---  Number of Read Commands
0x01  0x038  6      2804065408  ---  Date and Time TimeStamp
0x02  =====  =               =  ===  == Free-Fall Statistics (rev 1) ==
0x02  0x010  4               0  ---  Overlimit Shock Events
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4            3120  ---  Spindle Motor Power-on Hours
0x03  0x010  4            3119  ---  Head Flying Hours
0x03  0x018  4               2  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               0  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4               0  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              34  ---  Current Temperature
0x05  0x010  1              33  ---  Average Short Term Temperature
0x05  0x018  1              33  ---  Average Long Term Temperature
0x05  0x020  1              40  ---  Highest Temperature
0x05  0x028  1              27  ---  Lowest Temperature
0x05  0x030  1              38  ---  Highest Average Short Term Temperature
0x05  0x038  1              30  ---  Lowest Average Short Term Temperature
0x05  0x040  1              35  ---  Highest Average Long Term Temperature
0x05  0x048  1              33  ---  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              65  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4              28  ---  Number of Hardware Resets
0x06  0x010  4              18  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0xff  =====  =               =  ===  == Vendor Specific Statistics (rev 1) ==
0xff  0x008  7               0  ---  Vendor Specific
0xff  0x010  7               0  ---  Vendor Specific
0xff  0x018  7               0  ---  Vendor Specific
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2           27  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           28  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4     11394806  Vendor specific

root@freenas[~]# 

Any hints about what I am dealing with?

Maybe there’s something issuing an invalid “SMART” command that aborts before completion?

That’s what it seems to hint at, when you look at this…

…leading up to this:

First at around 18 days (power-on time), then 3 days later, then again 10 days later.

Is there anything that this timeframe matches up with? Does TrueCommand do anything SMART-related outside of what TrueNAS already does? Maybe Scrutiny? Maybe some other application?

I run short self tests on all devices daily at 6 am local time except on Saturdays and long self tests weekly at 6 am on Saturdays.

Additionally I pull SMART data daily at midnight for Scrutiny and every five minutes for Grafana from two different cron jobs. Both of these use smartctl -x.

So yes, quite a bit of SMART things going on here.

How did you make out with this? I am also using Scrutiny, have it running “docker exec ix-scrutiny-scrutiny-1 /opt/scrutiny/bin/scrutiny-collector-metrics run” every hour, also have scrubs and SMART tests scheduled, and three times now have gotten “ATA error count increased from 0 to 1”. The most interesting thing is they all happen at 7:05/7:06PM local time…

I have come to the conclusion to ignore it.

Yea I figure this is a nuisance alert that will forever be at the bottom of the priority pile.