Errors: Permanent errors have been detected in the following files

TrueNAS General
, ,
1

Amateur here. Hoping to get help diagnosing a drive that has error on one particular file that leaves my pool in an “unhealthy” state.

  pool: HC320
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 06:10:42 with 1 errors on Mon Mar  9 16:17:05 2026
config:

        NAME        STATE     READ WRITE CKSUM
        HC320       ONLINE       0     0     0
          sde2      ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        HC320/Zora@auto-2026-02-28_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-12_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-05_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        /mnt/HC320/Zora/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-08_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-02_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-09_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-02-27_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-03_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-10_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-07_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-11_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-04_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-01_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2
        HC320/Zora@auto-2026-03-06_00-00:/(!)Photos/Explore/2008/2008 Aug 21/IMG_9014.CR2

I have scrubbed several times, but the error does not go away and a ‘zpool clear’ does not work.

Smartctl -a:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red Plus
Device Model:     WDC WD80EFZZ-68BTXN0
Serial Number:    WD-CA0Z3PYK
LU WWN Device Id: 5 0014ee 214e0e74f
Firmware Version: 81.00A81
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5640 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/6088
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Mar 12 21:09:18 2026 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (18644) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 888) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   182   182   021    Pre-fail  Always       -       9891
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       70
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       6407
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       60
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       32
193 Load_Cycle_Count        0x0032   197   197   000    Old_age   Always       -       9013
194 Temperature_Celsius     0x0022   117   110   000    Old_age   Always       -       35
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      6299         -
# 2  Extended offline    Completed without error       00%      6287         -
# 3  Short offline       Completed without error       00%      6127         -
# 4  Short offline       Completed without error       00%      5960         -
# 5  Short offline       Completed without error       00%      5792         -
# 6  Short offline       Completed without error       00%      5624         -
# 7  Extended offline    Completed without error       00%      5616         -
# 8  Short offline       Completed without error       00%      5460         -
# 9  Short offline       Completed without error       00%      5291         -
#10  Extended offline    Completed without error       00%      5211         -
#11  Short offline       Completed without error       00%      5124         -
#12  Short offline       Completed without error       00%      4956         -
#13  Extended offline    Completed without error       00%      4878         -
#14  Short offline       Completed without error       00%      4788         -
#15  Short offline       Completed without error       00%      4620         -
#16  Extended offline    Completed without error       00%      4468         -
#17  Short offline       Completed without error       00%      4285         -
#18  Short offline       Completed without error       00%      4117         -
#19  Extended offline    Completed without error       00%      4084         -
#20  Short offline       Completed without error       00%      3949         -
#21  Short offline       Completed without error       00%      3782         -

Selective Self-tests/Logging not supported

The above only provides legacy SMART information - try 'smartctl -x' for more

Thanks.

2

Those are snapshots. Do you need to keep them? Do you have a good copy of the file IMG_9014.CR2 elsewhere?

1 Like
3

I do not need to keep the snapshots and have a good copy on a cold storage drive. Is it as simple as just deleting the snapshots and just copying and replacing that one file? Do I need to run a scrub after that?

4

This is one of those problems you can get by using a Stripe for a pool. No redundancy here. Just make sure you have a good backup of your data if you plan to run like this.

Your drive looks good. I did notice that your head loading is about 50% higher than the number of hours for this drive. While that is not a big deal, typically we try to steer people towards running the drives 100% of the time, but everyone has different needs.

1 Like
5

I use Syncthing that mirrors the entire drive between two machines at home. I can pull a copy from there too, but don’t know if that would also be “corrupted” (sorry, I don’t quite understand this stuff well).

That drive is set to the default, Always On so not sure what the head loading issue is? Could it be from rebooting the machine a lot? There were many 10-restarts per days when I was trying to (and still trying to) debug my way through network issues and what not. Maybe it’s just my imagination and it’s always been the case I notice louder ticking sounds and vibration through the metal case chasis.

6

It likely is corrupt too, but open the file, see if it seems fine. I don’t know what a .CR2 file is, but maybe it will work. And that is good that you do have backups. Better safe than sorry.

1 Like
7

Canon Raw picture file.

The original is corrupted, which may be insignificant if it just results in a single weird pixel, or devastating if the file is internally compressed and everything after the wrong byte is off. Check the external copy.
To clear the error you need to delete IMG_9014.CR2 in the pool and all the snapshots it’s in.
Consider expanding the pool to a mirror so ZFS has redundancy to correct such errors.

3 Likes