TrueNAS on proxmox stopped working : disk fault?

Hi,

I have trueNas installed on a proxmox VM for long months and all worked well until today :-(.

I got an I/O error displayed on proxmox VM and none of mounts were working on any machine …

I looked for any reason and saw somewhere to run this command :

smartctl -a /dev/sdc

This is the result (executed on the proxmox host) :

root@pve:~# smartctl -a /dev/sdc
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.12-1-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 2.5 5400
Device Model:     ST5000LM000-2AN170
Serial Number:    WCJ5R1TS
LU WWN Device Id: 5 000c50 0e01f81e3
Firmware Version: 0001
User Capacity:    5,000,981,078,016 bytes [5.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5526 rpm
Form Factor:      2.5 inches
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Aug 28 20:51:12 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x73) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 811) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30a5) SCT Status supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   050   050   006    Pre-fail  Always       -       46297920
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   099   099   020    Old_age   Always       -       1867
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       416
  7 Seek_Error_Rate         0x000f   089   060   045    Pre-fail  Always       -       856162986
  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       9525 (33 20 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       9
183 SATA_Downshift_Count    0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   098   098   000    Old_age   Always       -       2
188 Command_Timeout         0x0032   098   098   000    Old_age   Always       -       8590065666
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   056   049   040    Old_age   Always       -       44 (Min/Max 42/44)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   097   097   000    Old_age   Always       -       7033
193 Load_Cycle_Count        0x0032   066   066   000    Old_age   Always       -       69753
194 Temperature_Celsius     0x0022   044   051   000    Old_age   Always       -       44 (0 24 0 0 0)
195 Hardware_ECC_Recovered  0x001a   077   064   000    Old_age   Always       -       46297920
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       200
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       200
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       6502 (105 108 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       56802342940
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       92179484164
254 Free_Fall_Sensor        0x0032   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 1
ATA Error Count: 2
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2 occurred at disk power-on lifetime: 9525 hours (396 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 7f 00 40 40 00  33d+10:35:03.126  READ FPDMA QUEUED
  60 00 01 80 00 00 40 00  33d+10:35:03.008  READ FPDMA QUEUED
  60 00 01 7f 00 40 40 00  33d+10:35:03.008  READ FPDMA QUEUED
  60 00 01 80 00 00 40 00  33d+10:35:03.007  READ FPDMA QUEUED
  60 00 01 81 00 00 40 00  33d+10:35:03.006  READ FPDMA QUEUED

Error 1 occurred at disk power-on lifetime: 9525 hours (396 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 7f 00 40 40 00  33d+10:20:04.912  READ FPDMA QUEUED
  60 00 10 ff ff ff 4f 00  33d+10:20:04.795  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  33d+10:20:04.794  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  33d+10:20:04.794  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  33d+10:20:04.779  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

This disk is “only” passed to my VM, I don’t understand why it could be the cause of my issues …

Maybe this could help to understand, this is my proxmox “hardware” section

… I really hope my HDD is not broken …

Thank you for any help

In addition, this is messages I have when I launch the VM

Please read this carefully: