Getting Pool 'boot' has encountered an uncorrectable I/O failure - smartcrt check successful

bigbrovar · November 21, 2024, 8:45am

Hello everyone,

I have been having an issue with my truenass (system config below) where login screen and ssh access is stuck until I do a power reboot. This has been happening about a week from now. Once system boots up… everyone works fine until it happens again usually about 24 hours later.

When this is happening, all my shares seem to work fine and are accessible. I am just unable to login via GUI or ssh.

I decided to connect the server to my kvm switch and noticed that the console was frozen with the constant error message


Pool 'boot' has encountered an uncorrectable I/O failure and has been suspended.

As usually after a power cycle everything comes back up. I did a config backup and I have spare disk to replace the boot drive. However I did some smartcrt check by running

smartctl -t long /dev/ada3

The result are below


smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, 
=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 10 minutes for test to complete.
Test will complete after Thu Nov 21 08:49:25 2024 WAT
Use smartctl -X to abort test.
root@dune[~]# smartctl -a /dev/ada3      
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, 
=== START OF INFORMATION SECTION ===
Model Family:     Phison Driven SSDs
Device Model:     KINGSTON SA400S37120G
Serial Number:    50026B7784FAD8A9
LU WWN Device Id: 5 0026b7 784fad8a9
Firmware Version: S3E00100
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
TRIM Command:     Available
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Nov 21 08:52:14 2024 WAT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(  120) seconds.
Offline data collection
capabilities: 			 (0x11) SMART execute Offline immediate.
					No Auto Offline data collection support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					No Selective Self-test supported.
SMART capabilities:            (0x0002)	Does not save SMART data before
					entering power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (  10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   100   100   000    Old_age   Always       -       100
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       12061
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       79
148 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      -       0
149 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      -       0
167 Write_Protect_Mode      0x0000   100   100   000    Old_age   Offline      -       0
168 SATA_Phy_Error_Count    0x0012   100   100   000    Old_age   Always       -       261
169 Bad_Block_Rate          0x0000   100   100   000    Old_age   Offline      -       0
170 Bad_Blk_Ct_Erl/Lat      0x0000   100   100   010    Old_age   Offline      -       0/0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age   Always       -       0
173 MaxAvgErase_Ct          0x0000   100   100   000    Old_age   Offline      -       0
181 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
182 Erase_Fail_Count        0x0000   100   100   000    Old_age   Offline      -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
192 Unsafe_Shutdown_Count   0x0012   100   100   000    Old_age   Always       -       33
194 Temperature_Celsius     0x0022   037   055   000    Old_age   Always       -       37 (Min/Max 21/55)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
199 SATA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       0
218 CRC_Error_Count         0x0032   100   100   000    Old_age   Always       -       261
231 SSD_Life_Left           0x0000   100   100   000    Old_age   Offline      -       100
233 Flash_Writes_GiB        0x0032   100   100   000    Old_age   Always       -       0
241 Lifetime_Writes_GiB     0x0032   100   100   000    Old_age   Always       -       26
242 Lifetime_Reads_GiB      0x0032   100   100   000    Old_age   Always       -       412
244 Average_Erase_Count     0x0000   100   100   000    Old_age   Offline      -       1
245 Max_Erase_Count         0x0000   100   100   000    Old_age   Offline      -       2
246 Total_Erase_Count       0x0000   100   100   000    Old_age   Offline      -       1225

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 1

ATA Error Count: 0
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 0 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 40  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d0 01 00 4f c2 40 08      00:00:00.000  SMART READ DATA
  b0 d1 01 01 4f c2 40 08      00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
  b0 da 00 00 4f c2 40 08      00:00:00.000  SMART RETURN STATUS
  b0 d5 01 00 4f c2 40 08      00:00:00.000  SMART READ LOG
  b0 d5 01 01 4f c2 40 08      00:00:00.000  SMART READ LOG

Error -1 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 40  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d5 01 06 4f c2 40 08      00:00:00.000  SMART READ LOG
  b0 da 00 00 4f c2 40 08      00:00:00.000  SMART RETURN STATUS
  b0 d0 01 00 4f c2 40 08      00:00:00.000  SMART READ DATA
  b0 d5 01 06 4f c2 40 08      00:00:00.000  SMART READ LOG
  b0 d5 01 01 4f c2 40 08      00:00:00.000  SMART READ LOG

Error -2 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 40  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d5 01 06 4f c2 40 08      00:00:00.000  SMART READ LOG
  b0 da 00 00 4f c2 40 08      00:00:00.000  SMART RETURN STATUS
  b0 d0 01 00 4f c2 40 08      00:00:00.000  SMART READ DATA
  b0 d5 01 06 4f c2 40 08      00:00:00.000  SMART READ LOG
  b0 d5 01 01 4f c2 40 08      00:00:00.000  SMART READ LOG

Error -3 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 40  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d5 01 06 4f c2 40 08      00:00:00.000  SMART READ LOG
  b0 da 00 00 4f c2 40 08      00:00:00.000  SMART RETURN STATUS
  b0 d0 01 00 4f c2 40 08      00:00:00.000  SMART READ DATA
  b0 d5 01 06 4f c2 40 08      00:00:00.000  SMART READ LOG
  b0 d5 01 01 4f c2 40 08      00:00:00.000  SMART READ LOG

Error -4 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 40  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d0 01 00 4f c2 40 08      00:00:00.000  SMART READ DATA
  b0 d1 01 01 4f c2 40 08      00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
  b0 da 00 00 4f c2 40 08      00:00:00.000  SMART RETURN STATUS
  b0 d5 01 00 4f c2 40 08      00:00:00.000  SMART READ LOG
  b0 d5 01 01 4f c2 40 08      00:00:00.000  SMART READ LOG

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     12061         -
# 2  Extended offline    Completed without error       00%     12060         -

Selective Self-tests/Logging not supported

I don’t know what to make of it. I also would appreciate input on how i can further troubleshoot this error.

System config

Motherboard make and model
- Asus ROG B550I Gaming
CPU make and model
- AMD Ryzen 5 PRO 4650G with Radeon Graphics
RAM quantity
- 64GB ECC
Hard drives, quantity, model numbers, and RAID configuration, including boot drives
- Boot Drive = 120GB KINGSTON SA400S37120G
RAID Config = Z1
1 x WD Red Plus - 8TB - WDC WD80EFAX-68LHPN0
8 x HGST Deskstar NAS - 8TB - HDN728080ALE604 x
Hard disk controllers
- Onboard x 4 Sata
- [SATA HBA - 5 SATA to a M.2 NVMe slot,]
Network cards
- Intel i266
- Mellanox MCX311A-XCAT CX311A ConnectX-3 EN 10G Ethernet 10GbE SFP+

NugentS · November 21, 2024, 10:13am

Whats the boot disk plugged into?

elvisimprsntr · November 21, 2024, 11:08am

Hopefully not this

SmallBarky · November 21, 2024, 2:26pm

What is your SATA HBA you listed under Hard disk controllers? You seem to have more SATA devices than ports. Listing boot drive and 9 HD, 10 ports

The spec sheet for motherboard lists a Intel i225, that is the Intel problem NIC chipset. You will have to do an internet serach to see what the exact, reported problems were with that. It may just be 2.5Gbs and is fine as 1Gbps

It’s hard to tell from your drive listing if you have 9 HD all in Raid-Z1.

Fleshmauler · November 21, 2024, 2:31pm

Suspect - is the boot drive connected to this? If so, any changes when connected directly to motherboard?

Can always give this a test, should take like 5 minutes max.

bigbrovar · November 21, 2024, 4:16pm

Thanks for the reply. I can not quite remember but I do think it might have been plugged to a SATA HBA (connected to one of the PCIE 3 slot on the motherboard.) and not directly to the board. It has been a while but I can open it up to check. This is my first PC build so please forgive if I am using the wrong terminology here

bigbrovar · November 21, 2024, 4:21pm

I will try this out and revert. I would also confirm where the boot disk is connected to. What is best practice?

The board is an Is an AMD board that takes AMD AM4 socket.

Fleshmauler · November 21, 2024, 4:22pm

Depends on your exact model of HBA - I’m suspect of anything that connects to an m.2 slot. Lotta these kinda boards that ‘work fine’ until they don’t.

Your motherboard is very likely completely fine.

bigbrovar · November 21, 2024, 4:22pm

One more thing. Would changing where the connection point of the drives impact on the over system integrity like the raid (sorry if this is a dumb question)

Fleshmauler · November 21, 2024, 4:23pm

Good question - it won’t matter if you change the connection order on ZFS unless you have something very wrong with how your hardware presents the drives. The reason I’m suspect of the unspecified HBA is because ZFS needs direct access to the HDDs; some HBAs are unacceptable for ZFS application due to not doing this & can/will cause problems.

Edit: there are more possible ways some HBAs could cause problems from chipset overheating to bandwidth issues, etc.

SmallBarky · November 21, 2024, 4:33pm

I would just make sure you have a saved backup of your configuration and maybe a spare SSD to replace the boot drive if problems persist. Could be bad cable, HBA or drive.
Just be prepared with the backup config and USB with current version of TrueNAS so you can install on a new boot device and recover quickly.

Posting the Make and Model of the HBA you are using may get feedback

bigbrovar · November 21, 2024, 4:45pm

Thanks so much for all the help guys. I am not close to the system now but I sshed in and here is what I found about the sata hba plus some pictures I took when I first got it

It seems to use JMB585 Chip and comes with a built in heat sink. Below is what lspci says about it. I think I have 2 sata connection to it. I am not sure but it seems one of the connection is the boot drive. The rest were connected directly to the board (for slot)

08:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller

bigbrovar · November 21, 2024, 4:45pm

I am unable to upload the picture or post a link to the product page. Let me know if you need more information. It is an M.2 to 5-SATA expansion card.

Fleshmauler · November 21, 2024, 4:53pm

This 100% looks like what I wouldn’t want to see… that being said - you aren’t sure if the boot drive is connected to it, so I’ll withhold judgment until you can validate physically when you’re on site.

Try moving boot drive straight to motherboard if possible - if problems stop, get rid of that HBA. If problems persist, then HBA is likely working ok, and you should do a clean install on a replacement boot drive & upload your config.

I’m guessing it looks something like this:

SmallBarky · November 21, 2024, 4:56pm

You can try going through the forum tutorial to see if doing the tasks bumps your status. Just copy and paste the following in a reply window and send. It should bring up the bot and a little chat

@TrueNAS-Bot start tutorial

bigbrovar · November 21, 2024, 4:57pm

@TrueNAS-Bot start tutorial

bigbrovar · November 21, 2024, 5:08pm

Thanks took the tutorial and here is the image of the m.2 to sata which I used.

bigbrovar · November 21, 2024, 5:49pm

I have one question. In case we isolate this to be a problem with the m.2 to sata hba. How do I ultimate resolve this? Moving the boot directly to the board means that I would be moving one of 5 data disk to the sata hba… I have 5 disks and 1 boot disk. The board has just 4 sata port.

Is there a recommended sata hba I can use which is reliable?

SmallBarky · November 21, 2024, 6:00pm

Do you have an available PCI-e slot. LSI HBA, plain or in IT mode is usual recommendation.

bigbrovar · November 21, 2024, 6:35pm

This would have been ideal unfortunately the only PCI-e slot is used up by a 10Gbe network card. Is it possible to have a network card (10Gbe) and a sata hba in one with something that is tested and reliable.