Dmesg - contains strange errors?

phier · December 20, 2024, 7:58pm

hello,
i found strange messages / errors? in my dmesg on truenas, is that something i have to be worried about?

thanks!

ahcich35: Timeout on slot 12 port 0
ahcich35: is 00000000 cs 00000000 ss 00207000 rs 00207000 tfd 40 serr 00000000 cmd 0004da17
(ada2:ahcich35:0:0:0): READ_FPDMA_QUEUED. ACB: 60 80 c0 75 de 40 58 07 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 3 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 31 port 0
ahcich35: is 00000000 cs 80000000 ss 00000000 rs 80000000 tfd c0 serr 00000000 cmd 0004df17
(ada2:ahcich35:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 17 port 0
ahcich35: is 00000000 cs 00020000 ss 00000000 rs 00020000 tfd c0 serr 00000000 cmd 0004d117
(ada2:ahcich35:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 0 port 0
ahcich35: is 00000000 cs 00000001 ss 00000000 rs 00000001 tfd c0 serr 00000000 cmd 0004c017
(ada2:ahcich35:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 24 port 0
ahcich35: is 00000000 cs 01000000 ss 00000000 rs 01000000 tfd c0 serr 00000000 cmd 0004d817
(ada2:ahcich35:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 4 port 0
ahcich35: is 00000000 cs 00000000 ss 00000038 rs 00000038 tfd 40 serr 00000000 cmd 0004c517
(ada2:ahcich35:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 50 82 24 40 a5 02 00 08 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 3 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 18 port 0
ahcich35: is 00000000 cs 00000000 ss 001c0000 rs 001c0000 tfd 40 serr 00000000 cmd 0004d417
(ada2:ahcich35:0:0:0): READ_FPDMA_QUEUED. ACB: 60 80 c8 ac 7d 40 0c 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 3 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 21 port 0
ahcich35: is 00000000 cs 00200000 ss 00000000 rs 00200000 tfd 80 serr 00000000 cmd 0004d517
(aprobe0:ahcich35:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich35:0:0:0): CAM status: Command timeout
(aprobe0:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 17 port 0
ahcich35: is 00000000 cs 00000000 ss 000e0000 rs 000e0000 tfd 40 serr 00000000 cmd 0004d317
(ada2:ahcich35:0:0:0): READ_FPDMA_QUEUED. ACB: 60 80 b0 e4 d6 40 27 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 3 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 20 port 0
ahcich35: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 80 serr 00000000 cmd 0004d417
(aprobe0:ahcich35:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich35:0:0:0): CAM status: Command timeout
(aprobe0:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: Timeout on slot 25 port 0
ahcich35: is 00000000 cs 02000000 ss 00000000 rs 02000000 tfd c0 serr 00000000 cmd 0004d917
(ada2:ahcich35:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 5 port 0
ahcich35: is 00000000 cs 00000020 ss 00000000 rs 00000020 tfd c0 serr 00000000 cmd 0004c517
(ada2:ahcich35:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: Timeout on slot 4 port 0
ahcich35: is 00000000 cs 00000000 ss 00000070 rs 00000070 tfd 40 serr 00000000 cmd 0004c717
(ada2:ahcich35:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 80 2c 85 40 16 00 00 08 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 3 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 8 port 0
ahcich35: is 00000000 cs 00000100 ss 00000000 rs 00000100 tfd 80 serr 00000000 cmd 0004c817
(aprobe0:ahcich35:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich35:0:0:0): CAM status: Command timeout
(aprobe0:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: Timeout on slot 11 port 0
ahcich35: is 00000000 cs 00000800 ss 00000000 rs 00000800 tfd c0 serr 00000000 cmd 0004cb17
(ada2:ahcich35:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 0 more tries remain
Limiting closed port RST response from 1150 to 200 packets/sec
ahcich35: Timeout on slot 22 port 0
ahcich35: is 00000000 cs 00400000 ss 00000000 rs 00400000 tfd c0 serr 00000000 cmd 0004d617
(ada2:ahcich35:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 1 port 0
ahcich35: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd c0 serr 00000000 cmd 0004c117
(ada2:ahcich35:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: Timeout on slot 22 port 0
ahcich35: is 00000000 cs 00400000 ss 00000000 rs 00400000 tfd c0 serr 00000000 cmd 0004d617
(ada2:ahcich35:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 0 more tries remain
ahcich35: Timeout on slot 13 port 0
ahcich35: is 00000000 cs 00000000 ss 0000e000 rs 0000e000 tfd 40 serr 00000000 cmd 0004cf17
(ada2:ahcich35:0:0:0): READ_FPDMA_QUEUED. ACB: 60 80 70 fb 54 40 fc 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 3 more tries remain
ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich35: Timeout on slot 27 port 0
ahcich35: is 00000000 cs 00000000 ss 08000000 rs 08000000 tfd 40 serr 00000000 cmd 0004db17
(ada2:ahcich35:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 d8 23 40 40 0a 00 00 00 00 00
(ada2:ahcich35:0:0:0): CAM status: Command timeout
(ada2:ahcich35:0:0:0): Retrying command, 3 more tries remain

Stux · December 22, 2024, 1:42am

Seems fairly concerning.

winnielinnie · December 22, 2024, 2:02am

Hardware?

NickF1227 · December 22, 2024, 3:03am

The hard drive ada2 is probably failing or has failed. It will need to be replaced. Theoretically it could be a bad SATA cable or even controller too. You can start the the cable, then the drive.

This message in particular indicates that CAM cannot even identify the drive any longer.

(aprobe0:ahcich35:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich35:0:0:0): CAM status: Command timeout

The system even tried to do a soft reset of the device and it failed

ahcich35: AHCI reset: device not ready after 31000ms (tfd = 00000080)

phier · December 22, 2024, 10:27am

Hello,
Thanks.

Is there a way to to find out the date when these messages were written into the log?

Also does it mean that drive is not active or part of my raid Z? Can I find out that somehow?

Thanks

NickF1227 · December 22, 2024, 4:07pm

Look at /var/log/messages and grep for the same message there, that file will have timestamps and includes the same information.

phier · December 22, 2024, 8:30pm

@NickF1227 thanks

I can see it’s happening like twice per Month?

Does it mean data is still there and its. Somehow works and has to be replaced immediately?

See Screenshoot.

Thanks!

phier · January 1, 2025, 1:50pm

So Shall I replace it immediately? Thanks

phier · January 5, 2025, 1:54pm

Regarding the replacement

I just disconnect and install new one, boot at it will be somehow maanged by OS?

Thx