Sporadic power-on resets for various drives

TrueNAS General
,
1

Hello!

I built out a new storage server and I’m having some problems with it. The server will run fine for days (even up to a month!) but then I will start to constantly see power-on resets in dmesg. The resets are followed by a 30s io stall until something times out, and then everything seems to recover. The resets are also sometimes followed by read/write/checksum errors (sometimes all three!) on one or more drives in the zpool.

These are 8 new 10Tb SAS drives, and the errors don’t seem to follow the actual drives. I see no errors in the SMART reports for any drives

I’m using an LSI9300-16i with two MiniSAS HD to quad SATA cables which are connected to the SATA ports on a Supermicro BPN SAS-747TQ backplane. I don’t have sideband hooked up to the backplane - is this relevant at all?

Both controllers on the LSI9300 have been updated to FW version 16.00.12.00 - the special version released on the TrueNAS forums which, from the patch notes, purpoted to fix a similar issue with SATA drives.

I’m not sure how much it matters but this server has a SuperMicro X11SSZ-TLN4F and 64Gb of ECC memory. I’m using a 750W power supply, which I think should be plenty, but I have also tested a different supply to rule that out as the source of the issue.

I was originally running TrueNAS as a VM in Proxmox (which I have been doing on my old server for close to a decade without any issues). I thought that I had root-caused these issues to something to do with Proxmox, because the resets seemed to be correlated to the host kernel seeing a stall and reducing a perf systcl. Manually adjusting that sysctl back to the default value seemed to abate the issue for a while - seems like that was a red herring.

I have since switched to running TrueNAS direclty on the host but, unfortunately, that hasn’t helped at all.

Currently, this is running TrueNAS Scale 25.10.7.

My only remaining ideas are:

  1. move the LSI9300 to a different PCIe port
  2. swap out the LSI9300

I will try 1. today, but I thought I would post this in case anybody has any other ideas, given that nothing else I’ve tried has helped at all.

I would really appreciate any ideas re: root-cause, or any suggestions for debugging!

Thanks!

2

sporadic por across a pile of sas→sata breakout runs usually lands on cable/backplane/power for me more than the drives themselves

id try:
-confirm those minisas hd → 4x sata leads are the right breakout type for an lsi it hba. wrong ones throw random link resets
-run one cable / 4 drives overnight. if the por follows the cable not the bay, swap that lead
-sideband on the 747tq is mostly blink/locate. data path can work without it, so leave that for later
-keep dmesg -w up and smartctl -x a drive that just reset. look for command timeout / phy reset storms
-move the 9300 to another pcie slot like u planned. keep it off a riser if its on one

3

I tried moving the hba to a different PCIe slot but immediately saw PORs on boot and when I resumed the scrub on the pool. I figured it would take days to start to show up again, but I guess not.

Since that didn’t work, I went ahead and swapped the HBA to a SAS9211-8i, which uses MiniSAS → 4x SATA instead of the MiniSAS HD cables I was using before, and I still see the PORs on boot.

Which, I guess, leaves only the backplane. Huge bummer because I designed a lot of custom parts around this backplane :confused:

Any way I can confirm for sure that it’s the backplane? This already seems pretty conclusive to me but I don’t really want to buy a direct replacement, and re-designing all my parts for a different backplane is going to suck.

In that same vein, any recommendations for a different backplane? I probably should have gone with one that has MiniSAS-HD connectors instead of SATA, which I will try to do this time around…

4

if a 9211 still por on boot on the same backplane, yeah that pretty much pins the bpn.

cheapest confirm without buying another backplane:
-yank the minisas→sata leads off the 747tq
-run 2 drives on a plain sata breakout or motherboard ports overnight w/ a scrub
-if por stops off the backplane, u found it

sideband still wont fix link resets. for a swap later look for a sas3 backplane w/ minisas hd (sff-8643) so u can drop the sata adapter leads entirely.

5

The Supermicro BPN SAS-747TQ backplane is a passive backplane. It is unlikely there is an issue with the backplane. it’s likely power or cables Make sure you are not using a reverse breakout cable from the card to the backplane. There are Forward cables meant to go from the HBA to the backplane (or individual drives), and reverse cables meant to go from 4 motherboard ports to a single connector on a backplane They look the same but are wired different.

6

Despite the fact that I’ve switched cables multiple times, (to your point @PhilD13) I still feel like it’s the cables… I spent some time yesterday swapping cables after switching back to the 9300 and now the errors are affecting entirely different drives - which really seems to point to the cables.

Does anybody have any recommendations for where to shop for these cables? I think most of the ones I have I’ve gotten from Amazon or eBay. That’s what I’ve always done and I’ve only started to have problems recently, maybe I just got lucky all those years. Cables are definitely the cheapest thing to try to change, so it probably makes sense to try that first.

7

The Art of Server guy has an ebay site that has tested and already setup for Truenas HBA’s and cables. Other things too. It’s where I get my cards and cables.

Art of Server

8

Great, thanks for the suggestion! I ordered a couple of cables from there, should be here in a few days.

In the mean time, I went out and wiggled the cables at the backplane while watching dmesg and the PORs almost immediately went away lol. Who knows how long that’ll hold up. It was enough for resilvering to finish, at least, so that’s nice.

9

I replaced the cables with a set from Art of Server and at first it seemed pretty promising, but the issue has returned after a few days of not seeing it

Lots of these being spammed:

[Oct 6 13:17] sd 0:0:3:0: device_block, handle(0x000a)
[  +1.001617] sd 0:0:3:0: device_unblock and setting to running, handle(0x000a)
[  +0.205490] sd 0:0:3:0: Power-on or device reset occurred
[Oct 6 13:18] sd 0:0:3:0: device_block, handle(0x000a)
[  +1.001580] sd 0:0:3:0: device_unblock and setting to running, handle(0x000a)
[  +0.197080] sd 0:0:3:0: Power-on or device reset occurred
[Oct 6 13:20] sd 0:0:3:0: device_block, handle(0x000a)
[  +1.001391] sd 0:0:3:0: device_unblock and setting to running, handle(0x000a)
[  +0.186942] sd 0:0:3:0: Power-on or device reset occurred
[Oct 6 13:21] sd 0:0:3:0: device_block, handle(0x000a)
[  +1.001470] sd 0:0:3:0: device_unblock and setting to running, handle(0x000a)
[  +0.205738] sd 0:0:3:0: Power-on or device reset occurred

with these seemingly worse ones peppered in

[ +26.355456] sd 0:0:3:0: attempting task abort!scmd(0x0000000024a4191c), outstanding for 30612 ms & timeout 30000 ms
[  +0.000013] sd 0:0:3:0: [sdd] tag#373 CDB: Read(16) 88 00 00 00 00 00 8c 2e 63 50 00 00 07 b8 00 00
[  +0.000004] scsi target0:0:3: handle(0x000a), sas_address(0x5000c500ca258f55), phy(3)
[  +0.000008] scsi target0:0:3: enclosure logical id(0x500062b201460900), slot(1)
[  +0.000005] scsi target0:0:3: enclosure level(0x0000), connector name(     )
[  +0.004218] sd 0:0:3:0: task abort: SUCCESS scmd(0x0000000024a4191c)
[  +0.000020] sd 0:0:3:0: attempting task abort!scmd(0x00000000eda7e6d4), outstanding for 30612 ms & timeout 30000 ms
[  +0.000007] sd 0:0:3:0: [sdd] tag#348 CDB: Read(16) 88 00 00 00 00 00 8c 2e 6b 10 00 00 07 b8 00 00
[  +0.000004] scsi target0:0:3: handle(0x000a), sas_address(0x5000c500ca258f55), phy(3)
[  +0.000007] scsi target0:0:3: enclosure logical id(0x500062b201460900), slot(1)
[  +0.000005] scsi target0:0:3: enclosure level(0x0000), connector name(     )
[  +0.003355] sd 0:0:3:0: task abort: SUCCESS scmd(0x00000000eda7e6d4)
[  +0.000062] sd 0:0:3:0: attempting task abort!scmd(0x000000003d204396), outstanding for 30612 ms & timeout 30000 ms
[  +0.000010] sd 0:0:3:0: [sdd] tag#322 CDB: Read(16) 88 00 00 00 00 00 8c 2e 7a 50 00 00 00 30 00 00
[  +0.000005] scsi target0:0:3: handle(0x000a), sas_address(0x5000c500ca258f55), phy(3)
[  +0.000008] scsi target0:0:3: enclosure logical id(0x500062b201460900), slot(1)
[  +0.000006] scsi target0:0:3: enclosure level(0x0000), connector name(     )
[  +0.003545] sd 0:0:3:0: task abort: SUCCESS scmd(0x000000003d204396)
[  +0.000037] sd 0:0:3:0: attempting task abort!scmd(0x0000000064f02d79), outstanding for 30612 ms & timeout 30000 ms
[  +0.000002] sd 0:0:3:0: [sdd] tag#321 CDB: Read(16) 88 00 00 00 00 00 8c 2e 72 c8 00 00 07 88 00 00
[  +0.000002] scsi target0:0:3: handle(0x000a), sas_address(0x5000c500ca258f55), phy(3)
[  +0.000002] scsi target0:0:3: enclosure logical id(0x500062b201460900), slot(1)
[  +0.000001] scsi target0:0:3: enclosure level(0x0000), connector name(     )
[  +0.003684] sd 0:0:3:0: task abort: SUCCESS scmd(0x0000000064f02d79)

This has just started happening again so it’s hard to say for sure how many drives it will hit, but so far it is only affecting a single drive - one of the drives that was previously problematic. Does this seem like something that might be coming from the drive at this point?