LSI Equivalent Card Issues

TrueNAS General
,
1

I ultimately resolved my issue, but had a heck of a time resolving it. I’m posting here to capture some of what I’ve found in the hopes it saves some other people some time.

  • Issue
    • LSI “equivalent” card crashes within minutes of power up
  • Resolution
    • Return and get a genuine LSI card
  • Additional details
    • I went through the following 3 amazon orders
      • External PCI Express SAS/SATA HBA, SAS2008 Chip, X8, 6Gb/s, Same as 9200-8E
      • ipolex External PCI Express SAS/SATA HBA, SAS2008 Chip, 8-Port 6Gb/s, Same as 9200-8E
      • 6G External PCIe Controller Card, SAS/SATA HBA Host Bus Adapter, SAS 2008 Chip, PCIE X8, SAS 2.0, 2X Mini SAS SFF-8088 Ports,Same as 9200-8E
    • Each of them exhibited the same behavior: within minutes after boot, the cards would crash, spilling awful looking info into the system logs (provided below), at which point they would get removed from the PCI bus entirely. Within the first bit before crashing, they would begin to transfer files, but with horrendous amounts of retries slowing transfers to unacceptable rates.
    • Note also that some of these cards exhibited:
      • 1x wouldn’t reliably link at x8 PCIe width (so maybe this one should just be ruled out entirely since there were other issues with it…)
      • 2x wouldn’t reliably boot passed the option rom, preventing the system from booting entirely (although, looking back, I’m not sure if the 3rd just didn’t have the option rom loaded)
    • Note that 2 of these had the latest FW, except for one that I was not able to get anywhere close enough to a stable configuration where it would have been safe to try to reflash it (worried it would remove off the PCI bus mid programming)
    • Note that this happened regardless of whether there was a sas cable connected or not (most of the testing was done without the sas cable plugged in in order eliminate variables)
    • I saw from a few different places that some of these cards may be targeted for server chassis that have high airflow requirements. Because of this I tried pointing an internal fan at the heat spreader, which seemed to prevent the crashing from happening within 1-3 mins, but still ended up happening somewhere out in the dozens of minutes area… (note also that later on I ended up just pointing a DataVac at this to see if I could simulate the high airflow of a server rack on boot, but the card still crashed… do with that as you will…)
    • Hunted around for anything similar and found this link describing the issue

Provided below is part of the boot that exhibited the issue (lots of smartd and a few other subsystems removed for brevity; note for this run 1x HDD (sda) via expansion bay behind sas adaptor in question):

Jul 03 21:57:21 entropic kernel: sd 0:0:0:0: device_block, handle(0x000a)
Jul 03 21:57:23 entropic kernel: sd 0:0:0:0: device_unblock and setting to running, handle(0x000a)
Jul 03 21:57:23 entropic kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
Jul 03 21:57:23 entropic kernel: sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 03 21:57:23 entropic kernel: mpt2sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5e843b6100180f31)
Jul 03 21:57:23 entropic kernel: mpt2sas_cm1: removing handle(0x000a), sas_addr(0x5e843b6100180f31)
Jul 03 21:57:23 entropic kernel: mpt2sas_cm1: enclosure logical id(0x5e843b6100180f3f), slot(0)
...
Jul 03 21:57:53 entropic kernel: mpt2sas_cm1: handle(0xa) sas_address(0x5e843b6100180f31) port_type(0x1)
Jul 03 21:57:54 entropic kernel: scsi 0:0:2:0: Direct-Access     ATA      TOSHIBA MG04ACA6 FS4B PQ: 0 ANSI: 6
Jul 03 21:57:54 entropic kernel: scsi 0:0:2:0: SATA: handle(0x000a), sas_addr(0x5e843b6100180f31), phy(17), device_name(0x0000000000000000)
Jul 03 21:57:54 entropic kernel: scsi 0:0:2:0: enclosure logical id (0x5e843b6100180f3f), slot(0)
Jul 03 21:57:54 entropic kernel: scsi 0:0:2:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Jul 03 21:57:54 entropic kernel: scsi 0:0:2:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Jul 03 21:57:54 entropic kernel: sd 0:0:2:0: Power-on or device reset occurred
Jul 03 21:57:54 entropic kernel: sd 0:0:2:0: Attached scsi generic sg0 type 0
Jul 03 21:57:54 entropic kernel: sd 0:0:2:0: [sda] 1465130646 4096-byte logical blocks: (6.00 TB/5.46 TiB)
Jul 03 21:57:54 entropic kernel:  end_device-0:0:2: add: handle(0x000a), sas_addr(0x5e843b6100180f31)
Jul 03 21:57:54 entropic kernel: sd 0:0:2:0: [sda] Write Protect is off
Jul 03 21:57:54 entropic kernel: sd 0:0:2:0: [sda] Mode Sense: 7f 00 10 08
Jul 03 21:57:54 entropic kernel: sd 0:0:2:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 03 21:57:54 entropic kernel: sd 0:0:2:0: [sda] Attached SCSI disk
Jul 03 21:57:55 entropic kernel: mpt2sas_cm1: log_info(0x31120311): originator(PL), code(0x12), sub_code(0x0311)
Jul 03 21:57:55 entropic kernel: sd 0:0:2:0: [sda] tag#1442 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=0s
Jul 03 21:57:55 entropic kernel: sd 0:0:2:0: [sda] tag#1442 CDB: Read(10) 28 00 00 00 00 40 00 00 01 00
Jul 03 21:57:55 entropic kernel: I/O error, dev sda, sector 512 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
Jul 03 21:57:55 entropic kernel: sd 0:0:2:0: Power-on or device reset occurred
Jul 03 21:57:56 entropic kernel: mpt2sas_cm1: log_info(0x31120311): originator(PL), code(0x12), sub_code(0x0311)
Jul 03 21:57:57 entropic kernel: sd 0:0:2:0: Power-on or device reset occurred
Jul 03 21:57:57 entropic kernel: mpt2sas_cm1: log_info(0x31120311): originator(PL), code(0x12), sub_code(0x0311)
Jul 03 21:57:58 entropic kernel: sd 0:0:2:0: Power-on or device reset occurred
Jul 03 21:57:59 entropic kernel: mpt2sas_cm1: log_info(0x31120311): originator(PL), code(0x12), sub_code(0x0311)
Jul 03 21:57:59 entropic kernel: sd 0:0:2:0: [sda] tag#1408 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=0s
Jul 03 21:57:59 entropic kernel: sd 0:0:2:0: [sda] tag#1408 CDB: Read(10) 28 00 00 00 00 40 00 00 01 00
Jul 03 21:57:59 entropic kernel: I/O error, dev sda, sector 512 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
Jul 03 21:58:00 entropic kernel: sd 0:0:2:0: Power-on or device reset occurred
...
Jul 03 21:58:39 entropic kernel: sd 0:0:2:0: device_block, handle(0x000a)
Jul 03 21:58:41 entropic kernel: sd 0:0:2:0: device_unblock and setting to running, handle(0x000a)
Jul 03 21:58:41 entropic kernel: sd 0:0:2:0: [sda] Synchronizing SCSI cache
Jul 03 21:58:41 entropic kernel: sd 0:0:2:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 03 21:58:41 entropic kernel: mpt2sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5e843b6100180f31)
Jul 03 21:58:41 entropic kernel: mpt2sas_cm1: removing handle(0x000a), sas_addr(0x5e843b6100180f31)
Jul 03 21:58:41 entropic kernel: mpt2sas_cm1: enclosure logical id(0x5e843b6100180f3f), slot(0)
...
Jul 03 21:59:22 entropic kernel: mpt2sas_cm1: handle(0xa) sas_address(0x5e843b6100180f31) port_type(0x1)
Jul 03 21:59:23 entropic kernel: scsi 0:0:3:0: Direct-Access     ATA      TOSHIBA MG04ACA6 FS4B PQ: 0 ANSI: 6
Jul 03 21:59:23 entropic kernel: scsi 0:0:3:0: SATA: handle(0x000a), sas_addr(0x5e843b6100180f31), phy(17), device_name(0x0000000000000000)
Jul 03 21:59:23 entropic kernel: scsi 0:0:3:0: enclosure logical id (0x5e843b6100180f3f), slot(0)
Jul 03 21:59:23 entropic kernel: scsi 0:0:3:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Jul 03 21:59:23 entropic kernel: scsi 0:0:3:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Jul 03 21:59:23 entropic kernel: sd 0:0:3:0: Power-on or device reset occurred
Jul 03 21:59:23 entropic kernel: sd 0:0:3:0: Attached scsi generic sg0 type 0
Jul 03 21:59:23 entropic kernel: sd 0:0:3:0: [sda] 1465130646 4096-byte logical blocks: (6.00 TB/5.46 TiB)
Jul 03 21:59:23 entropic kernel:  end_device-0:0:3: add: handle(0x000a), sas_addr(0x5e843b6100180f31)
Jul 03 21:59:23 entropic kernel: sd 0:0:3:0: [sda] Write Protect is off
Jul 03 21:59:23 entropic kernel: sd 0:0:3:0: [sda] Mode Sense: 7f 00 10 08
Jul 03 21:59:23 entropic kernel: sd 0:0:3:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 03 21:59:24 entropic kernel: sd 0:0:3:0: [sda] Attached SCSI disk
...
Jul 03 22:03:21 entropic kernel: mpt2sas_cm1: SAS host is non-operational !!!!
Jul 03 22:03:22 entropic kernel: mpt2sas_cm1: SAS host is non-operational !!!!
Jul 03 22:03:23 entropic kernel: mpt2sas_cm1: SAS host is non-operational !!!!
Jul 03 22:03:24 entropic kernel: mpt2sas_cm1: SAS host is non-operational !!!!
Jul 03 22:03:25 entropic kernel: mpt2sas_cm1: SAS host is non-operational !!!!
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: SAS host is non-operational !!!!
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: _base_fault_reset_work: Running mpt3sas_dead_ioc thread success !!!!
Jul 03 22:03:26 entropic kernel: sd 0:0:3:0: [sda] Synchronizing SCSI cache
Jul 03 22:03:26 entropic kernel: sd 0:0:3:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5e843b6100180f3d)
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: removing handle(0x000b), sas_addr(0x5e843b6100180f3d)
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: enclosure logical id(0x5e843b6100180f3f), slot(255)
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5e843b6100180f31)
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: removing handle(0x000a), sas_addr(0x5e843b6100180f31)
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: enclosure logical id(0x5e843b6100180f3f), slot(0)
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5e843b6100180f3f)
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: expander_remove: handle(0x0009), sas_addr(0x5e843b6100180f3f), port:255
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: unexpected doorbell active!
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: sending diag reset !!
...
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: Invalid host diagnostic register value
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: System Register set:
Jul 03 22:03:26 entropic kernel: 00000000: ffffffff
Jul 03 22:03:26 entropic kernel: 00000004: ffffffff
Jul 03 22:03:26 entropic kernel: 00000008: ffffffff
Jul 03 22:03:26 entropic kernel: 0000000c: ffffffff
Jul 03 22:03:26 entropic kernel: 00000010: ffffffff
Jul 03 22:03:26 entropic kernel: 00000014: ffffffff
Jul 03 22:03:26 entropic kernel: 00000018: ffffffff
Jul 03 22:03:26 entropic kernel: 0000001c: ffffffff
Jul 03 22:03:26 entropic kernel: 00000020: ffffffff
Jul 03 22:03:26 entropic kernel: 00000024: ffffffff
Jul 03 22:03:26 entropic kernel: 00000028: ffffffff
Jul 03 22:03:26 entropic kernel: 0000002c: ffffffff
Jul 03 22:03:26 entropic kernel: 00000030: ffffffff
Jul 03 22:03:26 entropic kernel: 00000034: ffffffff
Jul 03 22:03:26 entropic kernel: 00000038: ffffffff
Jul 03 22:03:26 entropic kernel: 0000003c: ffffffff
Jul 03 22:03:26 entropic kernel: 00000040: ffffffff
Jul 03 22:03:26 entropic kernel: 00000044: ffffffff
Jul 03 22:03:26 entropic kernel: 00000048: ffffffff
Jul 03 22:03:26 entropic kernel: 0000004c: ffffffff
Jul 03 22:03:26 entropic kernel: 00000050: ffffffff
Jul 03 22:03:26 entropic kernel: 00000054: ffffffff
Jul 03 22:03:26 entropic kernel: 00000058: ffffffff
Jul 03 22:03:26 entropic kernel: 0000005c: ffffffff
Jul 03 22:03:26 entropic kernel: 00000060: ffffffff
Jul 03 22:03:26 entropic kernel: 00000064: ffffffff
Jul 03 22:03:26 entropic kernel: 00000068: ffffffff
Jul 03 22:03:26 entropic kernel: 0000006c: ffffffff
Jul 03 22:03:26 entropic kernel: 00000070: ffffffff
Jul 03 22:03:26 entropic kernel: 00000074: ffffffff
Jul 03 22:03:26 entropic kernel: 00000078: ffffffff
Jul 03 22:03:26 entropic kernel: 0000007c: ffffffff
Jul 03 22:03:26 entropic kernel: 00000080: ffffffff
Jul 03 22:03:26 entropic kernel: 00000084: ffffffff
Jul 03 22:03:26 entropic kernel: 00000088: ffffffff
Jul 03 22:03:26 entropic kernel: 0000008c: ffffffff
Jul 03 22:03:26 entropic kernel: 00000090: ffffffff
Jul 03 22:03:26 entropic kernel: 00000094: ffffffff
Jul 03 22:03:26 entropic kernel: 00000098: ffffffff
Jul 03 22:03:26 entropic kernel: 0000009c: ffffffff
Jul 03 22:03:26 entropic kernel: 000000a0: ffffffff
Jul 03 22:03:26 entropic kernel: 000000a4: ffffffff
Jul 03 22:03:26 entropic kernel: 000000a8: ffffffff
Jul 03 22:03:26 entropic kernel: 000000ac: ffffffff
Jul 03 22:03:26 entropic kernel: 000000b0: ffffffff
Jul 03 22:03:26 entropic kernel: 000000b4: ffffffff
Jul 03 22:03:26 entropic kernel: 000000b8: ffffffff
Jul 03 22:03:26 entropic kernel: 000000bc: ffffffff
Jul 03 22:03:26 entropic kernel: 000000c0: ffffffff
Jul 03 22:03:26 entropic kernel: 000000c4: ffffffff
Jul 03 22:03:26 entropic kernel: 000000c8: ffffffff
Jul 03 22:03:26 entropic kernel: 000000cc: ffffffff
Jul 03 22:03:26 entropic kernel: 000000d0: ffffffff
Jul 03 22:03:26 entropic kernel: 000000d4: ffffffff
Jul 03 22:03:26 entropic kernel: 000000d8: ffffffff
Jul 03 22:03:26 entropic kernel: 000000dc: ffffffff
Jul 03 22:03:26 entropic kernel: 000000e0: ffffffff
Jul 03 22:03:26 entropic kernel: 000000e4: ffffffff
Jul 03 22:03:26 entropic kernel: 000000e8: ffffffff
Jul 03 22:03:26 entropic kernel: 000000ec: ffffffff
Jul 03 22:03:26 entropic kernel: 000000f0: ffffffff
Jul 03 22:03:26 entropic kernel: 000000f4: ffffffff
Jul 03 22:03:26 entropic kernel: 000000f8: ffffffff
Jul 03 22:03:26 entropic kernel: 000000fc: ffffffff
Jul 03 22:03:26 entropic kernel: mpt2sas_cm1: diag reset: FAILED
2

Yeah, cheap != better at all!

Hopefully, for each of the failed items, you left a review on Amazon with the word truenas in it and “doesn’t work”. That can save even more people trouble.

3

Well what’s funny was both the replacement cards were cheaper than the amazon ones… Sigh… And I get it. It was a knock off card – I’m used to kludging stuff together and am ready for a failure should one go down… But I would at least expect them to… Ya know… be up and operational for any length of time at all. I did leave reviews on all 3 warning others too, because man… What a waste of time. :slightly_smiling_face: