LSI 9300 16I flashed to IT mode but not recognized by sas3flash in Truenas

I’ve spent hours digging through forums trying various things but I am struggling to find the right answer. I am building my first Truenas system (SCALE). Installation was no issue but it seems the OS is unable to properly see my HBA card. The card is LSI SAS 9300-16I. I successfully flashed the card in IT mode. I tried with and without including the unnecessary bits that do things like provide the build in boot gui but it made no difference in Truenas seeing the hba.

I had no issues with flashing by booting into shell and using sas3flash. The card shows up as 2 controllers (2 X 8 port) and i did flash both of them to the recommended version for Truenas which is of course also IT mode.

Working backwards from not being able to see any drives connected to the controller in Truenas I assume it is related to truenas not seeing the card properly. This is the output from sas3flash after ssh to the OS
"
truenas_admin@truenas[~]$ sudo sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

    No Avago SAS adapters found! Limited Command Set Available!
    ERROR: Command Not allowed without an adapter!
    ERROR: Couldn't Create Command -list
    Exiting Program.

"
This is the relative output from lscpci
"
04:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
06:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
"

This is the relative output from lscpci -vv
"
Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
Subsystem: Broadcom / LSI SAS 9300-16i
!!! Unknown header type 7f
Interrupt: pin ? routed to IRQ 17
IOMMU group: 17
Region 0: I/O ports at 4000 [size=256]
Region 1: Memory at 81440000 (64-bit, non-prefetchable) [size=64K]
Region 3: Memory at 81400000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at 81300000 [disabled] [size=1M]
Kernel modules: mpt3sas

Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
Subsystem: Broadcom / LSI SAS 9300-16i
!!! Unknown header type 7f
Interrupt: pin ? routed to IRQ 18
IOMMU group: 18
Region 0: I/O ports at 3000 [size=256]
Region 1: Memory at 81240000 (64-bit, non-prefetchable) [size=64K]
Region 3: Memory at 81200000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at 81100000 [disabled] [size=1M]
Kernel modules: mpt3sas
"
Photos of sas3flash working from thumb drive. In these photos I don’t currently have the unnecessary bits flashed as per recommendation in some forums.

Same photos of sas3flash working from thumb drive but now with unnecessary bios flashed which makes no difference in operability with Truenas OS

I flashed with 2 versions of mpt files from different sources but my understanding is that bios doesn’t matter here anyway.

The most useful guides/resources I used.

There was one more but I have to find it again.

As a comparison, here’s what my fully flashed 9300-8i looks like:

% sudo sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02) 
Copyright 2008-2017 Avago Technologies. All rights reserved.

	Adapter Selected is a Avago SAS: SAS3008(C0)

	Controller Number              : 0
	Controller                     : SAS3008(C0)
	PCI Address                    : 00:01:00:00
	SAS Address                    : 5001234-1-1234-e123
	NVDATA Version (Default)       : 0e.01.00.07
	NVDATA Version (Persistent)    : 0e.01.00.07
	Firmware Product ID            : 0x2221 (IT)
	Firmware Version               : 16.00.12.00
	NVDATA Vendor                  : LSI
	NVDATA Product ID              : SAS9300-8i
	BIOS Version                   : 08.37.02.00
	UEFI BSD Version               : 18.00.00.00
	FCODE Version                  : N/A
	Board Name                     : SAS9300-8i
	Board Assembly                 : CP400i-IT
	Board Tracer Number            : 12345678

	Finished Processing Commands Successfully.
	Exiting SAS3Flash.

SAS address and board tracer number anonymised because why not.

I note a few fields listed as N/A on your card that are populated on mine.

  1. Care to show the exact steps you followed when flashing?
  2. Also, are you booting in legacy BIOS mode or UEFI mode?
  3. Did you doublecheck the output when flashing, that there were no errors reported?

Editing your OP to add answers makes it difficult to follow since I would have to read your whole post to see what you replied with. And I rather not do that.

Anyway, this could also be a thermal issue, the card overheats. That would possibly explain the !!! Unknown header type 7f in your lspci.

Your response was faster than I expected as I was still adding to my post. Some of the NA is now explained which is BIOS stuff I understand is unnecessary. Regardless I just reflashed it to include those for the sake of having all of the photos of my efforts. I still have a couple of N/A but I don’t know that those things have any effect on what I am experiencing. I don’t have any errors when flashing. I followed the steps here. GitHub - EverLand1/9300-8i_IT-Mode: A guide on how to flash IT mode firmware to your LSI Avago 9300-8i RAID Controller, making it an HBA, or Host Bus Adapter.. Based on learnings from guides I did flash both c0 and c1 otherwise following the guide only flashes c0 I presume because it defaults to only the first adapter found. (I only have 1 card but it operates as 2 x 8 port)

Sorry, I literally just posted and had no idea someone would review so quickly. I was really trying to make it LESS effort for folks trying to assist by being complete and answering questions up front. I am done editing my initial post though

That’s fine, I jumped the gun.

That would be rather odd to overheat immediately after boot with no drives attached and extreme cooling in my rig which includes 2 fans adjacent to the card blowing directly on the heatsink. If it is in fact overheating then I think the card would have to be faulty. I know these run HOT but I prepared for that and I have not even asked anything of it yet. I will start researching the possibility though. “!!! Unknown header type 7f” and see if there is any action I can take. Currently I would have to assume the card is faulty if it is overheating so quickly

The problem is solved.

I will update with how in just a few minutes in case anyone else find their way to this post later. I wanted to update that this is solved right away thought so that nobody is wasting their time assisting. I want to retrace my steps just to confirm before I post what the solution was.

Thank you neofusion, ultimately your response calling out the abnormality “!!! Unknown header type 7f” led me down the path to resolution.

My digging around this unknown header type landed with this message being primarily associated with GPU cards and possibly overheating. I believe though that it doesn’t necessarily mean it is an overheating issue but overheating can cause this particularly with GPUs.

What I did, I powered off my system and found that my card was not even warm. Next I disconnected the aux power plug to the card and made sure no drives/backplanes were connected to it. (no change in outcome)

What worked: I moved the card to another slot. The slots are identical at least in appearance and I have no other cards installed. Just to make sure this wasn’t the effect of simply reseating the card, I put it back in the original slot and the issue returned.

The card is now detected within Truenas as I would expect and the connected drives are detected as well.

This is now the output of the same commands in my initial post.
"
03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
Subsystem: Broadcom / LSI SAS 9300-16i
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
IOMMU group: 16
Region 0: I/O ports at 4000 [size=256]
Region 1: Memory at 81440000 (64-bit, non-prefetchable) [size=64K]
Region 3: Memory at 81400000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at 81300000 [disabled] [size=1M]
Capabilities:
Kernel driver in use: mpt3sas
Kernel modules: mpt3sas

05:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
Subsystem: Broadcom / LSI SAS 9300-16i
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 17
IOMMU group: 17
Region 0: I/O ports at 3000 [size=256]
Region 1: Memory at 81240000 (64-bit, non-prefetchable) [size=64K]
Region 3: Memory at 81200000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at 81100000 [disabled] [size=1M]
Capabilities:
Kernel driver in use: mpt3sas
Kernel modules: mpt3sas
"

"
truenas_admin@truenas[~]$ sudo sas3flash -listall
[sudo] password for truenas_admin:
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

    Adapter Selected is a Avago SAS: SAS3008(C0)

Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr

0 SAS3008(C0) 16.00.12.00 0e.01.00.03 08.37.00.00 00:03:00:00
1 SAS3008(C0) 16.00.12.00 0e.01.00.03 08.37.00.00 00:05:00:00
"

"
truenas_admin@truenas[~]$ sudo sas3flash -c 0 -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

    Adapter Selected is a Avago SAS: SAS3008(C0)

    Controller Number              : 0
    Controller                     : SAS3008(C0)
    PCI Address                    : 00:03:00:00
    SAS Address                    : 500062b-2-014d-0580
    NVDATA Version (Default)       : 0e.01.00.03
    NVDATA Version (Persistent)    : 0e.01.00.03
    Firmware Product ID            : 0x2221 (IT)
    Firmware Version               : 16.00.12.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9300-16i
    BIOS Version                   : 08.37.00.00
    UEFI BSD Version               : 18.00.00.00
    FCODE Version                  : N/A
    Board Name                     : SAS9300-16i
    Board Assembly                 : N/A
    Board Tracer Number            : N/A

    Finished Processing Commands Successfully.
    Exiting SAS3Flash.

"

"
truenas_admin@truenas[~]$ sudo sas3flash -c 1 -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

    Adapter Selected is a Avago SAS: SAS3008(C0)

    Controller Number              : 1
    Controller                     : SAS3008(C0)
    PCI Address                    : 00:05:00:00
    SAS Address                    : 500062b-2-014d-0d00
    NVDATA Version (Default)       : 0e.01.00.03
    NVDATA Version (Persistent)    : 0e.01.00.03
    Firmware Product ID            : 0x2221 (IT)
    Firmware Version               : 16.00.12.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9300-16i
    BIOS Version                   : 08.37.00.00
    UEFI BSD Version               : 18.00.00.00
    FCODE Version                  : N/A
    Board Name                     : SAS9300-16i
    Board Assembly                 : N/A
    Board Tracer Number            : N/A

    Finished Processing Commands Successfully.
    Exiting SAS3Flash.

"

1 Like

Thanks for the followup.

You may have a dodgy PCI-slot, or there might be BIOS settings that are different/incorrect for that specific slot.

Either way, well done troubleshooting this issue.