Fault 2668h installing LSI9305-16i on a machine running TrueNAS on proxmox with PCI passthrough

Darkhog · February 23, 2025, 11:32pm

TL;DR : I bought a Broadcom LSI 9305-16i card from Amazon, it came with IT firmware 13, I updated to firmware 16 but also updated the BIOS, which broke non-interactive boot. Do NOT update the card Bios, but if you do you can learn from my mistake on how to fix it.

Purchased a Broadcom LSI 9305-16i on Amazon and sold by Aeon Micro, Inc., received the card and running sas3flash -list in TrueNAS will show the Firmware version and if it is in IT mode. Here are the relevant parts of that, showing that it is on old version 13 (when version 16.00.12.00 was released in December 2020) -

	Firmware Product ID: 0x2228 (IT)
	Firmware Version: 13.00.00.00
	NVDATA Vendor: LSI
	NVDATA Product ID: SAS9305-16i
	BIOS Version: 08.31.00.00
	UEFI BSD Version: 13.00.00.00
	Board Name : SAS9305-16i

I wanted the current Firmware Version, which is 16.00.12.00, so downloaded the firmware from Broadcom, Legacy items, Legacy Host Bus Adapters, Firmware, which was this file - SAS9305_16i_IT_P.bin - found in 9305_16i_Pkg_P16.12_IT_FW_BIOS_for_MSDOS_Windows in the Firmware directory, and flashed via sas3flash -c 0 -f SAS9305_16i_IT_P.bin which flash completed but hung the adapter on “Resetting Adapter…”, but after reboot it was on proper firmware, all good.

But then I made the poor decision to also update the Bios. I didn’t realize the Bios doesn’t matter in TrueNas, it is used if you are not in IT mode and you want to boot from one of the disks attached, which I did not and you probably don’t either. I made the mistake of updating the Bios (from 08.31.00.00 to 08.37.02.00, and UEFI from 13.00.00.00 to 18.00.03.00), which caused TrueNAS to pause on boot with error:

	Unable to load Avago Technologies 
	MPT BIOS MPT BIOS Fault 02h encountered at adapter PCI(01h,00h,00h) 
	Firmware Fault Code: 2668h 
	Press any key to continue…

Drives still worked, I just needed to hit a key on every boot. To fix that, I needed to clear that bios completely using the command sas3flash -c 0 - o -e 5 which unfortunately wouldn’t run directly in TrueNas.

    Executing Operation: Erase Flash

    Erasing Boot Services Region...

            Erase Flash Command not Supported on this platform.

    Firmware Returned Exception. IOCStatus=0x1, IOCLogInfo=0x0
    Due to error remaining commands will not be executed.
    Unable to Process Commands.
    Exiting SAS3Flash.

Figuring out how to do this via EUFI boot disk was a chore, and there are probably many ways to do it, but I thought I would document it here so others could benefit.

I needed to find a USB drive, format it with FAT32, download Shell.efi from the tianocore edk2-archive ShellBinPkg for x64, put it at \EFI\BOOT\ and rename it to BOOTX64.efi on the USB, then put sas3flash.efi from the SAS3FLASH_P15 package, sas3flash_udk_uefi_x64_rel directory at the root. Once in Shell, shift wouldn’t work with my keyboard, so I needed to type : then arrow left, then fs0 to get fs0: to select my usb drive (map -r in Shell to show drives, your usb may not be fs0). Once there, I could run sas3flash.efi -list or any other command, including the command to erase the bios (sas3flash.efi -c 0 -o -e 5 for mine, on controller 0), after which sas3flash.efi -list showed -

	Firmware Product ID: 0x2228 (IT)
	Firmware Version: 16.00.12.00
	NVDATA Vendor: LSI
	NVDATA Product ID: SAS9305-16i
	BIOS Version: N/A
	UEFI BSD Version: N/A
	FCODE Version: N/A

It was a journey. If it just came with the version 16 of the firmware, which was released in December 2020, I would have spent a lot less time getting this card to work and be on current (4 years old) firmware. But unfortunately I think these cards come with a variety of firmware, not always the current, and hopefully this helps your journey be shorter.

Darkhog · March 20, 2025, 7:24pm

Adding a couple other quick things I learned and modified as I get rolling with this card - maybe this helps someone else.

A number of places I see this card people talk about adding a fan to keep it cool. Reading the data sheets, it seems like this card is meant to have air flow over it as part of case airflow. I don’t know what a good temperature is, but I’d like to keep is reasonable. 80C seems too high, under 65C is probably fine…?

In my testing of installing it stock, with no fan, the card got up to 80C. Pointing a fan in the direction of the card it easily comes down to 55C. My case doesn’t have room for a fan installed on the heatsink, as the card is just above the power supply. As a workaround I bought 15x15x15 heatsinks with thermal conductive adhesive tape. I then taped additional heat sinks wherever I could fit them, and oriented them to allow the most airflow ‘up’. Adding those additional heatsinks and adding no fans brings the temp down from 80C to 70C.

My card is right above my Seasonic power supply, which while in hybrid mode doesn’t turn on the fan. If I turn off hybrid mode it turns the fan on and with just that draw, the card now comes down to 56C, which seems great.

To get the temperature of the card, I run mprutil show adapter which has the temp in there. I used chatgpt to build a simple script that runs on a 5 minute cron job and logs the temp. If it is above a threshold (currently 75C) also logs an alert and sends a mail. Here are some pictures of my modifications, as well as what the stock card looks like.