My experience buying and flashing a used LSI HBA (SAS9207-8i, aka SAS9208-8i, aka 2308) to replace my "dodgy port multiplier"

axhxrx · April 27, 2024, 7:40am

So, after the recent Dragonfish upgrade, my cheap no-name SATA port multiplier card stopped working. This problem was reported by multiple people and is being tracked as bug NAS-128478 by the TrueNAS engineers, but in the meantime I read through a lot of discussion about SATA port multipliers vs host bus adapters (HBAs), and in particular the “legendary” (according to the docs) LSI HBAs.

Although I think using junk hardware in general is a fine and wonderful thing, I found several persuasive arguments (like this one) that a used HBA was a much better value than a cheap port multiplier, and was also less likely to encounter various problems similar to the one that I was currently dealing with.

So, I decided to replace my port multiplier card with a used LSI HBA. This post is the documentation of my experience, which contains all the info that I would have liked to find in one place, rather than the various places I did find it. It’s just my experience, which was successful, but any corrections are welcome.

Summary

Find a used card somewhere online that seems like it isn’t fake, and order it
Order new cables to go from the HBA to the drives
Install the card, which in my case required modifying the card’s hardware slightly
Obtain the correct firmware and BIOS for the card from Broadcom (which LSI became part of)
Use the tool built into TrueNAS SCALE to upgrade the firmware and BIOS of the card, and make sure it is in “IT mode” (and not “RAID mode”)
Reconnect all the drives and make sure they all work
Celebrate with a frosty beverage

Find a used card somewhere online that seems like it isn’t fake, and order it

While many people recommend eBay as a source for used LSI HBAs, I am in Japan and the USD prices and shipping were too expensive for me, so I found a third party seller listing on Amazon Japan, for what purported to be a " 9208 8i Multiport PCI Express 3.0 SAS SATA HBA High Speed 6Gb HBA Card 8 Port Fibre Channel Adapter Interface Controller Card for Linux". Curiously, the LSI logo seems to have been photoshopped out.

I really wanted the “9207-8i” because I found a lot of positive comments about that card. But I took a chance and ordered this one, and after it arrived I found out that:

indeed, it has the LSI logo on the board, even though it is removed from the photos on the listing, and
although the “board name” of this product is “SAS9208-8i” the “NVDATA Product ID” of it is “SAS9207-8i”. So I presume it is roughly the same thing with different marketing/business details. (Not 100% sure about that, but it seems right.)

I bought it for JPY 4,696, which is currently USD 29.94.

Order new cables to go from the HBA to the drives

Most HBA cards like this come with one SFF-8087 connector port per 4 SATA connections, so single-SFF-8087 to quadruple-SATA cables are common. I bought 2 of these cables, since the SAS9208-8i has 2 such ports, so that I would be able to connect the 8 SATA drives that had been connected to my port multiplier card (which just had 8 regular SATA ports on it). Thus, these two new cables replace 8 regular SATA cables.

Install the card, which in my case required modifying the card’s hardware slightly

The card fits into a regular PCI slot, just like the outgoing port multiplier. However, the card is designed for a lower-profile rackmount server, while my home NAS is a trasher gaming PC from 2018. So the card could not be installed because the metal tab used to screw the card to the case to secure it was too low, and too wide to stick out through the back of the PC case. This problem is specific to my case, and would happen with some cases and not others.

So I unscrewed the metal tab and removed it from the card. This makes the card less stable, though. In my case, I superglued an old corn-cob pipe to the back of the PC case in a way that the stem of the pipe holds the edge of the card in place and prevents it from easily coming out of the motherboard PCI slot. That fits with the aesthetic of my own machine, but there are videos online about how to “convert a low-profile PCI card backplate” and various ways to do it, if you need something sturdier.

I did not connect any drives at this stage, because I wanted to upgrade the firmware and BIOS before doing so.

Obtain the correct firmware and BIOS for the card from Broadcom (which LSI became part of)

I went to the Broadcom website’s “Support Documents and Downloads” page to find the firmware. At first I did not find any matches, but then I noticed there is a checkbox to “Include Legacy Products” that isn’t initially visible, but appears after you start searching. I searched for Product Group “Storage Adapters, Controllers, and ICs” and then within that category, “Host Bus Adapters”.

Even after clicking the “Include Legacy Products” box, I didn’t find anything for “9208-8i” which is what the sticker on the actual PCI card itself read. Instead the search UI returned hundreds of hits that did not appear to be related.

So, I went to the main root of the website, www.broadcom.com, and entered “9208-8i” in the search box at the top right. This yielded 4,321 results and I didn’t see anything useful on the first page. Since I had learned that the “9207-8i” was a common card, and was probably very similar to or identical to this card, I changed the search and tried again.

Bingo! I found a link like this:

[zip] 9207-8i_Package_P20_IR_IT_Firmware_BIOS_for_MSDOS_Windows
Package_P20_Firmware_BIOS_for_MSDOS_Windows
DATE: 04/04/2016 SIZE: 1737 KB

I downloaded this file (have to agree to terms in the web UI, so I couldn’t just use wget to download it from the CLI).

(BTW, the reason these files are on the Broadcom website is that the original LSI company was acquired by Avago in 2014, and then they also acquired Broadcom, but started using the name Broadcom for more of their stuff.)

Use the tool built into TrueNAS SCALE to upgrade the firmware and BIOS of the card, and make sure it is in “IT mode” (and not “RAID mode”)

(UPDATE: The following describes how I did this with TrueNAS SCALE’s built-in tools, but per this comment below you can instead do it from the computer’s (pre-boot) UEFI shell, sometimes still referred to as the “BIOS”, or, as described in this comment by installing the card into a Windows machine, or your TrueNAS machine booted from WindowsPE (a form of Windows that can be booted from a USB memory stick that you can make yourself), and then using LSI Storage Authority. These techniques might work with a wider variety of HBA cards than the process I used.)

I SSH’d into my TrueNAS server, as the admin user:

➜  ~ ssh admin@truenas.local
Last login: Sat Apr 27 14:40:37 2024 from 10.0.0.5

	TrueNAS (c) 2009-2024, iXsystems, Inc.
	All rights reserved.
	TrueNAS code is released under the modified BSD license with some
	files copyrighted by (c) iXsystems, Inc.

	For more information, documentation, help or support, go here:
	http://truenas.com

Welcome to TrueNAS

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

admin@truenas[~]$

I wanted to use the built-in sas2flash tool to check the card. (Depending on the card, it might be sas3flash instead, but in my case it is the former. Both tools come with TrueNAS SCALE.

However, the tool is not in the default admin user’s $PATH:

admin@truenas[~]$ which sas2flash
sas2flash not found
admin@truenas[~]$ sudo -i
[sudo] password for admin:

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# which sas2flash
/usr/local/sbin/sas2flash
root@truenas[~]#

So, I continued as root. The first step was to check the card’s details, with sas2flash -list:

root@truenas:~# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

	Adapter Selected is a LSI SAS: SAS2308_2(D1)

	Controller Number              : 0
	Controller                     : SAS2308_2(D1)
	PCI Address                    : 00:04:00:00
	SAS Address                    : 500605b-0-06af-93a0
	NVDATA Version (Default)       : 10.00.00.02
	NVDATA Version (Persistent)    : 10.00.00.02
	Firmware Product ID            : 0x2214 (IT)
	Firmware Version               : 16.00.00.00
	NVDATA Vendor                  : LSI
	NVDATA Product ID              : SAS9208-8i
	BIOS Version                   : 07.29.00.00
	UEFI BSD Version               : N/A
	FCODE Version                  : N/A
	Board Name                     : SAS9208-8i
	Board Assembly                 : H3-25591-00A
	Board Tracer Number            : SV32231490

	Finished Processing Commands Successfully.
	Exiting SAS2Flash.
root@truenas:~#

OK cool! My board was indeed recognized, and moreover, it looks like it was already in “IT mode”. I knew from reading the internet that “IT mode” passes each disk through to the OS, which is generally what TrueNAS SCALE users want, so that ZFS controls the disks directly. (The “IR mode” is RAID mode where the card itself handles RAID.)

That didn’t matter, though, because the way you switch from IR mode to IT mode is by reflashing (overwriting/installing) the firmware. There are two variants of firmware. Since my card had an older firmware installed, and the community strongly recommends upgrading to the latest to get bug fixes, I was going to do that anyway. So doing that would have ensured the card was in “IT mode” anyhow.

However, I was lucky here in that I got what seems to be a real, LSI-branded card. There is a possibility that sas2flash will not recognize your HBA card at this step. In that case, you have to do some more things, and it maybe gets complicated. I don’t know about it, though, since I lucked out and got a supported LSI card.

Next I upgraded the BIOS and firmware. There is a PDF manual in the archive with the firmware that explains the details. In my case, I only have one card but I used the “all” variants of the commands for simplicity.

(In the transcripts below, 9207_8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows is the name of the folder extracted from the downloaded .zip file.)

BIOS:

root@truenas:~# sas2flash -biosall 9207_8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows/sasbios_rel/mptsas2.rom
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

	Ignoring All other commands except for output flags

	Attempting to flash Boot Service to LSI SAS SAS2308_2(D1) :

		Validating BIOS Image...

		BIOS Header Signature is Valid

		BIOS Image has a Valid Checksum.

		BIOS PCI Structure Signature Valid.

		BIOS Image Compatible with the SAS Controller.

		Attempting to Flash BIOS Image...

		Verifying Download...

		Flash BIOS Image Successful.

		Updated BIOS Version in BIOS Page 3.

	Finished Processing Commands Successfully.
	Exiting SAS2Flash.
root@truenas:~#

Firmware:

root@truenas:~# sas2flash -fwall 9207_8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows/Firmware/HBA_9207_8i_IT/9207-8.bin
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

	Ignoring All other commands except for output flags

	Attempting to flash firmware to LSI SAS SAS2308_2(D1) :

	Executing Operation: Flash Firmware Image

		Firmware Image has a Valid Checksum.
		Firmware Version 20.00.07.00
		Firmware Image compatible with Controller.

		Valid NVDATA Image found.
		NVDATA Version 14.01.00.00
		Checking for a compatible NVData image...

		NVDATA Device ID and Chip Revision match verified.
		NVDATA Versions Compatible.
		Valid Initialization Image verified.
		Valid BootLoader Image verified.

		Beginning Firmware Download...
		Firmware Download Successful.

		Verifying Download...

		Firmware Flash Successful.

		Resetting Adapter...
		Adapter Successfully Reset.

	Finished Processing Commands Successfully.
	Exiting SAS2Flash.
root@truenas:~#

Confirm everything updated OK:

root@truenas:~# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

	Adapter Selected is a LSI SAS: SAS2308_2(D1)

	Controller Number              : 0
	Controller                     : SAS2308_2(D1)
	PCI Address                    : 00:04:00:00
	SAS Address                    : 500605b-0-06af-93a0
	NVDATA Version (Default)       : 14.01.00.06
	NVDATA Version (Persistent)    : 14.01.00.06
	Firmware Product ID            : 0x2214 (IT)
	Firmware Version               : 20.00.07.00
	NVDATA Vendor                  : LSI
	NVDATA Product ID              : SAS9207-8i
	BIOS Version                   : 07.39.02.00
	UEFI BSD Version               : N/A
	FCODE Version                  : N/A
	Board Name                     : SAS9208-8i
	Board Assembly                 : H3-25591-00A
	Board Tracer Number            : SV32231490

	Finished Processing Commands Successfully.
	Exiting SAS2Flash.
root@truenas:~#

Reconnect all the drives and make sure they all work

So, as I detailed in a previous post, after upgrading to Dragonfish, four of my drives had disappeared. Although I was happy to upgrade my hardware in general, this missing-disks situation was the concrete problem I was trying to fix.

I connected the two new cables to the HBA card, connected the 8 SATA ends to the drives that had been connected to the old port multiplier card, and booted the server.

All the drives were back, and the missing zpool was automatically found and imported as normal. Everything worked again, the same as before, except now the 8 drives in question are connected to an LSI HBA instead of a “dodgy port multiplier”.

Celebrate with a frosty beverage

I considered a beer, but I wanted to stay productive, so made it an iced green tea instead.

Stux · April 27, 2024, 9:01am

Congrats.

I think this should be a resource.

etorix · April 27, 2024, 8:35pm

Thanks for the extensive write-up—and demonstration that SAS HBAs are not expensive.

Two comments:
Flashing firmware is best done from the UEFI shell. Just download the package for UEFI, put the firmware and EFI tool on a FAT32-formatted USB thumbdrive and boot to to UEFI shell. No issue with root, no messing with a live ZFS pool, no need to disconnect drives.

As you found out the hard way, many server cards come in low profile (half-height) form. Some cards are sold with both low- and high-profile brackets; used cards may come with only one bracket.
If customs are not too much of a hassle, Chinese sellers on eBay must have the high-profile bracket you need.

Davvo · April 27, 2024, 8:57pm

This is a great educational topic, agreed that should be made a resource.

oxyde · April 27, 2024, 9:01pm

Thanks for share your experience, i’m in a similar situation and your post Will help me for sure

nabsltd · April 27, 2024, 10:29pm

I know that this is a TrueNAS forum, but by far the easiest way I have found to flash LSI HBAs is using LSI Storage Authority on Windows.

If you don’t have a Windows machine handy that can hold the HBA, you can use a WinPE boot disk (like Sergei Strelec’s), then install LSI Storage Authority into the running Windows session.

LSI Storage Authority happily takes a BIOS downloaded from the Broadcom site and flashes it to any HBA with the same controller chip with no need to jump through hoops. I just did this to a IBM branded 9300-8i.

axhxrx · April 28, 2024, 1:17am

Thank you for the follow-up comments! I have linked to them in the relevant sections of my post, so that the info can be found easily when reading it for the first time.

Marlenio · April 30, 2024, 7:13am

Same situation, ASM1166 controller with 10 ports, on a 6 disk pool 2 disks are not seen. While waiting for the bug to be resolved, I followed your advice and purchased a 9207 directly with IT firmware in China. A priori I imagine that the performance will be better than the ASM116, and in any case I want to have the highest possible security for my TrueNAS pools.
A curiosity @axhxrx: with the installation of the LSI controller have you noticed any changes in the performance of the to pool in terms of read/write data transfer?

axhxrx · April 30, 2024, 8:43am

@Marlenio No, I did not notice any performance difference from the SMB share tthrough which I access that pool, but it was called slow for a reason — ten year old small disks, which I only access via a 1Gbps LAN connection anyhow. So I got like 115MB/sec or so both before and after, and I never did any fio tests or anything like that.

However, one thing I totally did notice, and I really wish I had tested more (but I didn’t, and now I have replaced all the disks, too): the web UI got dramatically more responsive, and this is irrespective of the 24.04 upgrade. Even on the previous 23.10 version (and prior) I always thought “Ugh, the web UI is so slow! Is this normal?”

Specifically, any time I navigated to the Storage Dashboard page, it took like 5 seconds, and sometimes even 10 or more seconds, to load.

Since I changed out the ASM1166 port multiplier card I had for the LSI HBA, and replaced those old disks, that page just loads in 1 second or so. This is true in both the old and new OS.

I unfortunately didn’t definitively test that before getting rid of the old disks — so it could have been the disks themselves, perhaps, and not the SATA card.

Jacob · April 30, 2024, 9:38am

@axhxrx
Does your new card get hot?
(I know my LSI SAS 9300-16i has a reputation for producing a lot of heat.)

Great write-up. I agree that it should be a resource.

Marlenio · April 30, 2024, 9:44am

@Jacob, I’ll answer your question too. As far as I know, the controller heats up quite a lot, it’s not for nothing that the chipset has a passive heatsink installed as standard. Personally I will install a small fan so as to keep the temperatures under control. Theoretically I think it is enough to replace the fixing screws with a longer type. As soon as it arrives from China I will see how to proceed.

etorix · April 30, 2024, 10:53am

A HBA is a little PowerPC computer running its own OS. It expects some airflow on the heatsink.

@axhxrx The unresponsive UI issue you saw may well have been the middleware querying all drives to gather the information it wants to display. With a HBA all drives are queried simultaneously, as expected, but a port multipliers makes it that drives are queried in sequence, one after the other.

ericloewe · April 30, 2024, 10:59am

Fun fact: For the SAS 9400 and newer, Broadcom moved to ARM instead.

axhxrx · April 30, 2024, 11:19am

@Jacob Yes, my new (used) LSI 9208-8i card gets hot, much hotter to the touch than the outgoing SATA port multiplier card, even when the drives it’s connected to aren’t doing anything.

That’s based on the not-very-scientific test of putting my finger on the heat sync. (I also have no idea how hot that old card would have been under load.)

Jmarc · May 3, 2024, 5:29am

Great write-up, well written and entertaining. Thanks!
Now I do want to see a pic of the back of the pc with the corn-cob pipe holding the card