I upgrade to a 9400-16i so I could run a supported HBA. Things work ok, but I have 2 devices for every drive now. It looks like it’s enumerating once through the SAS expander, and once directly to the HBA.
Other than TrueNAS showing I have 12 drives that are unassigned, this is working ok. Is there a way to a) add multipath or b) Have TrueNAS ignore one of the interfaces?
Yeah, that is what I was afraid of. Seems kinda silly though that an ‘enterprise’ OS doesn’t support it. I know it used to be in FreeNAS, but apparently people kept nuking their data.
It appears some, (or all?), of the 16 port or 24 port LSI cards are actually using 2, (or 3?), 8 port SAS HBA chips. Using lspci may show if their are 2 LSI SAS HBA chips on your 9400-16i card.
Thus, if a user has a 1 SAS expander connector from 1 SAS chip and another SAS expander connector to a 2nd SAS chip on the same card, you get multi-pathing. Not Wide Porting. I would guess that careful selection of 2 SAS connectors on the card, and wired to the SAS expander back plane would result in Wide Porting instead.
Their can be another cause IF using SAS dual ported disks, with a dual path back plane. However, the disk shown, ST16000NE000, is SATA, so this is not relevant.
That it is! If I could do multpath, I would consider moving all to SAS instead of SATA. But looking at this more, wide path doesn’t give you high availability. What started this whole thing was my computer crashed for some unknown reason (nothing in logs or import events) and the most likely cause was my old HBA. There has not been a firmware update for the old one in 5 years. 9400s are cheap enough now and still supported. When I powered it on and saw 12 extra drives, I began to investigate why and the answer was multi path.
However, if a 4 lane data cable fails completely, and you have a second 4 lane data cable on the same expander back plane, all the disks will remain functional. With the caveat that any transactions in progress through the failed cable, will cause errors. Hopefully, re-tryable to the other cable.
Wide-porting is designed to provide more bandwidth to your drives when needed.
For example let’s say you have a SAS3 controller with a single cable so 12Gb/s x 4 lanes = 48Gb/s theoretical throughput. Now let’s say you have 30 drives capable of 250MB/s or 2Gb/s that would be a total of 60Gb/s. So the bottleneck theoretically would be your single SAS cable. If you now wide-ported it so you had two SAS cables going from the same HBA to the same I/O module you would essentially double your throughput to 96Gb/s.
Yes SAS drives are needed for this.
It also means if you lost one cable everything would continue just at the single cable speed.
This just works out of the box plug and play no on disk info or config needed like multipath hence why TrueNAS moved over as you keep most of the functionality and it’s much more simplified.
SATA drives on a SAS Expander back plane also benefit from Wide Porting. In fact, just using a 4 lane SAS cable to the SAS Expander is in some ways already Wide Porting. This is because the SAS controller chip will send traffic through any of the 4 lanes to the disks, SAS or SATA.
To be clear, this Wide Porting, 2 lanes, 4 lanes or more, is not perfect load sharing. It’s clumsy and a transaction started on one lane must use that lane until complete. So, this can lead to hot lanes depending on the exact configuration and little to nothing a user / storage configurator can do about it.
This “hot lanes” could be like a read from several disks, that the SAS controller chip has decided to use the same lane for it’s traffic. Thus, as the reads come in, if they are more than the lane can handle at once, the rest is simply delayed.
Note that SATA drives on a SAS Expander back plane use SATA tunneling over SAS protocol. Thus, supporting Wide Porting. This is different from direct connecting a SATA disk to a SAS HBA port, which transforms the SAS HBA port into a SATA port, (for the duration of a SATA disk being attached…).
Complicated I know. And I may not have all the details correct.
Actually, looking at this a bit further, it should ‘just work’. The SAS3416 and CSE-826 Expander definitely support it. In fact, I was probably running widepath before with my SAS3224 HBA.
And after going around in circles for hours, I am pretty sure the issue is the firmware on the HBA is ancient. It’s version 8 and version 24 is out. That is at least a good place to start. Thanks guys for leading me down the right direction.
Because I do not want someone to have to go through the trouble I did, I’ll document my journey.
I bought an LSI/Broadcom 9400-16i off eBay. I installed it in the same PCIe slot as my previous HBA and cabled it identically. Success! Until I looked at the TrueNAS storage tab and saw I had 12 unassigned drives. I posted here. I knew that was an answer, but I hated the idea of having a cable in my machine that didn’t go anywhere. My OCD was front and center. I tried a bunch of different things to get this to work, including moving cables around. Nothing.
The next day, after a set of fresh eyes, I realized the firmware on the 9400 was ancient. Of course, all my thumb drives were USB-A and my USB-C to USB-A adapter was at work.
Come home today, put the new FW on a thumb drive, realize EFI doesn’t support ExFAT, reformat the USB drive, and re-flash the HBA to version 24.0.0.0, boot up TrueNAS.. and still had 12 unassigned drives. WTAF?
I fumble around for 2 hours trying to figure out what the hell is going on. Of course, server machines don’t boot quickly, but at least I have IPMI so I can do most of the work from my lounge chair! Anyway, I went back downstairs to unplug one of the cables and resume my investigation tomorrow, and I noticed my expander was on P1 and P2, NOT P0 and P1.
So, if you have duplicate drives in your system, make sure your firmware is up to date, and make sure your expander is plugged into ports P0 and P1 or P2 and P3. Don’t be like me!
Thank for everyone here who helped guide me in the right direction.
