Dual LSI 9300-8i card issues on TrueNAS Scale

TrueNAS General
,
1

I am running a TrueNAS SCALE server on a custom-built system:

Motherboard: Gigabyte GA-X99-UD4 v3
CPU: Intel i7-5820K
RAM: 64 GB DDR4
HBAs: 2× LSI 9300-8i (SAS3008, IT mode)
NIC: Dell 0W1GCR 10GBE rj45

Originally, I was using a single Adaptec ASR-81605ZQ 12 Gb/s 16-port card to connect all my SAS HDDs and SATA SSDs. About a week ago, I wanted to add a few more drives, so I attempted to add an LSI 9211-8i alongside the Adaptec. After doing this, TrueNAS stopped recognizing drives, and the Adaptec eventually stopped functioning.

I then replaced the Adaptec entirely and tried running two Inspur LSI 9300-8i cards together instead. This appeared to work initially, but now I can only get one HBA to function at a time. On each reboot, it is random which LSI card binds to the mpt3sas driver. The other card is detected by lspci but shows no “Kernel driver in use” and doesn’t expose any disks.

The problem only occurs when two LSI 9300-8i HBAs are installed at the same time.

The drives are a mix of SAS HDDs and SATA SSDs, all with adequate cooling. Both HBAs have dedicated 60 mm fans mounted directly on the heatsinks, keeping them warm but not hot (estimated ~40–50 °C).

I am relatively new to TrueNAS and server hardware, so any insight would be greatly appreciated. I’m happy to provide logs or additional system details if needed.

2

Not enough PCI lanes could be a problem, but I guess they still should work, only be slow.

This is what Rev 1.0 of the board supports.

2 x PCI Express x16 slots, running at x16 (PCIE_1, PCIE_2)* For optimum performance, if only one PCI Express graphics card is to be installed, be sure to install it in the PCIE_1 slot; if you are installing two PCI Express graphics cards, it is recommended that you install them in the PCIE_1 and PCIE_2 slots.

2 x PCI Express x16 slots, running at x8 (PCIE_3, PCIE_4)* The PCIE_4 slot shares bandwidth with the PCIE_1 slot. When the PCIE_4 slot is populated, the PCIE_1 slot will operate at up to x8 mode.* When an 28 lane CPU is installed, the PCIE_2 slot operates at up to x8 mode and the PCIE_3 operates at up to x4 mode.(All PCI Express x16 slots conform to PCI Express 3.0 standard.)

Š

3 x PCI Express x1 slots(The PCI Express x1 slots conform to PCI Express 2.0 standard.)

The other usual suspects: not enough power, or overheating, but you said they are cooled.