Added a Disk, Can't See It

Added a new disk to the case, plugged it in…it doesn’t show up in the OS.

I have a couple of LSI 9211-8i 6G SAS HBA FW:P20 IT Mode cards, running on an ASUS PRIME B660M-A CSM DDR4 LGA1700 mATX motherboard. Six drives already present and fully functional.

Please run the following commands and paste the results here with the results of each in a separate </> box:

  • lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID
  • lspci
  • sudo sas2flash -list
  • sudo sas3flash -list
  • sudo zpool status -v
  • sudo zpool import
root@truenas[~]# lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID
NAME          MODEL                 ROTA PTTYPE TYPE     START           SIZE PARTTYPENAME             PARTUUID
sda           WDC WD201KFGX-68BKJN0    1 gpt    disk           20000588955648                          
├─sda1                                 1 gpt    part       128     2147418624 Linux swap               99432ef1-9634-4b82-88de-df6438a5005d
└─sda2                                 1 gpt    part   4194432 19998441389568 Solaris /usr & Apple ZFS 43e9436d-3f8a-430a-b4ba-fb6028ce96df
sdb           WDC WD201KFGX-68BKJN0    1 gpt    disk           20000588955648                          
├─sdb1                                 1 gpt    part       128     2147418624 Linux swap               cee0c4f7-6f23-4faf-8c0b-359b8b415802
└─sdb2                                 1 gpt    part   4194432 19998441389568 Solaris /usr & Apple ZFS 91f76a5f-3418-4ad9-aaab-34c89f4d04f7
sdc           WDC WD201KFGX-68BKJN0    1 gpt    disk           20000588955648                          
├─sdc1                                 1 gpt    part       128     2147418624 Linux swap               886987a1-c6a4-4705-8c83-ba9e7cc3a45d
└─sdc2                                 1 gpt    part   4194432 19998441389568 Solaris /usr & Apple ZFS 0b1a6f66-8b64-49da-aaab-661b22a0e38e
sdd           WDC WD201KFGX-68BKJN0    1 gpt    disk           20000588955648                          
├─sdd1                                 1 gpt    part       128     2147418624 Linux swap               21dc8cc5-64e9-41e2-bc3c-be5c39ab8944
└─sdd2                                 1 gpt    part   4194432 19998441389568 Solaris /usr & Apple ZFS 671bcebe-f955-4246-9de6-8d0eb6f16a59
sde           WDC WD201KFGX-68BKJN0    1 gpt    disk           20000588955648                          
├─sde1                                 1 gpt    part       128     2147418624 Linux swap               929aac2f-4a94-4e54-b201-05861afffa5a
└─sde2                                 1 gpt    part   4194432 19998441389568 Solaris /usr & Apple ZFS 99a0bc63-87b7-44e2-a174-08535631723d
sdf           WDC WD201KFGX-68BKJN0    1 gpt    disk           20000588955648                          
├─sdf1                                 1 gpt    part       128     2147418624 Linux swap               89116cc7-843f-492e-8869-d501f79dc32f
└─sdf2                                 1 gpt    part   4194432 19998441389568 Solaris /usr & Apple ZFS 53b54420-0eb5-4667-a59a-0fed5e2d788e
nvme0n1       SPCC M.2 PCIe SSD        0 gpt    disk             512110190592                          
├─nvme0n1p1                            0 gpt    part      4096        1048576 BIOS boot                e2cf923d-c7ee-4894-878a-4fbff67af3f8
├─nvme0n1p2                            0 gpt    part      6144      536870912 EFI System               07584949-32d5-4dc3-bc1e-7c534e828bb1
├─nvme0n1p3                            0 gpt    part  34609152   494390287872 Solaris /usr & Apple ZFS dd353f72-1fe5-4732-85d2-f9c8b0aff61e
└─nvme0n1p4                            0 gpt    part   1054720    17179869184 Linux swap               12b63977-a0b8-4305-a1ec-06cc6709d204
  └─nvme0n1p4                          0        crypt             17179869184
root@truenas[~]# lspci
00:00.0 Host bridge: Intel Corporation Device 4630 (rev 05)
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-S GT1 [UHD Graphics 730] (rev 0c)
00:06.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 (rev 05)
00:0a.0 Signal processing controller: Intel Corporation Platform Monitoring Technology (rev 01)
00:0e.0 RAID bus controller: Intel Corporation Volume Management Device NVMe RAID Controller
00:14.0 USB controller: Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller (rev 11)
00:14.2 RAM memory: Intel Corporation Alder Lake-S PCH Shared SRAM (rev 11)
00:15.0 Serial bus controller: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #0 (rev 11)
00:16.0 Communication controller: Intel Corporation Alder Lake-S PCH HECI Controller #1 (rev 11)
00:17.0 SATA controller: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] (rev 11)
00:1a.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #25 (rev 11)
00:1c.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #1 (rev 11)
00:1c.1 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #2 (rev 11)
00:1c.4 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #5 (rev 11)
00:1f.0 ISA bridge: Intel Corporation Device 7a86 (rev 11)
00:1f.3 Audio device: Intel Corporation Alder Lake-S HD Audio Controller (rev 11)
00:1f.4 SMBus: Intel Corporation Alder Lake-S PCH SMBus Controller (rev 11)
00:1f.5 Serial bus controller: Intel Corporation Alder Lake-S PCH SPI Controller (rev 11)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (17) I219-V (rev 11)
01:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT SSD Controller (rev 03)
04:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
05:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
root@truenas[~]# sudo sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18) 
Copyright (c) 2008-2014 LSI Corporation. All rights reserved 

        Adapter Selected is a LSI SAS: SAS2008(B2)   

        Controller Number              : 0
        Controller                     : SAS2008(B2)   
        PCI Address                    : 00:04:00:00
        SAS Address                    : 500605b-0-062d-bec0
        NVDATA Version (Default)       : 0f.00.00.24
        NVDATA Version (Persistent)    : 0f.00.00.24
        Firmware Product ID            : 0x2213 (IT)
        Firmware Version               : 15.00.00.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : Undefined
        BIOS Version                   : 07.29.01.00
        UEFI BSD Version               : 07.22.04.00
        FCODE Version                  : N/A
        Board Name                     : IBM 6Gb Perf HBA
        Board Assembly                 : H3-25113-03A
        Board Tracer Number            : SP31719738

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.
root@truenas[~]# sudo sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02) 
Copyright 2008-2017 Avago Technologies. All rights reserved.

        No Avago SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.
root@truenas[~]# sudo zpool status -v
  pool: MainStorage
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 1 days 02:19:10 with 0 errors on Mon Dec  9 02:19:12 2024
config:

        NAME                                      STATE     READ WRITE CKSUM
        MainStorage                               ONLINE       0     0     0
          raidz2-0                                ONLINE       0     0     0
            99a0bc63-87b7-44e2-a174-08535631723d  ONLINE       0     0     0
            53b54420-0eb5-4667-a59a-0fed5e2d788e  ONLINE       0     0     0
            671bcebe-f955-4246-9de6-8d0eb6f16a59  ONLINE       0     0     0
            91f76a5f-3418-4ad9-aaab-34c89f4d04f7  ONLINE       0     0     0

errors: No known data errors

  pool: Mirror
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 1 days 00:57:15 with 0 errors on Mon Dec  2 00:57:16 2024
config:

        NAME                                      STATE     READ WRITE CKSUM
        Mirror                                    ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            0b1a6f66-8b64-49da-aaab-661b22a0e38e  ONLINE       0     0     0
            43e9436d-3f8a-430a-b4ba-fb6028ce96df  ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:13:13 with 0 errors on Sun Dec 22 03:58:14 2024
config:

        NAME         STATE     READ WRITE CKSUM
        boot-pool    ONLINE       0     0     0
          nvme0n1p3  ONLINE       0     0     0

errors: No known data errors
root@truenas[~]# sudo zpool import
no pools available to import

Mmmh, lspci lists two LSI SAS2008 controller, sas2flash -list only one? Does one have to select one first? I never have had more than installed. :smile:

BTW, why do you have two of them installed for 6 disk (now 7 disks). Googling the Board Assembly leads to a -8i card, and you will probably have onbord-SATA ports as well?

Not sure that is the newest firmware, I have the same controller with 20.00.00.00.

Okay, that has 3 PCIE slots, physically all x16, internally one x16, one x4, one x1. Those HBA will need x8 slots. Not sure if they will work in x4 or x1 slots… :neutral_face:

BTW, why do you have two of them installed for 6 disk (now 7 disks). Googling the Board Assembly leads to a -8i card, and you will probably have onbord-SATA ports as well?

I have room in the case for 20 disks, the cards are there for futureproofing.

Okay, that has 3 PCIE slots, physically all x16, internally one x16, one x4, one x1. Those HBA will need x8 slots. Not sure if they will work in x4 or x1 slots…

What does that mean in English?

OK - a x64 CPU does I/O through PCIe lanes and each lane has a fixed speed (depending on the version of PCIe).

PCIe slots expose a certain number of lanes (which can but does not have to be reflected in the physical size of the slot). A fully populated full length PCIe slot has 16 lanes, but could have 16, 8, 4, 2 or 1 PCIe lanes connected.

At the same time, the card that you put in the slot can have physically different number of channels exposed on the bit you insert in the slot, and it can have different requirements for how many lanes it actually needs to use to operate.

So a card with a full length insert may work with 16 or 8 or 4 lanes.

What @prez02 is saying is that you have 2 HBAs which need to be in slots with 8 lanes exposed. But your MB only has one slot with at least 8 lanes, and that may be why the 2nd HBA which at best is going to be in a x4 slot, may not work. (It may work sufficient to be visible in lspci but not working sufficiently for sas2flash to see it or for the O/S to see the disks attached to it.

See What is PCIe? Understanding PCIe Slots, Cards and Lanes - Crystal Group

2 Likes

Neither of you is running the most up-to-date firmware, which is 20.00.07.00.

1 Like

But your MB only has one slot with at least 8 lanes, and that may be why the 2nd HBA which at best is going to be in a x4 slot, may not work.

So…I’m completely screwed? I asked for a ton of input when building the machine, and these are the things I was told to buy.

Neither of you is running the most up-to-date firmware, which is 20.00.07.00.

Firmware for what? How does one update it? Is it likely to achieve anything?

Well I disconnected the disk from the PCIe card, and reconnected it using a connector that definitely branched from the same place as the two that are connecting two of my existing drives. No change.

Your firmware is especially ancient, so updating it will likely at least give you more stability.

However, I suspect it’s not the cause of your current issues and I don’t feel comfortable guiding you through the update process. Maybe see if you can find a video of someone doing it to gauge if you think you can understand and replicate the process. Or maybe get in touch with whomever you bought that card from. The title you gave it was “LSI 9211-8i 6G SAS HBA FW:P20 IT” but it’s definitely not running Phase 20, so it was at least improperly labeled.

Does it spin up? How are you powering the drive?

My mistake, it is indeed .07

grafik

1 Like

Got the cards off eBay.

Does it spin up? How are you powering the drive?

Not really sure how to tell, given it’s buried in the case. It’s hooked to a Cooler Master MWE 1250W Gold Modular Power Supply.

If you swap the card with the one that’s working… and then nothing works… then the card is probably faulty.

Okay, at this stage I suspect a faulty drive or possibly the 3.3v “issue”.
I don’t see you saying what brand and model the new drive actually is, please let us know.

1 Like

WD Red 20TB, same as the existing drives.

3.3v issue?

Explanation of the 3.3v power disable feature:

But I don’t recall seeing WD Reds being affected by that, only their enterprise line and at least some of their white label drives.

Not necessarily. Running a 8x card in 4x slot could or should work, but at reduced bandwith.

Some mainboards have PCI-E slots that are open at the end, so that longer card fit. Sometimes people even cut them open themselves.

You are in the lucky position of having 3 long x16slots, even though that is misleading with regards to their “electrical” capability.

If you want to use a graphic card, you’ll have to put the second HBA in a x1. If that works, I guess the speed will not be great…

Should work. PCIe cards should work with fewer lanes than their actual width.

Ie, if it fits it should work, or at a minimum it’s non-conformant.

1 Like