Help with loss of pool

TrueNAS General
1

Hi guys,

Bit of a noob here… I’ve been using truenas for video editing for four years now with no problems but I had a major problem today, I changed HBA card from LSI 9223 9200 9240-8i to LSI SAS 9300-16I 12GB so I could add hard drives and my pool disappeared. I first noticed that 8 drives out of 16 are not detected any more, even in the BIOS. After trying to re import it using zpool import in shell it tells me `root@truenas[~]# zpool import
pool: Norbert
id: 11699626345925506105
state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.

config:

    Norbert     UNAVAIL  insufficient replicas
      mirror-0  ONLINE
        sdb2    ONLINE
        sdf2    ONLINE
      mirror-1  ONLINE
        sde2    ONLINE
        sdd2    ONLINE
      mirror-2  ONLINE
        sdi2    ONLINE
        sdh2    ONLINE
      mirror-3  ONLINE
        sdc2    ONLINE
        sdg2    ONLINE
      mirror-4  UNAVAIL  insufficient replicas
        sdo2    UNAVAIL
        sdl2    UNAVAIL
      mirror-5  ONLINE
        sde2    ONLINE
        sdg2    ONLINE
      mirror-6  DEGRADED
        sdj2    UNAVAIL
        sdd2    ONLINE
      mirror-7  DEGRADED
        sdp2    UNAVAIL
        sdf2    ONLINE`

Ive tried zpool import -f -F -R /mnt Norbert cannot import 'Norbert': no such pool or dataset Destroy and re-create the pool from a backup source.

I’ve connected the HDDs can’t see to another truenas and windows but the HDDs are not detected…

Here’s my server specs :
MB : Supermicro x10SRi-F
CPU : Intel Xeon E52670V3
Ram : ECC 128GB
Chelsio T520-CR Dual Port 10Gb Ethernet
Pool : 8x vdevs of 2 mirrored 6TB Red Plus HDDs
Truenas Scale ElectricEel-24.10.2

And as stupid as I am, I did not make a backup of the pool. But I am wondering what are the chances of 8 drives failing all at once ? And what I also don’t understand is that I can only see 8 drives + 1 boot drive in the Storage > Disks menu but when I try to import it using zpool import, almost all are detected…

Let me know if I have forgotten to provide any information.

Thank you for your help !

2

Did you try putting the old card back in, the way things were, to see if you could at least get your pool back?

3

Thanks for the quick reply Jorsher.
Yes I did. But the 8 HDDs were still missing

4

Give more information on your system.

How are you connecting the disks to the controllers? SATA? SAS to SATA? Backplane? Did you possibly damage some cables? I only see four disks ‘missing’.

If you’re using SAS to SATA, the 8i would have only supported 8 disks, and you mentioned switching to the 9300-16i to add disks. Was the original pool 8 disks, and you expanded to 16 disks, then it stopped working?

5

I’m using a NORCO RPC-4224 4U. 8 HDDs were connected from the backplate (SAS) to SATA on the MB and 8 HDDs were connected from the backplate (SAS) to the HBA card (SAS). So the original pool was 16 HDDs and I wanted to add an extra 4.

As you said, that’s the weird thing. On the Storage > Disk menu I only see 8 (missing 8) HDDs but when I try to import via Shell it shows 4 disks missing.

Screenshot 2025-03-13 at 22.41.49
Screenshot 2025-03-13 at 22.41.491587×680 44.3 KB

6

Did you move all 16 original disks to the HBA? I would try it, if you haven’t already.

I have a Norco 4224 myself, but it had a hardware failure 5-10 years ago that I haven’t got around to troubleshooting (it’s across the ocean). I’ve read that the backplane in them doesn’t have the best reputation. Hopefully this is nothing related.

7

Yes I did try that too but nothing. I took out the backplane and tested it with some other SSDs and they seemed to work fine, so I’m afraid the HDDs could be the problem. I’ve tried putting one of the HDDs that doesn’t show up in another Truenas server, but it couldn’t see it neither.

I’m gonna head to bed and try again tomorrow. Thank again for your help Jorsher !

8

Move drives around and see if different drives come and go.

Could be a power issue, or a failure. Etc.

I have a Norco 4224 as well.

Your issue is you are missing two drives in a mirror.

Pool should come back if you can get one of those drives to come back.

9

Hi Stux, thanks for your reply.

So can I trust the zpool import result over the info in the webgui Storage > Disks not showing half of the HDDs ? I’ve tried moving around the HDDs but never got that one last HDD to show. Also before leaving the office I touched all of them and 8 HDDs were warm and 8 were cold… Could that mean anything ? I’ll try shuffling them tomorrow. Thanks for the help !

10

This is your issue.

1 Like
11

That probably means that the disks weren’t spinning - which may mean a power issue or that the controller hasn’t told them to spin up.

Can you please run the following so we can confirm your HBA is in IT mode:

  • sudo sas2flash -list
  • sudo sas3flash -list

Having both mirrors offline in one vDev is definitely what is stopping the pool importing in a degraded state, but it is a consequential issue rather than the root cause.

If the missing drives move when you shuffle SATA cables, that would indicate a controller issue. Of the same drives stay missing, then more likely to be a drive or power issue.

That said, I cannot see why the same drives in the same positions with the same power cables would create a power issue.

1 Like
12

Hi guys !

I ended up getting my HDDs replaced as they were still under warranty, 8 of them were “dead”. I did lose some medias but I did some LTO backups a week before so not all is lost.

Thanks again for all the help !

13

Apparently not as low as we all thought.

What is worrying is that the new controller appears at first sight to have been the cause of these failures!!! So before you attach any more drives to this controller, we should probably try to work out if this is the case, and if so why.

14

I did end up changing the HBA with another one that I know works on my other truenas, I then did a long S.M.A.R.T test just to check all the drives and I got this error

Screenshot 2025-03-28 at 15.48.26
Screenshot 2025-03-28 at 15.48.26396×270 17.7 KB

I’m thinking could it be the PSU ? Or maybe the backplane ?

Thanks for the help Protopia !

1 Like
15

Please run the following commands and post the output from each inside a </> box:

  • lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID
  • sudo zpool status -v
  • sudo zpool import
  • lspci
  • sudo sas2flash -list
  • sudo sas3flash -list
  • sudo storcli show all
  • sudo zpool import -R /mnt Norbert
    and if that doesn’t work:
  • sudo zpool import -R /mnt -f -n Norbert
16

Here are the results :

* lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID

root@truenas[~]# lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID 
NAME   MODEL                     ROTA PTTYPE TYPE  START          SIZE PARTTYPENAME             PARTUUID
sda    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sda1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS eb978542-13d7-400b-8514-26eed9152dcb
sdb    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sdb1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 2e83c96a-077e-4c6c-b3b7-79446c45cad5
sdc    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sdc1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS b493bc10-3a53-4c69-9e2e-2eca5e1df6b5
sdd    WDC WD60EFZX-68B3FN0         1 gpt    disk        6001175126016                          
└─sdd1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 66a5aa90-fdba-4d33-824a-3d4b7ade3285
sde    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sde1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS cd2aea39-8b8a-4aed-9c7d-0634406cece5
sdf    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sdf1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 23690572-540e-4f91-a912-83a62ba2e9cb
sdg    Samsung SSD 850 EVO 250GB    0 gpt    disk         250059350016                          
├─sdg1                              0 gpt    part     40     272629760 EFI System               a8b5eadd-a71c-11ed-b4fc-000743065f96
└─sdg2                              0 gpt    part 532520  249779191808 FreeBSD ZFS              a8b92ad7-a71c-11ed-b4fc-000743065f96
sdh    Samsung SSD 840 Series       0 gpt    disk         250059350016                          
├─sdh1                              0 gpt    part     40     272629760 EFI System               a92a62cf-a723-11ed-b241-000743065f96
└─sdh2                              0 gpt    part 532520  249786679296 FreeBSD ZFS              a93b238f-a723-11ed-b241-000743065f96
sdi    WDC WD60EFZX-68B3FN0         1 gpt    disk        6001175126016                          
└─sdi1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS c9096ed8-a1ea-42d5-af93-9509a967cf23
sdj    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sdj1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 6c9f049a-8368-4c81-9d20-5b2561486fad
sdk    WDC WD60EFZX-68B3FN0         1 gpt    disk        6001175126016                          
└─sdk1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 3c810069-2e17-4970-8355-0a479bd56db9
sdl    WDC WD60EFRX-68L0BN1         1 gpt    disk        6001175126016                          
└─sdl1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 2c46080b-669d-4b8d-9976-dd5c9cbf94de
sdm    WDC WD60EFZX-68B3FN0         1 gpt    disk        6001175126016                          
└─sdm1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS c0d3d2be-c9e7-411e-ba9c-b4f709b2563d
sdn    WDC WD60EFZX-68B3FN0         1 gpt    disk        6001175126016                          
└─sdn1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 247a6aac-331e-436c-91d7-b9b6d66948f7
sdo    WDC WD60EFRX-68L0BN1         1 gpt    disk        6001175126016                          
└─sdo1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS a4f875eb-57e3-47af-9540-a5a2791cd977
sdp    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sdp1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 6f237cbe-0b71-4c0e-b371-3ff48144c81d
sdq    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sdq1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS ea995e41-6b89-49e7-9942-ec2429dfde41
sdr    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sdr1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 8ccdaef8-78a7-4434-afb2-ef257b344314
sds    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sds1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 9da0673f-2bf3-4f45-ad86-62e24e434881
sdt    WDC WD60EFRX-68L0BN1         1 gpt    disk        6001175126016                          
└─sdt1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS f86682b7-e37c-41f8-926f-dd8773af680c
sdu    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sdu1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS 468c7ffc-92ba-421c-a59d-81f3be87a01c
sdv    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sdv1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS f5c76277-e2f6-4aef-a7cb-3af032598cef
sdw    WDC WD60EFPX-68C5ZN0         1 gpt    disk        6001175126016                          
└─sdw1                              1 gpt    part   2048 6001173463040 Solaris /usr & Apple ZFS d7c29a57-1b90-4f89-a172-d7cb04d6cca3

* sudo zpool status -v

 pool: Norbert
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: resilvered 3.91M in 00:00:03 with 0 errors on Tue Apr  8 09:59:38 2025
config:

        NAME                                        STATE     READ WRITE CKSUM
        Norbert                                     DEGRADED     0     0     0
          mirror-0                                  ONLINE       0     0     0
            spare-0                                 ONLINE       0     0     0
              cd2aea39-8b8a-4aed-9c7d-0634406cece5  ONLINE       0     0     0
              6c9f049a-8368-4c81-9d20-5b2561486fad  ONLINE       0     0     3
            9da0673f-2bf3-4f45-ad86-62e24e434881    ONLINE       0     0     0
          mirror-1                                  DEGRADED     0     0     0
            8ccdaef8-78a7-4434-afb2-ef257b344314    ONLINE       0     0     0
            2e83c96a-077e-4c6c-b3b7-79446c45cad5    REMOVED      0     0     0
          mirror-2                                  ONLINE       0     0     0
            b493bc10-3a53-4c69-9e2e-2eca5e1df6b5    ONLINE       0     0     1
            ea995e41-6b89-49e7-9942-ec2429dfde41    ONLINE       0     0     0
          mirror-3                                  ONLINE       0     0     0
            6f237cbe-0b71-4c0e-b371-3ff48144c81d    ONLINE       0     0     0
            eb978542-13d7-400b-8514-26eed9152dcb    ONLINE       0     0     0
          mirror-4                                  ONLINE       0     0     0
            c0d3d2be-c9e7-411e-ba9c-b4f709b2563d    ONLINE       0     0     0
            d7c29a57-1b90-4f89-a172-d7cb04d6cca3    ONLINE       0     0     0
          mirror-5                                  ONLINE       0     0     0
            247a6aac-331e-436c-91d7-b9b6d66948f7    ONLINE       0     0     0
            f5c76277-e2f6-4aef-a7cb-3af032598cef    ONLINE       0     0     0
          mirror-6                                  ONLINE       0     0     0
            a4f875eb-57e3-47af-9540-a5a2791cd977    ONLINE       0     0     0
            468c7ffc-92ba-421c-a59d-81f3be87a01c    ONLINE       0     0     0
          mirror-7                                  ONLINE       0     0     0
            c9096ed8-a1ea-42d5-af93-9509a967cf23    ONLINE       0     0     0
            f86682b7-e37c-41f8-926f-dd8773af680c    ONLINE       0     0     0
          mirror-8                                  ONLINE       0     0     0
            23690572-540e-4f91-a912-83a62ba2e9cb    ONLINE       0     0     0
            3c810069-2e17-4970-8355-0a479bd56db9    ONLINE       0     0     0
          mirror-9                                  ONLINE       0     0     0
            66a5aa90-fdba-4d33-824a-3d4b7ade3285    ONLINE       0     0     0
            2c46080b-669d-4b8d-9976-dd5c9cbf94de    ONLINE       0     0     0
        spares
          6c9f049a-8368-4c81-9d20-5b2561486fad      INUSE     currently in use

errors: No known data errors

  pool: boot-pool
 state: ONLINE
status: One or more features are enabled on the pool despite not being
        requested by the 'compatibility' property.
action: Consider setting 'compatibility' to an appropriate value, or
        adding needed features to the relevant file in
        /etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d.
  scan: resilvered 28.3M in 00:00:00 with 0 errors on Mon Mar 17 10:59:24 2025
config:

        NAME                                                     STATE     READ WRITE CKSUM
        boot-pool                                                ONLINE       0     0     0
          mirror-0                                               ONLINE       0     0     0
            ata-Samsung_SSD_850_EVO_250GB_S22SNSAG208778K-part2  ONLINE       0     0     0
            sdh2                                                 ONLINE       0     0     0

errors: No known data errors

* sudo zpool import

no pools available to import

* lspci

00:00.0 Host bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 (rev 01)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 01)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 01)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 01)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 01)
00:03.3 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 01)
00:04.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 (rev 01)
00:04.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 (rev 01)
00:04.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 (rev 01)
00:04.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 (rev 01)
00:04.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 (rev 01)
00:04.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 (rev 01)
00:04.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 (rev 01)
00:04.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 (rev 01)
00:05.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 01)
00:05.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug (rev 01)
00:05.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 01)
00:05.4 PIC: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC (rev 01)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
01:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapter
02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
04:00.0 RAID bus controller: Adaptec Series 8 12G SAS/PCIe 3 (rev 01)
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
07:00.0 RAID bus controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
ff:0b.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 01)
ff:0b.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 01)
ff:0b.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 01)
ff:0c.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0d.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0d.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0d.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0d.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 01)
ff:0f.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 01)
ff:0f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 01)
ff:0f.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 01)
ff:0f.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 01)
ff:0f.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 01)
ff:0f.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 01)
ff:10.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 01)
ff:10.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 01)
ff:10.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 01)
ff:10.6 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 01)
ff:10.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 01)
ff:12.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 01)
ff:12.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 01)
ff:12.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 01)
ff:12.5 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 01)
ff:13.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 01)
ff:13.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 01)
ff:13.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 01)
ff:13.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 01)
ff:13.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 01)
ff:13.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 01)
ff:14.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 01)
ff:14.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 01)
ff:14.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 01)
ff:14.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 01)
ff:14.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 01)
ff:14.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 01)
ff:14.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 01)
ff:14.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 01)
ff:16.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 01)
ff:16.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 01)
ff:16.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 01)
ff:16.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 01)
ff:16.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 01)
ff:16.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 01)
ff:17.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 01)
ff:17.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 Thermal Control (rev 01)
ff:17.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 ERROR Registers (rev 01)
ff:17.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 ERROR Registers (rev 01)
ff:17.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 01)
ff:17.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 01)
ff:17.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 01)
ff:17.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 01)
ff:1e.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 01)
ff:1e.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 01)
ff:1e.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 01)
ff:1e.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 01)
ff:1e.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 01)
ff:1f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 01)
ff:1f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 01)

* sudo sas2flash -list

LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18) 
Copyright (c) 2008-2014 LSI Corporation. All rights reserved 

        Adapter Selected is a LSI SAS: SAS2008(B2)   

        Controller Number              : 0
        Controller                     : SAS2008(B2)   
        PCI Address                    : 00:07:00:00
        SAS Address                    : 0000000-0-0000-0000
        NVDATA Version (Default)       : 14.01.00.08
        NVDATA Version (Persistent)    : 14.01.00.08
        Firmware Product ID            : 0x2213 (IT)
        Firmware Version               : 20.00.07.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9211-8i
        BIOS Version                   : 07.39.02.00
        UEFI BSD Version               : N/A
        FCODE Version                  : N/A
        Board Name                     : SAS9211-8i
        Board Assembly                 : N/A
        Board Tracer Number            : N/A

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

* sudo sas3flash -list

Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02) 
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

        Controller Number              : 0
        Controller                     : SAS3008(C0)
        PCI Address                    : 00:02:00:00
        SAS Address                    : 5003048-0-1bae-8702
        NVDATA Version (Default)       : 0b.02.00.07
        NVDATA Version (Persistent)    : 0b.02.00.07
        Firmware Product ID            : 0x2221 (IT)
        Firmware Version               : 13.00.00.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : LSI SAS3008
        BIOS Version                   : 08.31.00.00
        UEFI BSD Version               : 15.00.00.00
        FCODE Version                  : N/A
        Board Name                     : LSI3008-IR
        Board Assembly                 : N/A
        Board Tracer Number            : N/A

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.

* sudo storcli show all

CLI Version = 007.2807.0000.0000 Dec 22, 2023
Operating system = Linux 6.6.44-production+truenas
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = truenas
Operating System  = Linux 6.6.44-production+truenas
StoreLib IT Version = 07.2900.0200.0100

* sudo zpool import -R /mnt -f -n Norbert

cannot import 'Norbert': a pool with that name already exists
use the form 'zpool import <pool | id> <newpool>' to give it a new name
root@truenas[~]# sudo zpool import -R /mnt -f -n Norbert 
-n or -X only meaningful with -F
usage:
        import [-d dir] [-D]
        import [-o mntopts] [-o property=value] ... 
            [-d dir | -c cachefile] [-D] [-l] [-f] [-m] [-N] [-R root] [-F [-n]] -a
        import [-o mntopts] [-o property=value] ... 
            [-d dir | -c cachefile] [-D] [-l] [-f] [-m] [-N] [-R root] [-F [-n]]
            [--rewind-to-checkpoint] <pool | id> [newpool]

I did recreate a pool named Norbert before doing the S.M.A.R.T test. And this morning when I booted up the server I got this error :slight_smile:

Critical
Pool Norbert state is DEGRADED: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
The following devices are not healthy:
Disk WDC_WD60EFPX-68C5ZN0 WD-WX42D344EANE is UNAVAIL
Disk 2737588994972716541 is REMOVED
2025-04-08 10:00:46 (Europe/Zurich)
Dismiss
17

That firmware is ancient. You should be running 16.00.12.00 or higher.

18

Thanks for that neofusio.

There’s just this part : cd to the directory you copied the firmware file
Being a noob, I’m not really sure how to get the firmware file I downloaded to truenas to launch the update

19

You can either transfer it using a SMB share, or download it using using a shell using (all on one line): wget -O SAS3_FW_Phase16.0-16.00.12.00_Firmware.zip https://www.truenas.com/community/resources/lsi-9300-xx-firmware-update.145/download

If you use a shell I recommend you use ssh to login instead of the built-in shell in the GUI, you do not want the shell to reset while it’s updating the firmware.

Installing the firmware update does assume a certain level of experience. If navigating in a *nix OS based shell is entirely new to you, you have some reading ahead of you.

20

HBA firmware updates are best done from the UEFI shell, not from TrueNAS.

Basic instructions.

1 Like