SOLVED: All of my hard drives are in use but TrueNAS is showing 25 unassigned disks

Hi everyone. I have the similar issue as the topic starter in this thread, 25 disks unassigned (i.e. all disks from the EMC disk enclosure), lsblk reports those disks twice. My HBA is LSI SAS2008 with the most recent firmware (P20). Probably this is the wrong wiring in my case, but the issue in its current state appeared just last week after the upgrade to the recent Scale beta. Before I had a similar issue, but it was only 1 disk reported as unassigned, usually spare.

root@NAS[~]# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

	Adapter Selected is a LSI SAS: SAS2008(B2)

	Controller Number              : 0
	Controller                     : SAS2008(B2)
	PCI Address                    : 00:1f:00:00
	SAS Address                    : 500605b-0-055d-6ec0
	NVDATA Version (Default)       : 14.01.00.07
	NVDATA Version (Persistent)    : 14.01.00.07
	Firmware Product ID            : 0x2213 (IT)
	Firmware Version               : 20.00.07.00
	NVDATA Vendor                  : LSI
	NVDATA Product ID              : SAS9200-8e
	BIOS Version                   : 07.39.02.00
	UEFI BSD Version               : N/A
	FCODE Version                  : N/A
	Board Name                     : SAS9200-8e
	Board Assembly                 : N/A
	Board Tracer Number            : N/A

	Finished Processing Commands Successfully.
	Exiting SAS2Flash.
root@NAS[~]# zpool status -v data
  pool: data
 state: ONLINE
  scan: scrub repaired 896K in 01:45:30 with 0 errors on Sun Sep  8 02:09:32 2024
remove: Removal of vdev 1 copied 13.3M in 0h0m, completed on Wed May 31 23:14:16 2023
	7.15K memory used for removed device mappings
config:

	NAME                                      STATE     READ WRITE CKSUM
	data                                      ONLINE       0     0     0
	  mirror-0                                ONLINE       0     0     0
	    7c5e387d-5dff-416e-8fd4-fdfd6c81f4d9  ONLINE       0     0     0
	    95751702-50f6-45a9-baa3-42e5f515e7cc  ONLINE       0     0     0
	  mirror-2                                ONLINE       0     0     0
	    e31ae18d-c73c-4ddf-ac40-8bd4ac377043  ONLINE       0     0     0
	    f4834b69-e868-473f-a760-7b9a5a74c8c9  ONLINE       0     0     0
	  mirror-3                                ONLINE       0     0     0
	    fdc09682-4d40-41e5-b633-c752140e005b  ONLINE       0     0     0
	    f37a9e46-b221-455a-ae70-6c29a9526534  ONLINE       0     0     0
	  mirror-4                                ONLINE       0     0     0
	    7a45abb7-1e35-42a4-8a51-a3212bb2c8d9  ONLINE       0     0     0
	    f1ef9508-1dc9-437f-9751-84b8176be491  ONLINE       0     0     0
	  mirror-5                                ONLINE       0     0     0
	    443f8d84-822f-4b8f-a070-5908a6a227df  ONLINE       0     0     0
	    82208f8b-5d55-4713-b54c-5d66016ead26  ONLINE       0     0     0
	  mirror-6                                ONLINE       0     0     0
	    202a2512-a158-4f4b-8f5b-4ad4da8b18ff  ONLINE       0     0     0
	    cd5e1014-d925-4e5f-822f-ed18b74dfb04  ONLINE       0     0     0
	  mirror-7                                ONLINE       0     0     0
	    18a5b953-9306-49af-b31b-8543009815cb  ONLINE       0     0     0
	    7e215527-08ed-4def-888d-0d7a130689ae  ONLINE       0     0     0
	  mirror-8                                ONLINE       0     0     0
	    b2f54ca0-bfd1-43f8-bd98-91f42e4e3699  ONLINE       0     0     0
	    e59d3594-144e-45a9-8ecd-c12bafbd2011  ONLINE       0     0     0
	  mirror-9                                ONLINE       0     0     0
	    300d902c-2242-434b-b1f9-0d21a6fabc87  ONLINE       0     0     0
	    02af031b-3c52-4b89-97d6-fddee55288ce  ONLINE       0     0     0
	  mirror-10                               ONLINE       0     0     0
	    ca9a058e-cc74-4e44-9743-7bbf619c4c1e  ONLINE       0     0     0
	    dda2d8cb-3930-47fc-82c6-5953a2d89302  ONLINE       0     0     0
	  mirror-11                               ONLINE       0     0     0
	    73d0b951-4e4b-444a-8d22-4d1253d25f8a  ONLINE       0     0     0
	    047a185f-2dfe-4972-8464-621bf7678a8d  ONLINE       0     0     0
	  mirror-12                               ONLINE       0     0     0
	    0e551aff-b9a6-49ee-859f-8b3df4e02f05  ONLINE       0     0     0
	    ac1eb9b7-e81a-4c3c-be7c-ab4981b9d18d  ONLINE       0     0     0
	  mirror-13                               ONLINE       0     0     0
	    1363672e-56b2-47df-a46b-716cde8e7981  ONLINE       0     0     0
	    f0464b4d-1780-48ad-a9d0-49778c6dd14c  ONLINE       0     0     0
	  mirror-14                               ONLINE       0     0     0
	    11f22382-1c5d-4995-b18c-6db28d80c417  ONLINE       0     0     0
	    22c91bd4-ebe0-46cb-b12b-24145853af9f  ONLINE       0     0     0
	special	
	  mirror-15                               ONLINE       0     0     0
	    ef5ee94c-4359-40da-bc35-51c23e35465e  ONLINE       0     0     0
	    5f917f18-d1a6-4043-a751-9b0a76fcb8b4  ONLINE       0     0     0
	  mirror-16                               ONLINE       0     0     0
	    dae1cc96-548e-4786-bfa4-d6b025380c3b  ONLINE       0     0     0
	    34acf181-a8ce-4091-a15e-f6a4d7d733d1  ONLINE       0     0     0
	cache
	  sdbg1                                   ONLINE       0     0     0
	spares
	  2af7cb89-8b5c-4b92-92a3-c571e397501d    AVAIL

errors: No known data errors
root@NAS[~]# lsblk | grep disk | grep sd
sda       8:0    0 838.4G  0 disk
sdb       8:16   0 838.4G  0 disk
sdc       8:32   0 838.4G  0 disk
sdd       8:48   0 838.4G  0 disk
sde       8:64   0 838.4G  0 disk
sdf       8:80   0 838.4G  0 disk
sdg       8:96   0 838.4G  0 disk
sdh       8:112  0 838.4G  0 disk
sdi       8:128  0 838.4G  0 disk
sdj       8:144  0 838.4G  0 disk
sdk       8:160  0 838.4G  0 disk
sdl       8:176  0 838.4G  0 disk
sdm       8:192  0 838.4G  0 disk
sdn       8:208  0 838.4G  0 disk
sdo       8:224  0 838.4G  0 disk
sdp       8:240  0 838.4G  0 disk
sdq      65:0    0 838.4G  0 disk
sdr      65:16   0 838.4G  0 disk
sds      65:32   0 838.4G  0 disk
sdt      65:48   0 838.4G  0 disk
sdu      65:64   0 838.4G  0 disk
sdv      65:80   0 838.4G  0 disk
sdw      65:96   0 838.4G  0 disk
sdx      65:112  0 838.4G  0 disk
sdy      65:128  0 838.4G  0 disk
sdz      65:144  0 838.4G  0 disk
sdaa     65:160  0 838.4G  0 disk
sdab     65:176  0 838.4G  0 disk
sdac     65:192  0 838.4G  0 disk
sdad     65:208  0 838.4G  0 disk
sdae     65:224  0 838.4G  0 disk
sdaf     65:240  0 838.4G  0 disk
sdag     66:0    0 838.4G  0 disk
sdah     66:16   0 838.4G  0 disk
sdai     66:32   0 838.4G  0 disk
sdaj     66:48   0 838.4G  0 disk
sdak     66:64   0 838.4G  0 disk
sdal     66:80   0 838.4G  0 disk
sdam     66:96   0 838.4G  0 disk
sdan     66:112  0 838.4G  0 disk
sdao     66:128  0 838.4G  0 disk
sdap     66:144  0 838.4G  0 disk
sdaq     66:160  0 838.4G  0 disk
sdar     66:176  0 838.4G  0 disk
sdas     66:192  0 838.4G  0 disk
sdat     66:208  0 838.4G  0 disk
sdau     66:224  0 838.4G  0 disk
sdav     66:240  0 838.4G  0 disk
sdaw     67:0    0 838.4G  0 disk
sdax     67:16   0 838.4G  0 disk
sday     67:32   0 838.4G  0 disk
sdaz     67:48   0 838.4G  0 disk
sdba     67:64   0 838.4G  0 disk
sdbb     67:80   0 838.4G  0 disk
sdbc     67:96   0 838.4G  0 disk
sdbd     67:112  0 838.4G  0 disk
sdbe     67:128  0 838.4G  0 disk
sdbf     67:144  0 838.4G  0 disk
sdbg     67:160  0 238.5G  0 disk
sdbh     67:176  0 223.6G  0 disk

As for the wiring, my EMC VNX 5300 25-Slot disk enclosure has 2 controllers, in both primary ports are connected to HBA ports (so there are 2 cables connecting the enclosure and the HBA).

Sounds a bit like multipathing going on. Im not familiar with this in SCALE but am in CORE. Is there even a multipath tab these days in SCALE under storage or disks?

iX moved away from multipathing a few years ago so I would personally be tempted to wire the JBOD to just one of the controllers. Not sure if there is any on disk metadata that is stamped on the drives when multipathed but this can happen in CORE so there is a chance even after that change you still get funnies.

3 Likes

I agree with @Johnny_Fartpants as that sounds like a plausible cause.

Definitely a multipath situation. Do you recall if the pool was created under CORE or SCALE? We may be able to scrub any legacy gmultipath data.

But I’d definitely recommend changing your wiring to single-path here.

The pool was created in Scale. Cobia if I am not mistaken.
Btw, does not Scale use multipathing at all?! I always considered it a way to improve both reliability and performance…

iX made a statement about this back in 2020.

In a nutshell they took the stance that the small benefits outweighed the potential consequences for miss-configuration. Wide porting allows you to achieve very similar results with essentially zero potential negatives.

I personally used multipath a lot for about 5-6 years and it was great but I completely understand where they were coming from with this and I currently no longer use it as a result.

1 Like

Well, it sounds reasonable. The only issue is that the pool was created in Scale, which as far as I understand never had multipathing. Im I right that wide porting also undertakes using multiple wires to an array? If so than it has to be active at the moment. If not I have no idea or the actual topology anymore… How to check?

Again I can’t speak for SCALE as my personal knowledge is limited but as to wide porting this is essentially two cables from HBA to two ports on the same controller. Often ports A & B but this is dependent on HW.

In CORE just the act of cabling to two controllers on the JBOD would auto apply multipath whether you liked it or not. Not sure if the same still happens in SCALE.

I have just rebooted again to see if something will change. Now the pool is suspended due to tons of read errors on multiple disks, which I did not have before:

root@NAS[~]# zpool status -v data
  pool: data
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-JQ
  scan: scrub repaired 896K in 01:45:30 with 0 errors on Sun Sep  8 02:09:32 2024
remove: Removal of vdev 1 copied 13.3M in 0h0m, completed on Wed May 31 23:14:16 2023
	7.15K memory used for removed device mappings
config:

	NAME                                      STATE     READ WRITE CKSUM
	data                                      ONLINE       0     0     0
	  mirror-0                                ONLINE       0     0     0
	    7c5e387d-5dff-416e-8fd4-fdfd6c81f4d9  ONLINE       0     0     0
	    95751702-50f6-45a9-baa3-42e5f515e7cc  ONLINE       0     0     0
	  mirror-2                                ONLINE       0     0     0
	    e31ae18d-c73c-4ddf-ac40-8bd4ac377043  ONLINE       0     0     0
	    f4834b69-e868-473f-a760-7b9a5a74c8c9  ONLINE       0     0     0
	  mirror-3                                ONLINE       0     0     0
	    fdc09682-4d40-41e5-b633-c752140e005b  ONLINE       0     0     0
	    f37a9e46-b221-455a-ae70-6c29a9526534  ONLINE       0     0     0
	  mirror-4                                ONLINE       0     0     0
	    7a45abb7-1e35-42a4-8a51-a3212bb2c8d9  ONLINE       0     0     0
	    f1ef9508-1dc9-437f-9751-84b8176be491  ONLINE       0     0     0
	  mirror-5                                ONLINE       1     2     0
	    443f8d84-822f-4b8f-a070-5908a6a227df  ONLINE       3     2     0
	    82208f8b-5d55-4713-b54c-5d66016ead26  ONLINE       3     2     0
	  mirror-6                                ONLINE       0     2     0
	    202a2512-a158-4f4b-8f5b-4ad4da8b18ff  ONLINE       3     2     0
	    cd5e1014-d925-4e5f-822f-ed18b74dfb04  ONLINE       3     2     0
	  mirror-7                                ONLINE       0     4     0
	    18a5b953-9306-49af-b31b-8543009815cb  ONLINE       3     4     0
	    7e215527-08ed-4def-888d-0d7a130689ae  ONLINE       3     4     0
	  mirror-8                                ONLINE       0     4     0
	    b2f54ca0-bfd1-43f8-bd98-91f42e4e3699  ONLINE       3     4     0
	    e59d3594-144e-45a9-8ecd-c12bafbd2011  ONLINE       3     4     0
	  mirror-9                                ONLINE       0     2     0
	    300d902c-2242-434b-b1f9-0d21a6fabc87  ONLINE       3     2     0
	    02af031b-3c52-4b89-97d6-fddee55288ce  ONLINE       3     2     0
	  mirror-10                               ONLINE       0     2     0
	    ca9a058e-cc74-4e44-9743-7bbf619c4c1e  ONLINE       3     2     0
	    dda2d8cb-3930-47fc-82c6-5953a2d89302  ONLINE       3     2     0
	  mirror-11                               ONLINE       0     2     0
	    73d0b951-4e4b-444a-8d22-4d1253d25f8a  ONLINE       3     2     0
	    047a185f-2dfe-4972-8464-621bf7678a8d  ONLINE       3     2     0
	  mirror-12                               ONLINE       0     0     0
	    0e551aff-b9a6-49ee-859f-8b3df4e02f05  ONLINE       3   167     0
	    ac1eb9b7-e81a-4c3c-be7c-ab4981b9d18d  ONLINE       0     0     0
	  mirror-13                               ONLINE       0     0     0
	    1363672e-56b2-47df-a46b-716cde8e7981  ONLINE       0     0     0
	    f0464b4d-1780-48ad-a9d0-49778c6dd14c  ONLINE       3    12     0
	  mirror-14                               ONLINE       0    10     0
	    11f22382-1c5d-4995-b18c-6db28d80c417  ONLINE       3    10     0
	    22c91bd4-ebe0-46cb-b12b-24145853af9f  ONLINE       3    10     0
	special	
	  mirror-15                               ONLINE      31    18     0
	    ef5ee94c-4359-40da-bc35-51c23e35465e  ONLINE       3    18     0
	    5f917f18-d1a6-4043-a751-9b0a76fcb8b4  ONLINE       3    18     0
	  mirror-16                               ONLINE      34    10     0
	    dae1cc96-548e-4786-bfa4-d6b025380c3b  ONLINE       3    10     0
	    34acf181-a8ce-4091-a15e-f6a4d7d733d1  ONLINE       3    10     0
	cache
	  sdbg1                                   ONLINE       0     0     0
	spares
	  2af7cb89-8b5c-4b92-92a3-c571e397501d    AVAIL

errors: List of errors unavailable: pool I/O is currently suspended

Not sure what should I do next, try to recover those errors or just unplug one of the cables forever and then reboot again and see what happens?

So you haven’t made any cable changes yet just a reboot and now errors?

Exactly

Ok so nothing to lose at this point. I’d be tempted to unplug the SAS cable to the secondary controller and see what happens. If you can gracefully power down first then even better.

While it is shutting down can you please clarify it a bit. In the other thread I have seen that for the wide porting both HBA ports should be connected to the same JBOD. Does it mean to the same DAE controller (its primary ant extension ports) or to different controllers (primary ports)?

This sounds odd. You have a lot of write errors! And you have done nothing to the system at all except reboot it?

I would power down and verify all cables are solid. As most of your drives are having issues.

Did you have any data issues before?

1 Like

So to wide port you’d use a single HBA and run two SAS cables from the HBA to one controller on the JBOD. You could try and re-wire this way now or for now I’d be tempted to keep things simple and just run one cable from HBA to JBOD. As @joeschmuck said this could also be a cable issue so let’s try things slowly.

1 Like

I think it’s just the drives in the JBOD that’s having issues and the others are within the server. Correct?

I did upgrade ZFS version yesterday, but the pool was working fine after that. As for the rest I did not touch anything.

I nave replaced one failing disk a few months ago and had periodical issue with a 1 unassigned disk.

So I have shut the thing down, detached one of the cables, checked the other (unsurprisingly found no issue with it), booted up. No issues! Thank you guys!

root@NAS[~]# zpool status -v data
  pool: data
 state: ONLINE
  scan: scrub repaired 896K in 01:45:30 with 0 errors on Sun Sep  8 02:09:32 2024
remove: Removal of vdev 1 copied 13.3M in 0h0m, completed on Wed May 31 23:14:16 2023
	7.15K memory used for removed device mappings
config:

	NAME                                      STATE     READ WRITE CKSUM
	data                                      ONLINE       0     0     0
	  mirror-0                                ONLINE       0     0     0
	    7c5e387d-5dff-416e-8fd4-fdfd6c81f4d9  ONLINE       0     0     0
	    95751702-50f6-45a9-baa3-42e5f515e7cc  ONLINE       0     0     0
	  mirror-2                                ONLINE       0     0     0
	    e31ae18d-c73c-4ddf-ac40-8bd4ac377043  ONLINE       0     0     0
	    f4834b69-e868-473f-a760-7b9a5a74c8c9  ONLINE       0     0     0
	  mirror-3                                ONLINE       0     0     0
	    fdc09682-4d40-41e5-b633-c752140e005b  ONLINE       0     0     0
	    f37a9e46-b221-455a-ae70-6c29a9526534  ONLINE       0     0     0
	  mirror-4                                ONLINE       0     0     0
	    7a45abb7-1e35-42a4-8a51-a3212bb2c8d9  ONLINE       0     0     0
	    f1ef9508-1dc9-437f-9751-84b8176be491  ONLINE       0     0     0
	  mirror-5                                ONLINE       0     0     0
	    443f8d84-822f-4b8f-a070-5908a6a227df  ONLINE       0     0     0
	    82208f8b-5d55-4713-b54c-5d66016ead26  ONLINE       0     0     0
	  mirror-6                                ONLINE       0     0     0
	    202a2512-a158-4f4b-8f5b-4ad4da8b18ff  ONLINE       0     0     0
	    cd5e1014-d925-4e5f-822f-ed18b74dfb04  ONLINE       0     0     0
	  mirror-7                                ONLINE       0     0     0
	    18a5b953-9306-49af-b31b-8543009815cb  ONLINE       0     0     0
	    7e215527-08ed-4def-888d-0d7a130689ae  ONLINE       0     0     0
	  mirror-8                                ONLINE       0     0     0
	    b2f54ca0-bfd1-43f8-bd98-91f42e4e3699  ONLINE       0     0     0
	    e59d3594-144e-45a9-8ecd-c12bafbd2011  ONLINE       0     0     0
	  mirror-9                                ONLINE       0     0     0
	    300d902c-2242-434b-b1f9-0d21a6fabc87  ONLINE       0     0     0
	    02af031b-3c52-4b89-97d6-fddee55288ce  ONLINE       0     0     0
	  mirror-10                               ONLINE       0     0     0
	    ca9a058e-cc74-4e44-9743-7bbf619c4c1e  ONLINE       0     0     0
	    dda2d8cb-3930-47fc-82c6-5953a2d89302  ONLINE       0     0     0
	  mirror-11                               ONLINE       0     0     0
	    73d0b951-4e4b-444a-8d22-4d1253d25f8a  ONLINE       0     0     0
	    047a185f-2dfe-4972-8464-621bf7678a8d  ONLINE       0     0     0
	  mirror-12                               ONLINE       0     0     0
	    0e551aff-b9a6-49ee-859f-8b3df4e02f05  ONLINE       0     0     0
	    ac1eb9b7-e81a-4c3c-be7c-ab4981b9d18d  ONLINE       0     0     0
	  mirror-13                               ONLINE       0     0     0
	    1363672e-56b2-47df-a46b-716cde8e7981  ONLINE       0     0     0
	    f0464b4d-1780-48ad-a9d0-49778c6dd14c  ONLINE       0     0     0
	  mirror-14                               ONLINE       0     0     0
	    11f22382-1c5d-4995-b18c-6db28d80c417  ONLINE       0     0     0
	    22c91bd4-ebe0-46cb-b12b-24145853af9f  ONLINE       0     0     0
	special	
	  mirror-15                               ONLINE       0     0     0
	    ef5ee94c-4359-40da-bc35-51c23e35465e  ONLINE       0     0     0
	    5f917f18-d1a6-4043-a751-9b0a76fcb8b4  ONLINE       0     0     0
	  mirror-16                               ONLINE       0     0     0
	    dae1cc96-548e-4786-bfa4-d6b025380c3b  ONLINE       0     0     0
	    34acf181-a8ce-4091-a15e-f6a4d7d733d1  ONLINE       0     0     0
	cache
	  sdag1                                   ONLINE       0     0     0
	spares
	  2af7cb89-8b5c-4b92-92a3-c571e397501d    AVAIL

errors: No known data errors
root@NAS[~]# lsblk | grep disk | grep sd
sda       8:0    0 838.4G  0 disk
sdb       8:16   0 838.4G  0 disk
sdc       8:32   0 838.4G  0 disk
sdd       8:48   0 838.4G  0 disk
sde       8:64   0 838.4G  0 disk
sdf       8:80   0 838.4G  0 disk
sdg       8:96   0 838.4G  0 disk
sdh       8:112  0 838.4G  0 disk
sdi       8:128  0 838.4G  0 disk
sdj       8:144  0 838.4G  0 disk
sdk       8:160  0 838.4G  0 disk
sdl       8:176  0 838.4G  0 disk
sdm       8:192  0 838.4G  0 disk
sdn       8:208  0 838.4G  0 disk
sdo       8:224  0 838.4G  0 disk
sdp       8:240  0 838.4G  0 disk
sdq      65:0    0 838.4G  0 disk
sdr      65:16   0 838.4G  0 disk
sds      65:32   0 838.4G  0 disk
sdt      65:48   0 838.4G  0 disk
sdu      65:64   0 838.4G  0 disk
sdv      65:80   0 838.4G  0 disk
sdw      65:96   0 838.4G  0 disk
sdx      65:112  0 838.4G  0 disk
sdy      65:128  0 838.4G  0 disk
sdz      65:144  0 838.4G  0 disk
sdaa     65:160  0 838.4G  0 disk
sdab     65:176  0 838.4G  0 disk
sdac     65:192  0 838.4G  0 disk
sdad     65:208  0 838.4G  0 disk
sdae     65:224  0 838.4G  0 disk
sdaf     65:240  0 838.4G  0 disk
sdag     66:0    0 238.5G  0 disk
sdah     66:16   0 223.6G  0 disk
sdai     66:32   0 838.4G  0 disk

Next step is to check if connecting the second wire to the controller extension port works. I can do it w/o switching off, right?

In theory yes but call me old fashioned I always prefer shutting down.

I would first ask yourself do you NEED the second connection? Looks like you’re using SAS2 so each cable can provide 24Gb/s. Once you’ve setup your zpool it may not even be capable of pushing 24Gb/s and if not then there is little benefit in attaching the second cable but I’ll leave that for you to decide.

Glad things are looking better :+1:t2:

Sigh… You are probably right. Especially having that I have 10K rpm disks. But w/o checking I would not know if it can make things a bit better )). I have to come back to it when I’l add another DAE.

I recommend you run a Scrub before doing anything else.

I’m going to ask, what is your use case? Do you need ultra fast access and all of that? Are you running a 40Gb or 100Gb connection?

I ask because many people seem to think that everything must be set for fast when meduim well exceeds the need.

2 Likes