Pool contains offline vdevs

TrueNAS General
1

My system is Dragonfish-24.04.2 connecting to an external enclosure (JBOD) as part of datastore for Proxmox over iscsi.
The configuration at this stage is:

  • 4 x 1TB SSD (on the TrueNAS server) (mirror)
  • 8 x 4TB disk (JBOD) - as a stripped mirror (if my config. is right?)

pools
pools830×1015 63 KB

Issue: on boot the disks on the jbod show as “pool contains offline vdevs”. The disks are available for adding back into the pool.
Problem solving:
Following threads elsewhere I exported the pool (without destroying the data) and then successfully (?) reimported it.

Am I doing something wrong here? I’m into a new build - having migrated through FreeNAS, to TrueNAS Core and now to Scale. I’ve seen others having vdevs dropping out and I’m wondering if there is testing I should be doing? TrueNAS has always been stable in disk handling so this makes me a little nervous moving forwards.

The disks are new (as far as I know!) but the hardware is second-hand.

… one more thing! Going to shell - through the GUI - almost always requires a browser reload - a rare occurrence (for me) on previous variants of TN. Is there a preferred browser or a setting to fix?

Any input really appreciated. Thanks.

2

Hello and welcome to the forum.

Can you post your hardware specs especially the HBA you are using along with its firmware and how things are cables to the JBOD.

3

Thanks.
Truenas - 24.0.4.02 on Supermicro TR-2023-TR (2x AMD Epyc 7532) 512Gb ram. System on nvme (700Gb Crucial)
HBA - AOC-S3008L-L8E internally connected to a 16 port mini-sas connector
(I see from a previous thread you encountered this HBA . . . )
External connectors to JBOD (Supermicro CSE-847 847JBOD-14 45-Bay) - SAS/SATA2 (not 3)

1 pool (nvme x 1) is directly housed on the Truenas box
1 pool (Seagate ironwolf pro) 8 x 4TB in JBOD
Various other (largely WDs to be migrated in)

4

Thanks. Any idea of the firmware on that card? sas3flash -listall should tell you.

5

Thanks - firmware version is 16.00.12.00

6

Perfect. How many SAS cables do you have connected from the HBA to the JBOD?

7

Are you able to re-create the issue and share the output of zpool status before you import the pool?

8

I’ve two SAS (mini - internal) going to the external connector. Then two external SAS run to the JBOD and to one of the two backplanes.
The Supermicro backplane has two controllers. I’m struggling a bit to understand whether one of these connections from the backplane is redundant (well used for SAS drives - mine are SATA).
I’ve powered the machine up and down a few times and the main issue (export of VDEVs) has not recurred.

Have you had issues with this HBA? I am surprised that the drives take a little while to populate - slow rather than fast.

9

Thanks for the info. I have used the 9300-8e which is the same chip for many years now on many systems and its rock solid so that doesn’t concern me in the least. You also have the correct firmware which is a pleasant surprise. Your two cables to the JBOD are fairly pointless with those SATA drives and even with SAS would create a multipath which iX moved away from a few years ago and can create problems of its own. I would just run one cable to one controller for now to remove all other variables. I can’t see anything inherently wrong with your setup so perhaps try the above and let us know how things go.

1 Like
10

Don’t think that’s actually what they do.

It should just provide twice as many lanes upstream, allowing 8 drives to transmit simultaneously, and if the expander supports “databolt” more bandwidth too

11

Only if it’s wide-ported on the same controller. Different controllers on the JBOD would be different paths and for SATA it can only speak single path.

PS: unless some sort of zoning has been configured on the JBOD. By default each controller would see all available drives.

1 Like
12

Thanks very much.
So far things have settled after a couple more times of restarting.
My reading of the manuals are that the two controllers are support for SAS and multipath. I think Supermicro dropped in this board variant as it was moving to SAS3 (my conjecture)
Zoning . . . well that voodoo is beyond me unless it is already configured somehow . . .

13

Assuming you have a two Dual path backplanes…

Each one will have 3 or 4 connection on the backplane Main I/O, Main Passthrough, and Second Channel I/O, Second Channel Passthrough…

now on the back of the case you have 4 ports HBA 1, HBA 1 PASSTHROUGH and HBA 2, HBA 2 PASSTHROUGH…

So the JBOD internals could be wired into a few different configurations for a single JBOD with truenas (and all SATA drives) you would likely want HBA 1 to Main I/O backplane 1 then HBA 2 to Main I/O backplane 2 with each passthrough for main going back to the case… essentially not using the second channel on each backplane (this would not be the case for a SAS drive setup)

14

The manual (Supermicro) I have covered 1 controller whilst saying “there is a version of the board with two controllers” [period]
So I assumed the connectors duplicated (as you described).
Let me test and see if can get the Main I/O working to connect.
Thanks!