Expanding the Truenas Scale with a new shelf

TrueNAS General
1

Hello,

I have an 124 TB 16 bay Jupiter Callisto system, running a Truenas Scale.
It does have a 2x SAS port card for future expanshion.

I do have a 16bay jbod array, 124 TB as well. This one has 4 sas cables 8644 on its rear ports. I used it to connect to a PC with an adaptec 8885 raid card.

Now, what would be the best card to use in the truenas system to change from its 2 expansion port to a 4 sas port card to use an expanshion self?

Thanks!

2

Just to add to the original post.

What worries me is that even the iXsystems expansion shelves show connecting only one, or max 2 ports, but on different controllers.

I have not seen an expansion shelf with all 4 sas ports from it to the main system.

But then how do 16 bays or more get some performance on only 1 sas cable?

Thanks

3

Something like:
https://www.ebay.co.uk/itm/226128216954

You only need a single cable - but can use more than one for resiliency or extra bandwidth

Each port has 4 12Gb channels. Each HDD uses say 2Gb at most. The disk shelf will likley contain a SAS expander which will address all disks in the shelf. You have plenty of spare capacity with one SAS external cable to 16 HDDs.

If you use SSD that changes the calculation a bit - but you can go to 2 external cables to double your available bandwidth.

4

Thank you so much!

Now, i’ll open the jbod box to see what is actually inside. My thoughts are that there are sas expanders, but the hdds inside are linked up to 2 hba’s so hence the 4 ports of sas coming out of the jbod. It used to go to a 4x port areca to make the raid.

I would need to change the hbas of the jbod and connect the hdds to the lsi hba?
or change the actual sas card from the truenas box, to one that can accommodate 4 sas connections? Would that work?

Thanks!

I

5

The model number of the shelf (preferably even with a link to the manufacturer’s page) and/or detailed pictures of the hardware and its innards may help us undertsand what it is about.

6

Yup - pictures of the internals please

7

Hey!

So i opened up the jbod. Its just a bunch of disks and their sas connections go to a convertor from internal sas to external sas. No mitherboard, no nithing. That’s it:))

Its not an off the shelf shelf, it was a direct attached storage for a color grading workstation, came from them as a complete system.

So i would need to replace the sas card that’s currently in the truenas system that has only 2 connections with one that has 4 external sas connection it seems…

Or is it impossible?

Thanks!

IMG_8151
IMG_81513024Ă—4032 3.57 MB

IMG_8150
IMG_81503024Ă—4032 2.44 MB

IMG_8149
IMG_81493024Ă—4032 2.12 MB

IMG_8156
IMG_81563024Ă—4032 2.64 MB

IMG_8152
IMG_81524032Ă—3024 2.4 MB

8

I suspect that if you connect a single cable then you will gain access to all the drives.

Try it and see

9

No heatsink anywhere on the drive backplane?
That, and the four cables coming out suggest there’s no expander, so you’d need a 9305-16e HBA. And probably SAS drives because the total length of cabling is above SATA limits.

But if it used to be attached to a -8i8e RAID controller, with one or two cables only, then there must be a SAS expander somewhere.

10

Thank you so much!

I thought as much, a 16e card to put in the truenas was to way to go, and remove the current 2x external sas card currently in the truenas system.

Usually there is only one cable for the shelves, as stated in even the iXsystems expansion shelves manual. But those i guess have a mainboard inside them.

If in the future i would need to expand to another shelf for the truenas, what would be the way to go?

In the jbod, put a motherboard, then put an HBA of 16i, but how do i convert them to an external 1 sas cable? That;s what i dont get about the shelf expansion. How is that achieved, having 16bays and going out on only 1 sas cable?

Thank you so much!

11

Other option would be to use a sas expander inside the jbod… this can be powered by a molex for example, it does not need to be in a pci slot.

Intel RES3FV288

IMG_2153

1 Like
12

Thank you so much for the info!

In theory, i plug in the 16 drives to this intel beauty, and only one sas cable to the hba in the truenas and it will see all 16 drives?

It says that if supports 8 channels per external port, so that would mean to plug in both ports to have the 16drives presented to Truenas.

Or is there some magic happening in the connection that it will see all 16 drives via 1 cable/8 channels?

Sorry for all the questions, just cant wrap my head around the numbers…

Thanks!

13

One cable should work.

Each cable carries up to 48gbps.

14

I have one and it works exactly how you described. All I have in mine is just a PSU, this card and drives. Works like a charm.

1 Like
15

Try it and see.
If you can’t see all the drives then we will try and work out why - but I think it will just work

16

Hi!

Just a quick question, instead of the intel card, would this adaptec work? Seems mostly the same.

Thanks!

17

Yup, an expander is an expander.

18

Hey,

Also, a question for the next step.
I currently have 4x RaidZ1 x4 disks vdevs for my single pool, along with a Log NVME.
This is the most performant configuration i managed to get, about 1.5GB/s to my 2 clients, with smb multichannel turned on. (i still need to play around with 100gbe/25gbe tunables in the Scale, i haven’t got one to work ok, except the smb multichannel of the Scale upgrade changed everything from 700mb/s to 1.5GB/sec to clients…)

After i get these 16disks added, what would be the proper procedure to make it the most performant system i can get, inside the same pool?

Im not worried about 2 drives failing, as i have 2 spares for both of these enclosures and can exchange them as soon as i get an alert.

Also, i can destroy the current pool, i can backup the data and put it on again after the setup is complete.

Should i go 8x RaidZ1? I would need to keep one single pool. Mostly video editing.

What do you think?

All the best!