Standing up 36 x 18TB drive TrueNAS Scale server

I’ve been away from TrueNAS for a while, but am coming back into the fold and and standing up a fairly large system. My hardware is as follows:

SMC H12SSL-i mobo with Epyc 7302P CPU
SAS9305-16i HBA
SMC 846 36 bay chassis with SAS3 backplanes
8 sticks of the Samsung 32GB RAM

Amazon.com: MDD MAXDIGITALDATA MDD 18TB 7200RPM 256MB Cache SAS 12.0Gb/s 3.5inch Internal Enterprise Hard Drive (MDD18TSAS25672E) - 5 Years Warranty (Renewed) : Electronics 18TB SAS drives (these are white labeled Seagate X18 EXOS drives I’m pretty sure. Come with a 5 year warranty.

NIC will be a Mellanox ConnectX-4 Lx (25 Gbps)

Plan is to do 4 z2 vdevs with 9 drives in each, for a total of 28 data drives.

Do you guys see any issues with this hardware or configuration. This is primarily be a media server for my home network. It will replace a hardware raid solution that will become my backup for this new one. The hardware raid one consists of 24 8TB drives in RAID60 and 24 6TB drives in RAID 60 as sell. Has been stable for many years and many drive replacements, but drive failures are becoming more frequent (all WD Reds), and space is running low, so I figured it was time for a new primary storage solution.

EDIT, had to remove most of the links since I’m a new user.

2 Likes

This is quite some kind of “home” server… Have your own nuclear plant in the basement as well?

Mellanox NICs are not among the preferred. Other than that, and possibly the lack of spares or free slots for easy drive replacement, I see no obvious issue.

3 Likes

Two things I immediately thought:

  • with 9 disks in each vdev you’re getting close to the point where you might want to consider raidz3 for additional redundancy
  • I may be stating the obvious, with (checks notes) 648TB of raw storage you won’t be doing dedupe with “only” 256GB of RAM :slight_smile:

Are you hosting this at home, and do you own your own nuclear power station? :smiley:

Also: I just put a load of those Exos disks into a backup server. They run hot and loud, you don’t want them anywhere near living areas, imho. But I imagine this will be in a garage/basement/datacentre

This will consume a lot less power than my current setup with 48 6TB/8TB drives. I’ll only power that up once a month or so to do a backup from the new system.

Yes, this will be in my basement away from living areas.

As for power, I have a 60 kW solar setup and produce about 350 kWh daily during the summer. So power is essentially free. Used to do a lot of crypto mining (still do in the winter when I need heat) which is what the solar setup was scaled for. So I’m grid tied and have a LOT of extra power each month that flows into my “bank” at my poco. If I turned off all my solar tomorrow, I have enough kWh saved up to allow me to run for almost 3 years before I would start to get a bill. :smile:

Here are my panels:

I did order a couple of cold spares to have on hand. I’ll look for another NIC. I just happened to have the Mellanox “in stock” and run those on some windows boxes.

7 Likes

Ah, this is what crypto can be used for!

Mellanox may work, but there have been some mixed reports about the brand in the old forum.

I think the Mellanox X4 etc range should be fine. You’re planning on using Scale right?

Can’t imagine the drivers are flaky in Linux

Mellanox is owned by Nvidia these days

Yep, plan to use Scale. I’ll give the Mellanox a shot.

My plan is to connect 8 channels (2 cables) to the 24 bay backplane and 4 channels (1 cable) to the 12 bay backplane. I don’t know that Scale will care one way or another, or it there’s any performance difference. But I figured that would provide the same ratio of drives per link to both backplanes, which would be 3 drives per channel.

Finally had a chance to start putting this server together. Got the mobo installed and cable management so as to not impede air flow:

I did end up getting an Intel 25Gbps NIC.

Lots of HDs!

Running memtest86 now. It will take a while!

In hindsight, I should have moved it down into the rack in the basement before I started, lol.

But its not THAT loud after its been running for a few minutes and the fans settle down.

Power usage just running memtest86 with no HDs installed yet:

I’m sure that will pick up once there are 36 7,200 rpms spinners in there.

3 Likes

Off topic, but have to ask as I’m looking for ground mounts/panels and or an installer. so if you don’t mind from whom or where did you get the ground mounted panels and the panel mounts? Thanks.

@PhilD13 I did the ground mounts 8 years ago. I used IronRidge:

I purchased from Renvu back then. Did the install myself, including all the electrical work, pulling permits, etc. The Ironridge system is pretty neat in that you source the 3" pipe from your local pluming supply, they provide everything else.

Lately, I have buying solar stuff from Signature Solar. My most recent build (about to start) is a 4 car carport with 48 395W bifacial panels and a EG4 18k hydrid inverter and battery.

If you price it out, the 4 car carport might be a better deal compared to the Ironridge stuff.

And you get a water proof car port to boot.

Thanks for the info and the photos of installs you posted. This is very helpful as I’m just starting to get into research.

Ultimately I would like to go off grid completely if I could afford it. Low profile ground mounts intrigue me as I live in South Louisiana where hurricanes are common and I have space for ground mounts and can get used drill pipe. Ground mounting may make the install more robust. I also have lots of sun up to sun down roof space on 3 buildings that could be covered. The power company servicing me is not friendly at all to alternative energy sources and the local gov. is not so permit friendly to do it yourself installs as part of that. Pretty much a racket here designed to keep people away from alternative sources of power.

1 Like