96TB Mirror NAS, CPU/motherboard recommendations for pure storage

Usage

Pure storage, no applications, no VMs, primarily for backups from other TrueNAS servers, probably some random family file usage (SMB), possibly some media file streaming if I run out of space on another NAS (NFS). No deduplication or encryption necessary.

Goal

I’d like to future proof this build a little bit so that I can easily populate the entire 24 bays if desired without worrying about RAM or CPU bottlenecks (assume possible 200TB pool at some point in the future) but I don’t think I need high core counts, I’m looking for a power efficient server that can handle a large pool of drives and focus purely on serving files.

Build

I have 12x 16TB WD Red Pro drives that came into my possession early last year and figured this might be a good home for them.

I have a 24x bay Supermicro 846BE1C-R1K23B that I’ve swapped PSUs to PWS-920P-SQ, it has 2x 2.5 bays in the back near the PSU (no idea how that ties into this model number), has stock fans (kinda loud), and it has 4x SFF-8643 cables coming from the backplate/SAS expander so I’m 99% sure it’s a SAS3 backplate.

Since it has 2x 2.5 bays built in I figured I’d mirror some samsung 870 evo 500GB SSDs for the OS as I have them laying around.

I have a spare Mellanox ConnectX-4 that I’m not using which is overkill (25Gb) but I figured I’d use that for now unless I find a motherboard that has SFP+ built in, server LAN uses 10GbE SFP+ switches.

I was looking at a 9305-16i for the HBA, I’ve seen there are cheaper ones like the 9300-16i but it seems to require connecting an additional power cable and even then some people had issues on Reddit so I’d rather avoid that, plus I think the 9305 uses less power anyway.

Planning to try swapping out the stock fans in the case due to noise but still debating on how to do that, I was thinking of building a new fan wall out of foam to use 120mm fans instead of the 3x 80mm fans included as I’ve seen some do that on YouTube, I’m not sure how well that will work since that removes a lot of static pressure going to the backplate/HDDs.

Beyond that my plan starts to fall apart because I’ve been out of the game for a few years, I’ve read some posts on here saying that newer intel 13th and 14th gen have problems which is unfortunate because I was hoping to use something more modern than used power hungry xeons from ebay, I’ve also seen people mention AMD and EPYC CPUs but the docs say AMD only works well on Linux with scale and I’d rather just use core if it’s still more stable as I don’t need the extra features of scale.

The cheapest, lowest power CPU (within reason) that has enough PCIe lanes will be fine. On an old quad core Xeon e3-1275v6, CPU utilization was <5% unless I had a bunch of containers running. RAM you won’t need much, but more always allows more to be cached, which may not really matter depending on the workload.

Look at prices for 9400 and 9500 HBAs too. They may be similar in price, but I haven’t looked in a while. Ebay is possibly the best source for these.

I have the 12 bay Supermicro cases with the same PSUs and possibly the same 80mm fans. Even with the fan speed reduced to keep the drives at ~35c where I like them, they are loud. Important to note, those are in the cases that I’m using as JBODs and don’t have a system in the box generating heat. In a case shared with the CPU, may need more blowing…

Still, even if you get server grade 120mm fans, I’d expect them to pump the same amount of air with less noise. It’s still probably more noise than you’d want to be in a room with but I’m not sure.

I’m not confident desktop grade fans will be able to keep the drives as cool as I’d like, but if you try I’d like to hear the results.

1 Like