Hardware Selection for first Home NAS

TrueNAS General
1

Dear All,

I’d like to build my first own NAS and use TrueNAS Scale as the OS. I’ve been reading through the hardware documentation guide TrueNAS Community Hardware Guide 2021-01 Edition Revision 2a) as well as some hardware recommendation threads on this forum. I feel that I got the basic concepts of what’s needed to properly build a NAS and nailed it down to a builds (with some alternatives/options). I wanted to check with you, whether this makes sense, is a sane build, if you have improvements in mind or if I have totally missed something.

Requirements​

First a list of requirements/necessities/prerequisites for my NAS build.

  • I am to 98% settled on the Jonsbo N3 as my case, which only supports mITX motherboards. I know that this poses a big limitation, but I want a rather small, compact and minimalistic case. The only reasonable alternative I found was the Treasure Pro - but it is very hard to get and comes with a huge price tag.
  • It should be as silent as possible without impacting the performances.
  • It shall be fast, reactive and reliable.

Use case

My use case is quite simple and straight forward.

  • I will run TrueNAS bare metal and use it almost exclusively as a Storage/Data Server - no other applications/containers/etc. needed, since I have a separate server for these tasks.
  • Editing of medium sized files on the NAS (but no editing of large media files (4K movies etc.)).
  • Streaming of movies/series via the other sperate server.
  • A maximum of 3-4 simultaneous users/clients (both read/write) will have access to the NAS. Users have either MacOS or Linux systems.
  • All parts should be directly available (in the EU).

Build

Overview

Here a small overview of the build I have in mind.

Component Main choice Alternative 1 Alternative 2
Motherboard Supermicro X12STL-IF ASRock E3C246D2I Supermicro SoC board
CPU Intel Xeon E-2314
HDD/Storage Toshiba Enterprise MG08/09 (edit) Toshiba N300 Seagate EXO X X16
Boot Drive cheap SSD SuperDOM
RAM 1*32GB ECC UDIMM 1*32GB ECC RDIMM
PSU Seasonic Prime Fanless PX ATX Seasonic Focus GX 550W

Motherboard

The motherboard seems to be the central choice, when building a NAS. Due to the case I am limited to mITX motherboards. My main choice would be the Supermicro X12STL-IF. As an alternative I came up with the E3C246D2I from ASRock, which hosts 8 SATA connectors (via OcuLink) vs. the 6 on the Supermicro, but has an older socket type.

With a SoC board (from Supermicro) on the other hand I could go down a totally different road. Here the options seem to be a bit broader: Starting with the A2SDi-2C-HLN4F (2* 1.5GHz) through the A2SDi-4C-HLN4F (4* 2.2GHz) “up” to the A2SDi-8C±HLN4F or the A2SDi-H-TF (each 8*2.2GHz). One of the advantages of a SoC board would be the use of the (cheaper) RDIMMs as well as the lower energy consumption of the embedded CPU. Yet, I am absolutely not used to such boards. Is there anything to consider in particular when building on such a board? (E.g. do they have a shorter lifespan, need an additional CPU fan etc.) And do you have good experiences with these?

CPU

With the Xeon E-2314 I took the cheapest suitable CPU for the X12STL board, since I have the feeling it is still overpowering for my use case. Concerning the SoC board: Which of the above named mainboards is powerful enough for my use case?

HDD/Storage

I would start with 3 Toshiba Enterprise MG09 (edit) 16TB HDDs. I would put them in a 3-way-mirror, since I have read, that with such large drives it is almost imperative to have at least 2 fallback HDDs during resilvering when 1 drive fails. My decision to choose such large hard disks is due to the fact that they are helium sealed and thus quieter than their counterparts. Or is it better to start with a set of smaller drives (e.g. 3*8TB) and extend along the way? If so, are the differences in noise level noticeable? Would you opt for something different than a 3-way-mirror?

Boot drive

Here I am not settled yet - but I guess either SSD or SuperDOM is fine.

RAM

My choice would simply be the cheapest ECC RAM (UDIMM or RDIMM in the case of the SoC board) from Micron or Kingston. I would start with a single 32GB stick and upgrade if necessary. Or would you opt for 2*16GB dual channel?

PSU

The Prime Fanless is Platinum and delivers 450/500 Watts, whereas the Focus SGX is Gold and delivers up to 550 Watts - but comes with a smaller price. Both options seem valid in my opinion.

LAN

The last mentioned SoC board (A2SDi-H-TF) has 2*10GbE LAN ports. Is it advisable (in terms of foresight) for my use case? Or is 1GbE enough?

Questions

Besides the questions corresponding to the individual parts I have a few general ones:

  1. Which build is better for my intended use and has the best long-term prospect?
  2. Is there any bottle neck that I couldn’t spot in the given builds?
  3. Anything else I should be aware of besides the hardware when setting up TrueNAS?

Many thanks already for your help and support!

2

Have you considered the Fractal Design Node 304?

Excellent experience with A2SDi-H-TF, for pure storage. But not cheap.
Mind that C3558 is limited in flexible I/O lanes, and an A2SDi-4C will basically require to choose between the PCIe slot and 4 HDDs on the MINISAS HD. Oops!

In non-server case, take a board with a “passive” heatsink (no “+”, the small fan makes high-pitch noise), and tape a Noctua NF-A6x25 on it. Done, cooled, and totally silent.
I would expect a very long life-span for these embedded boards.

For HDDs, I’d go with whatever comes with the best price per TB. Few large drives. And raidz2 if you can front-load the cost. What’s the target capacity?
(Why specifically MG08? If you’re buying new, we’re at MG10.)

For boot, any small and cheap (≤15€) SSD.

A fanless PSU would rely on case cooling. That’s an extra complication. Get one with “hybrid” fan mode: It will most likely not even spin in normal use, but could take care of itself if needed.

Tough one… You cannot make full use of 10G with a few HDDs, but mini-ITX makes it a requirement to have everything on-board, and a -TF is likely not much more expensive than a -LN4F.

1 Like
3

Actually the 304 was my starting point. But I went on to the Jonsbo N3, due to its inner structure. Overall, it seemed a better NAS case. Would you rather take the 304? Any particular reason for that choice?

Thanks a lot for the clarification. So the A2SDi-4C can populate up to 8 SATA ports, but then has no open PCI lanes, right? Although overall I am leaning towards the A2SDi-H-TF, especially due to the 2*10GbE, I still want to consider all options and make a well informed decision in the end.

For the next few years, I think around 15-20TB will be sufficient. Although in the long run, it is hard to say - but I guess, I am not that data hungry afterall. With how many drives would you start raidz2? What are the reasons (besides the larger storage capacity) you would choose raidz2 over 3-way-mirror?

You are right. A while ago - when I started researching - the MG08 was cheaper here in Austria. Right now the MG08 and MG09 are around the same price - so I would go with the MG09. The MG10 on the other hand only seems to be available in very particular sizes (20 & 22TB). The Toshiba MG series I choose, since they offer the cheapest server grade HDDs and have quite good reviews. (I have edited it in the original post.)

Thanks for the insight, that totally makes sense.

This. The extra €30 is definitely worth it.

4

Good, straightforward cooling model, and standard ATX PSU.
I own a Jonsbo N1. I’m totally unconvinced by the screw-on rubber handles on the drives. Cooling and quietness are decent, but not great: The N1 is actually noisier than a Node 304, which holds one drive more.
And I don’t like the N3 having a backplane perpendicular to air flow. In my experience this configuration is either ineffective at cooling or require high power fans.

There’s still the x2 M.2 slot, perfect for a NVMe boot drive. The -2C further lacks this M.2.
No transcoding intended? No app pool?

Four, obviously. But preferably as many drives as your case supports, to avoid raidz expansion and its pitfalls.

That’s a rather big one, isn’t it?
4-wide Z2 = double the capacity of a 3-way mirror for 33% more cost
6-wide Z2 = double the capacity of a stripe of 3-way mirrors for the same cost
You really have to need IOPS (and/or do block storage) to go 3-way mirrors. But then you’d rather look at SSDs.

1 Like
5

How medium is “medium sized”?

Even if you’re not going to reach full 10 GbE speeds, there is strong possibility that 1GbE is not enough for directly editing media files on the TrueNAS system.

FWIW I am getting about 300 MByte/s, when copying a large media file off my TrueNAS onto my desktop using a 10GbE connection. When I repeated this with the same file, transfer speed went at times up to ca. 800 MByte/s, which means the data was partially served from ZFS’ ARC cache the second time around. So the HDD-transfer rates seem to be the limiting factor here, but would still easily saturate a 1GbE connection.

IIRC the minimum is 4 drives per raidz2 VDEV in TrueNAS and I recently started a pool with that configuration.

The problem with 3-way-mirror are twofold IMHO:

  • You waste 2/3rds of drive space
  • When you expand the pool (e.g. with another 3-way-mirror, i.e. 3 drives), you keep wasting 2/3rds of drive space no matter how large the pool.

With raidz2 you can start with a 4-drive VDEV, which has 50% storage efficiency. But with the new raidz expansion feature, you can add a 5th drive and then a 6th, etc. and each time your efficiency will go up, because at all times only the equivalent of two drives will be used for parity blocks.

One small caveat to this: The way raidz expansion is currently implemented, it doesn’t rebuild any parity blocks of existing data, so those will remain stored with the original storage efficiency (that is when the block was written), whereas new data blocks will be stored with the new storage efficiency.

However there is a workaround for that.

And that workaround is a script that will recursively create a copy of each file and move it back to the original location and thereby creating all new blocks.
My guess is that this will mess up your snapshots though, because old snapshots will still point to the old blocks, so I’d expect old snapshots to balloon in size, until they are deleted.

Re. 3-way-mirror: My understanding is 3-way-mirrors provide some performance benefits over raidz2. But I’m not sure, if those would apply to your use-case.
In any case, if that is a concern, perhaps use a smaller 2-way (or 3-way) mirror SSD pool for ongoing edits and - once they are complete - move the files to the larger HDD pool configured as raidz2.

1 Like
6

I agree on networking. Do 10Gbps over 1Gbps. A single drive copying from one machine to another can saturate the 1Gbps pretty easy. Conventional HD to HD copies, I can hit almost 2Gbps on large, zipped files.

posting basics but your probably want to read the VDEV layout whitepaper on different pool options. second link

BASICS

iX Systems pool layout whitepaper

Special VDEV (sVDEV) Planning, Sizing, and Considerations

1 Like
7

Thanks so much for your replies and help.

That’s a great insight, thanks!

Actually, I have another server for these kind of tasks (Jellyfin, arr stack etc.). I want to keep things seperate: My idea is, that the NAS does simply the data storing and the other server the transcoding/editing of the data etc. Or am I missing something obvious here?

Makes sense - just wanted to be sure. Sorry for asking the obvious.

Sorry for being unspecific. I was thinking along the way of music production as well as photo and movie editing once in a while - after a holiday or so. But the filesize is definetly <1GB. Most of the time I am actually coding - where the file size is negligible

Seems 10GbE is the way to go :smiling_face:

This (and considering other factors) will point heavily towards the Supermicro A2SDi-H-TF (edit).

Supermicro A2SDi-4C-HLN4F

That’s actually a great idea. Haven’t thought of it yet. Thanks!

Thanks so much for the compendium. Will definetly read it before setting up my system.

Concerning mirror vs raidz - I think you guys got me convinced of raidz! But then I will probibly look into smaller sized HDDs, since 4*16TB will get me 32TB, which I will definitely not saturate in the next 5 years. Do you have any recommendations - stable and noise being my main concerns? The 12TB and 16TB MG09 seem to cost almost the same.

8

No, I wanted to make sure you’re not going to regret the limited number of exposed PCIe lanes on Atom C3000. For pure SATA storage (possibly including some SATA SSD), you’re going to enjoy a very fine low-power little NAS (if only a bit on the expensive side).

Er… :thinking: Is this a typo for A2SDI-H-TF (10G), or are you thinking about an A2SDi-4C-HLN4F (with a 10G NIC (SFP+?) in the PCIe slot? (In the latter case, 4 SATA only[1], but that appears to be what you’re targetting anyway.)

Then go for 16 TB and enjoy the extra space…

  1. Mind the footnote: “Total combined PCIe lanes and SATA ports is up to 8.” (Not counting the x2 M.2 slot.) ↩︎

1 Like
9

First of all, don’t get any desktop drives or any other drives that use SMR technology. Make sure to get drives that use CMR (sometimes also called PMR).

As far as I understand it, drives with 5400 rpm are on the whole quieter than those with 7200 rpm, however quite a bit of noise also comes from the head moving across the platter. However 7200rpm drives will provide higher sequential transfer speeds (which is also a potential consideration for your use-case, unless you go with an SSD-pool for your ongoing edits).

Also within a series, drives of different capacities might have different noise levels, so check the data sheet. E.g. IIRC my 20TB WD Red Pro drives are less noisy than the next lower capacity. I suppose higher capacity means newer model with improved noise profile.

There appear to be three classes of drives that are suitable, and each vendor names them differently

  1. The “NAS” series (e.g. Seagate Ironwolf, WD Red Plus [Edit: I added the “Plus”. That “Plus” is important, skip the “WD Red” non-plus series, since the drives use SMR technology.] )
  2. The “NAS Pro” series (e.g. Seagate Ironwolf Pro, WD Pro)
  3. Enterprise series (e.g. Seagate Exos, WD Gold)

(I don’t know how the Toshiba models fit into this classification.)

#1 usually is 540rpm (though some Ironwolf models might have even lower rpms), while #2 and #3 are usually 7200rpm. The latter also generally provides a longer warranty period, which is also worth considering.
When I compared WD Gold drives vs WD Red Pro drives for my pool, the WD Red Pro drives were quieter according to the datasheet, so I went with them. (That’s a general thing for Enterprise series drives, since noise doesn’t matter all that much in a data center.)
How different the subjective noise profile between those two series is I don’t know.

Then it might be worth going with the larger capacity models. Some things to keep in mind:

  • The fuller your drives are, the slower write requests apparently will become, because it gets more difficult for the ZFS filesystem to find continuous empty blocks to write to. From what I’ve read, this effect will be really noticeable, once your drives are more than 80% full, but it starts at an earlier fill grade.

  • If you use snapshots on your pool to keep older versions of files around for a while (which you probably should), those will also take extra space, as they store the old versions of data blocks. Depending one how your editing software writes back the data, these might be just a few data blocks that have changed or - if the software overwrites the old file with a completely new version - the entire size of the changed files.
    (Again though with an SSD pool for editing, those snapshots would be primarily on that pool.)

  • And I (like most folks) found out that I ended up needing more capacity than I had anticipated.

1 Like
10

N300 → NAS
MG → Enterprise

Wrong. There are no longer any 5400 rpm drives in the NAS segment. WD sells (or sold?) some drives which it called “5400-class” but which are lower-performing 7200 rpm drives—and measurably spin at 7200 rpm!

1 Like
11

Sorry, my bad, I totally got confused by the naming scheme - although it seems to be quite logical in the end. Yet I couldn’t find a guide - similar to the other types of boards - for the corresponding naming convention.
As you rightfully mentioned, I meant the A2SDI-H-TF. I think it is quite future proof for my use case (12 SATA ports, 10GbE, 1 PCIe 3.0 x4 and M.2 Interface: PCIe 3.0 x2). Yes, it comes with a bigger price tag than I intended - but it’s a perfect match for my project and is very power efficient.

What is the deal with the Enterprise vs. NAS classification/labeling? Is there any real, meaning noticeable difference or is it just a marketing trick to attract consumers and charge higher margins?

Thanks a lot for the overview. These insights are really helpful to get the most out of TrueNAS for my use case!

12

The difference, if any, should be in fine tuning firmware.
As for the marketing trick, it’s maybe not the way you think: Enterprise drives are expected to be sold as bulk, in large quantities, so it is common to find them at retail for less than the more consumer-oriented NAS drives.

1 Like
13

Wow, I stand corrected. :smiley: I had no idea that they did this weird (and IMHO sleazy) change. (Which once again confirms the old saying: The problem isn’t what you don’t know; it’s what you do know, which ain’t so.)

@falseazure please note that I misspoke on the WD Red Series and I also corrected my original post. The WD Red Series is to be avoided, the WD Red Plus series is their real entry-level NAS series.

Taking etorix’ important caveat in mind, the main difference between the NAS Plus and the NAS Pro series (for WD anyway) appears to be the annual workload rating:

  • WD Red Plus drives have an annual workload rating of 180TB
  • WD Red Pro drives have an annual workload rating of 550TB

IMHO 180TB is too low for - say - a 10TB drive. If you have such a drive that’s 50% full and run a monthly scrub and a monthly long SMART test, that’ll already be a workload of 15TB per month, so you’ve exhausted your annual rated workload with maintenance tasks alone.

Have you ever heard and read about more details to this? The firmware differences between desktop drives and NAS/Enterprise drives have been written about (different behavior when encountering an error), but I’ve never read anything about the differences between NAS end Enterprise firmware. I’m curious though as to what those differences might be.

When researching 20TB WD Red Pro (NAS) vs 20TB Gold (Enterprise) drives recently, I only one difference in the datasheet:

  • The WD Red Pro drive specified an annual workload rating of 550TB.
  • The WD Red Gold drive did not explicitly list any such workload rating, but if you look at the footnotes, some calculations appear to be done on an annual workload of 550TB. Some web articles flat-out state that WD Gold has an annual workload rating of 550TB.

When buying a drive it is important to ensure that you are getting a drive from a retail channel, not an OEM channel, because the latter doesn’t come with a manufacturer’s warranty.
And frequently OEM drives find their way into retail as well. Consider asking the seller about this ahead of time, and - before you open the product box - check the serial number of the drive against the manufacturer’s warranty web site.

2 Likes
14

So am I…

As for the difference between WD Red Pro and WD Gold, the main one is that Red Pro are Western Digital drives while WD Gold are now rebranded HGST Ultrastar drives, so different hardware platforms designed by distinct engineering teams.

1 Like
15

Look down here, at UP Motherboards (Embedded):

(Although I still do find the difference between ‘H’ in -H-TF and -HLN4F confusing…)

1 Like
16

Just as a point of reference, and not judging any of your hardware choices, I built the following as a combined Proxmox/ virtualization host/TrueNAS Scale storage server. I couldn’t be more pleased with the results.I was trying to model it on the build that 45 Drives/45 Home Labs now offers, called their HL8

Gigabyte B550I Aorus Pro AX $169.99
AMD Ryzen 5 Pro 5650-GE $159.53
Nemix RAM 2X32GB DDR4 3200 ECC $168.88
M.2 to SATA 6 Port Adapter Card, ASM1166 $42.99
Noctua NH-L12Sx77 $74.90
3x Corsair Individually Sleeved SATA Cable $29.97
DIY 2.5 inch SSD Aluminium Alloy Bracket $9.59
10x SATA III Cable:8 inch Thin-SATA Cables $19.98
Corsair RM Series RM650 $89.99
Fractal Design Node 304 $99.99
10Gtek 10Gb PCI-E NIC Network Card $47.99
10Gtek 10G SFP+ DAC Cable $21.99

Total excluding drives: 935.79. These are the actual prices I paid, and you may be able to do better if you shop around. I stuffed this with 8 Samsung SM863A used enterprise server SSDs. You can obviously do something different if you are going bare metal TrueNAS. In my case, the four motherboard SATA ports are used by Proxmox and the 6 port M.2 to SATA dapter is passed through to TrueNAS Scale. I virtualized TrueNAS in a Proxmox VM and it works perfectly. The Node 304 only fits 6 3.5 inch hard drives, but using SSD drives, I could physically fit as many as 15 SSDs if I wanted to. I found these neat little SSD drive brackets on Amazon that allow you to stack SSDs. I could put 4-6 drives in each of the Node 304 drive caddies. Amazon.com: 2.5 inch PC SSD HDD Cages Bracket Solid State Drive Frame Multi Layer Box Stacking External HD Cabinet Docking Station Base SATA PATA Computer Hard Disk Rack Metal 4-Bay Mounting Bracket Black : Electronics

I am currently running two Wordpress websites, an instance of Nextcloud for my family, A Discourse forum for my wife’s website, Home Assistant, Joplin, Bitwarden, Heimdall, Librespeed, Portainer, Uptime Kuma, Cloudflared, Collabora, Watchtower, and Rsyncd all on the one server. I have plenty of resources to spare. Paperless NGX, Firefly III, Wallabag and either Drupal or Joomla are on my roadmap of things to do.

1 Like
17

Oh and I forgot to mention it only draws 40 watts at the wall and is dead silent.

1 Like
18

Thanks to all for your help and support.

So far, I’ve decided on some parts:

  • Supermicro A2SDI-H-TF. It’s definitely on the pricey side, but it perfectly fits my needs.
  • 4x Toshiba MG09 16TB, since they are on sale right now in my region.
  • Cheap M.2 NVMe SSD boot drive
  • 1x 32GB ECC RDIMM RAM

If the performance for editing files directly on the NAS proves insufficient, I will add a smaller 2-way mirror SSD pool for ongoing edits.

Do you have any specific model/brand in mind?

Thanks a lot for your suggestion - this seems like a very solid build. But I face a somewhat different use case and want a server grade motherboard/CPU.

19

No specific model, but Seasonic is my go-to option in full ATX size, and I’ve nothing bad to report on Fractal Design or Corsair for SFX(-L).
There’s no reason to go insanely expensive, but also no reason to go for the cheapest option on the PSU.

Second-hand DDR4-2400 RDIMM is cheap, so you can go 2*32, 4*16, or even 4*32 GB to fill these two memory channels.

1 Like
20

Thanks a lot for your reply. I would have opted for the Seasonic, but I am probably going for the Fractal Ion+ 2 Platinum 560W, since it is more than half the price of its counterpart here in Austria.

Thanks for the advice: A 32GB stick is around 50EUR - so I’ll definitely start with 2x32GB.

Now that I am finishing my build and focusing on the details, would you replace the factory fans (e.g. by some Noctua et al.)?