Choosing the right vdev layout for my NAS build

TrueNAS General
, ,
1

Dear TrueNAS community,

The past couple of weeks I’ve been reading a lot about DIY NAS builds, different file systems, ECC memory, OS options and more.

I’ve come to the point that I feel I have a broad understanding of the things to take into account when building the NAS, however I could still use some insights about picking the right vdev layout that works best for my specific situation.

I have a 2.5Gb home network with a hand full of devices: a desktop PC for music production, photo editing and gaming, a laptop (also for music production), Nvidia Shield and an Ubuntu server running various applications in docker containers.

The main reasons I want to add a NAS to my home network are:

  • Backing up personal documents and photos (files I won’t be accessing much)
  • Syncing photos from my phone to the NAS (using Nextcloud)
  • Store media for my Plex Server
  • Store music production projects and audio sample libraries, so I can both work on my Desktop and Laptop.
  • Store photos in RAW for editing in Adobe Lightroom

Because I do a lot of photo editing and music production, browsing through files should feel snappy and quick, even when accessed over the network. So the read speed of small files is important to me.

I’ve decided to go for a Mini ITX build. The case I have in mind has 5 external 3,5” bays and 1 internal 2,5” bay. I am aiming for 20 to 30 TB of NAS storage space.

I am able to afford a maximum of 4x 10TB drives at the moment. I’ve got the following ZFS configuration options in mind:

Option 1 | Start with 3 disks

  • Raidz1 with 1 vdev containing 3 disks (2 storage + 1 parity). I could later extent the pool with another vdev with 2 mirrored drivers.

Pros:

  • Room for expansion
  • I can use all 5 bays in the near future
  • Lower initial cost (3 drives instead of 4, compared to my other options)

Cons:

  • When I want to extent my storage capacity in the near future I get 2 different vdev configurations in the pool (a raidz1 vdev and a vdev with mirrored drives). Not sure how this impacts read/write performance.

**Option 2 | ** Start with 4 disks (and leave 1 bay open for some other purpose like L2ARC)

  • Raidz1 with 3 disks storage + 1 parity disk

Pros:

  • Storage capacity efficiency (3 out of 4 disks of usable storage)

Cons:

  • Not ideal number of drives in a raidz1 configuration considering the “magic” (2^n)+p formula. (Not even sure this is a con when using lz4 compression cause block sizes will vary anyway when compressed)
  • Can’t use the 5th bay for storage with data redundancy.

Option 3 | Start with 4 disks (and leave 1 bay open for some other purpose like L2ARC)

  • 2 identical vdevs (mirrored), with each 2 striped disks.

Pros:

  • From my understanding this configuration has the fastest read/write performance because of increased IOPS.
  • Faster resilvering in case a drive fails (not sure if it makes a big difference compared to Raidz1 with 4 disks)

Cons:

  • Lose 50% of storage capacity
  • Can’t use the 5th bay for storage with data redundancy.

NAS build I’ve got in mind:

Case: Jonsbo N2 (ITX)

  • 5x 3,5” and 1x 2,5” bays

MOBO: ASRock B550M-ITX/ac

  • 1x PCI-e 4.0 x16
  • 1x M.2 (PCI-e 4.0 x4 + SATA), 4x SATA-600

CPU: Ryzen 5 Pro 4650G (already bought this)

RAM: Kingston KSM32ED8/32HC (will upgrade to 64GB total later)

Storage:

  • 1x SSD (Truenas boot disk)
  • 3x or 4x 10 TB HDD 7200RPM (prob. Red Pro or IronWolf)

Right now I am leaning towards option 3 because of the increased IOPS. However, maybe I won’t even notice a difference because of how ZFS uses caching. What do you guys think?

I prefer speed over storage capacity. If possible, I’d like to have the feeling as if I am working from my local SSD. What do you guys think? Do you have any additional advice? Which option should I go for, or maybe there are other options that I haven’t thought about.

Thank for taking the time to read my post! :slight_smile:

2

Option one is out, you can’t combine different VDEVs in a pool. That would require 2 pools.

3 drives in a Raid-Z1 isn’t recommended, for large drives, unless you have backups of all data, due to possible pool failure when resilvering.

4 drives. Your choice is 4 x drives in Raid-Z2 or two VDEVs of 2x drives in Mirror pairs. Both give you two drives out of four for space.

iX Systems pool layout whitepaper

3

I reckon with 32GB it’ll “feel” like an SSD most of the time. The initial read from disk might seem laggy but after your working set has been read once and cached I predict you’ll be happy with the results.

I see the planned workload is a mix of “interactives” where a human sits at the keyboard and waits for the beeps to boop and “non-interactives” that take place in the background. Two questions:

  • Will non-interactives generate storage traffic sufficient to expire interactives out of the ARC cache? If so, can this activity be deferred to off-hours when everyone’s asleep?
  • How large is the interactive working set? Are these files many GB? Tens of MB? Ideally there’s enough cache to contain whatever you’re working with at the moment. You mention audio and photos (but not movies) so I reckon your files max out in the low tens of MB. 32GB RAM would be plenty.
4

Thanks for sharing these resources! The pdf gives very good practical examples of common layouts. Also good to know that I can scratch option 1 and 2.

It’s good to know that the mirrored vdev option does improve read/write speeds to some degree.

Good questions!

The non-interactives won’t be using that much traffic, so I am not worried about that. But I guess they will expire the interactives from time to time.

The interactive working sets vary in size depending on the project. I’ve got some instruments with audio libraries as big as 60GB, but I don’t think the whole set needs to be cached in memory, since I am only using it partially at a time.

I do store some movies as well, but I watch them rarely.

I am planning on upgrading to 64GB RAM once I’ve got some more to spend, especially if that increases performance.

5

Focusing on the client aspect for a moment:
I’ve noticed limited client caching when working with files on ordinary SMB shares. It seems an SMB client will read a file once and hold a copy in cache as long as the file remains open (ie. the lease is still active). Returning to a file after its been closed will invoke another transfer over the network. No luck finding knobs to force clients to cache more aggressively.

I had thought the client would simply check with the server to see if the file changed since the initial “download” but I guess this concept doesn’t exist in the world of SMB? Maybe someone else who uses SMB daily can chime in.

I say this because you might not get an SSD-like experience if your workflow involves reading the same large file(s) repeatedly without holding it/them open in the interim. No amount of zpool architecting or RAM in the ZFS host will cover this up.

My own use case is Stable Diffusion .safetensor files stored on my ZFS server. Quite often I rotate across 3-5 checkpoints which tend to run 6-7GB each. Due to repeated “downloads” from the server I had to abandon SMB and return to block storage via iSCSI just so my client could cache the content as if it were reading from a local drive.

6

This formula is long invalid.

L2ARC is all about speed, so NVMe would be much preferred. The best use of 5 bays and 4 drives at start would be raidz2, for resiliency, planning on future expansion.

Variation: Swap to a Node 304 case (likely better cooling than the N2): 6 bays. Start with a stripe of two mirrors, extend later with a third mirror.

Pointer to striped mirrors…

Ditto. Or even to a further special vdev. You have a x16 slot which can bifurcate to x8x4x4 for that.

7

You perfectly well can - only the TrueNAS UI keeps you from doing it.

Any performance “oddities” that might arise from that are of course yours to deal with, in that case.

But you can add arbitrary topology data vdevs to a single pool.

8

As far as I know the application I use for music production (Ableton Live) manages local caching. So I am not too worried about that.

The node 304 seems to be a good alternative. When I get home from work I’ll check if the case fits my limited space. But the 304 might be a better option considering the 6 bays.

So what you are saying is that each vdev consists of 2 disks that are mirrored. And for each extra vdev with mirrored pairs of disks I get a performance boost. I can start with two pairs and then later add another vdev.

Unfortunately, I need the free PCI slot for a 10Gb network card, since the ASRock B550M-ITX/ac only has an 1gbps ethernet connection. I think I’ll get an M.2 to SATA adapter to get 4 extra sata slots. Too bad I can’t use a NVME SSD.

Thanks again everyone for replying and helping me out. I really appreciate it.

9

x8x4x4 riser: one x8 slot on top for a half-height NIC, two M.2 on the sides for a special vdev mirror—all on CPU lanes.

1600×1600 229 KB

11

The node 304 barely fits the available space. So no room for the air to flow on the back of the case. That’s why I am now considering the Jonsbo N4 case:

image
image791×652 42.6 KB

This way I can go for a Micro ATX motherboard, which offers more options for future expansion. I was thinking about the MSI B550M PRO-VDH:

image
image843×587 54.7 KB

The board has two m.2 slots so I could use them for a special vdev in the future.

What do you guys think?

12

@etorix, How do you arrive at OP’s Mob/CPU combo supporting x8x4x4 bifurcation? That would be great! I’ve been searching for specs but haven’t found mention of this. Thanks

13

The cooling design on the Node 304 is pretty good.

It sucks air through a filter from the front, blows it across the drives, and then has a large exhaust fan to blow out the back, which provides plenty of room for a PSU, CPU and GPU with coolers. There’s room for a full height, half length, double width GPU… Or if you use the riser card linked above, a half height.

Meanwhile, there’s enough room above the CPU for large cooler if that’s your thing.

My Xeon D X10-SDV just needs a tiny cpu fan.

1 Like
14

That’s what the 4650G APU is capable of. I haven’t looked through the motherboard’s documentation to check whether this is actually supported. (Hopefully so, but I wouldn’t expect a gamer/consumer motherboard to be properly documented, so booting into BIOS and looking into options may be the easiest way to figure out.)
This also depends on the case, as the riser is for a half-height card in a full-height slot; it won’t work with a case which only provides half-height slots.

15

Understood.
I’m still on the hunt for a small, 4 * nvme platform for a TrueNAS build. Low power, 8 or 12 thread CPU, 64GB RAM. Reliable and durable.
Anyways, thanks again

16

X10SDV (Xeon D-1500): Low power (45 W max. for a 8C/16T D-1541), cheap DDR4 RDIMM. x16 slot bifurcates x4x4x4x4. Often comes with on-board 10G.
(ECC capable) AM4 or AM5 Ryzen (ECC capable) CPU (not APU) to bifurcate x4x4x4x4.

1 Like
17

Thanks I’ll have a look at that supermicro!

What’s a good resource for finding out what bifurcation modes are supported by the various AMD CPUs?

18

Generally:
CPU (chiplets, ECC support, no iGPU for 5000 and below) x4x4x4x4
APU (monolithic, ECC only for PRO, iGPU) x8x4x4

with the caveat that AMD has released some APUs with a disabled iGPU in the desktop CPU lines, a telltale sign being that these do not support ECC.
Then, it’s also a matter of implementation by the motherboard.

19

Thank you all for the valuable insights.

I’ve just received most of the components for my build. For now I’ve decided to stick with two HHD’s for the storage pool in a mirrored configuration, and later on expand with extra sets of two disks.

I’ve decided to go for the following setup:

Case: Silverstone CS351
MOBO: ASUS TUF Gaming B550M-PLUS
APU: AMD Ryzen 5 Pro 4650G

U-DIMM (ECC)
1x Kingston KSM32ED8/32HC

SSDs (Proxmox installation drives, mirrored) – also for the VM’s and a ZVOL for the Truenas Scale installation
2x Crucial MX500 2,5" 500GB SSD

HDD’s Mirrored VDEV for TrueNas Scale
1x Seagate Exos X18 (SATA, Standard model), 14TB
1x Seagate Exos X18 (SATA, Standard model), 16TB (I am aware I loose 2 TB of space)

HBA card (If possible, I leave this one out)
1 x Dell Perc H310i SAS/SATA HBA controller in IT-mode (LSI 9211-8i)

GPU (for PCI-e passthrough)
ASRock Intel Arc B570 Challenger 10GB OC

My goal
Because the build got a bit more expensive then expected, I want to get the most out of the hardware, and therefore I want to virtualize Truenas Scale in Proxmox. I’ve got my hands on a HBA card to be sure I can pass through the HHD’s according to the suggested method.

However, by doing so I lose the PCIe 3.0 x16 slot which I might want to use for a 10gbps NIC.

My question is: is there another way to pass through the HDD’s to the Truenas Scale VM? By for example connecting the two (4 in the near future) HDD’s to the M.2 slot with a SATA expansion card.

In the manual I read the following under storage:

image
image793×321 60.3 KB

Does this mean the M.2_1 slot is directly connected to the CPU? And if so, is it safe to connect the HDD’s for the storage pool to a M.2 SATA adapter card, and pass that through to the VM, or should I stick with the HBA card?

I was thinking about something like this:

image
image760×399 66.4 KB

20

The only good way to present HDDs to a virtualised TrueNAS is to passthrough their controller.
The recommended controllers are LSI HBAs and embedded SATA controllers from Intel/AMD chipsets. SATA controllers on M.2 cards are frowned upon around here. A better solution would be a M.2-to-PCIe slot adapter for the HBA.
Anyway, your 4650G Pro APU is Zen 2 so I suppose it lacks the lanes to feed M2_1 (AMD generations and code names utterly confuse me, so please double check that).

With that, possible courses of actions are:

  1. Use the LSI HBA;
  2. Put a M.2 module with an ASM controller and no port multiplier (check carefully) in the M2_2 slot (B550) and PRAY to the ZFS gods; or
  3. Put Proxmox and its VMs on NVMe so that you can pass through the B550 SATA controller.

Number 3. looks like the cleanest way with the least hardware… but your choice of a gaming motherboard may come back to bite you if the SATA controller is not isolated in its own IOMMU group.
If the board supports x8x4x4 bifurcation on the x16 slot, you can host two M.2 NVMe drives and a low-profile NIC on a suitable (and cheap) riser.

21

Thank you for your quick reply and laying out the options for me.

I am gonna see if this applies to my board. Not sure how, but I’ll try to figure this out.

This is what the manual says about bifurcation. But nothing about x8x4x4 so I am not sure if this is possible.

image
image791×237 41.8 KB

Also I read the following (but no specific info on 4th gen Ryzen PRO APU’s):

image
image931×485 111 KB

If it turns out to be an issue, i’ll stick to using the HBA card and use the onboard 2.5Gb/sec Realtek NIC.