Finishing my first server build – input on final drive purchase and pool config appreciated

TrueNAS General
1

I’ve been experimenting with TrueNAS over the last month, and decided this is the tool I’ll use for my storage, backup, and docker needs. When I started exploring the options for a home server, I didn’t get all drives I needed, but just enough to start learning. But know its time to get the missing storage space needed to take it into “production”.

My server is built around a SuperMicro A2SDi-4C-HLN4F motherboard. Currently with 2 x 1 TB NVMes, primarily as a pool with a mirrored vdev for my Docker containers, and 2 x 4 TB HDDs, also in a pooled with a mirrored vdev. (And a boot drive connected to the moderboards internal USB 3.0 port.)

I plan to use my server for three things:

  1. Primary storage for my photos.
  2. Backup target for 4-5 computers.
  3. Hosting a couple of services for my family, like Searxng, Readeck, n8n. Nothing exposed to the internet, remote access using Tailscale only.

Current storage need is around 1 TB for my photos and around 6 TB for the backups. I don’t expect to storage need for either to grow much near-term, but since I have 5 TB in the server right now, I need to get more.

My motherboard have 8 SATA ports, but given that the PCIe slot share lanes with them, and I use that for the second NVMe, I can only add two more HDDs.

There are so many ways to think about this, and would like input from more seasoned TrueNAS users!

Keep photos on a separate pool

Get 2 x 8 TB (or possibly 2 x 12 TB for more room to grow) and add as a new pool, with a mirrored vdev. This would then be used as the target for backups, and while my current pool would be for my photos only.

Since I’ll only work with my photos a couple of times per month, that would make room for spinning down the photo pool for power savings. The downside is that this would leave around 2,5 TB of more or less unused space. Could perhaps be used for another family member’s photo collection as well, with a similar usage pattern.

Put everything on a shared pool

I could also use my current drives as the first two of a single raidz pool to be used for both photos and backups. If I go this route, the options are plenty:

  • Get a third 4 TB drive, configure a raidz1, and expand that pool with a fourth 4 TB drive when I outgrow 3 x 4 TB.
  • Get a 8 TB drive, configure a raidz1 leaving half of the new drive unused for now, but laying the ground for replacing the two 4 TB drives with 8 TB further down the road.
  • Get two 4 TB drives, and go with raidz2. This makes least sense for me, since growing my total storage from there would be the hardest/most expensive. If I want raidz2, perhaps getting 2 x 8 TB for starters would be better, even if that would leave a lot of unused space for now.

What should influence my descision?

So, that’s the options I’m pondering. But no matter how much I read on the forums or elsewhere I can figure out on what factors I should base my decision. :slight_smile:

My server have 32 GB of RAM, and together with my current HDD pool (with 5400 RPM Ironwolfs) that’s enough to saturate my 1 Gbps ethernet. So I guess performance doesn’t have to be factored in?

1 Like
What is the cut-offs for drive sizes / vdev width where you think RAIDZ1 is not an option?
2

Good motherboard (although the limited number of flexible I/O lanes is a downside compared to C3758 boards), good planning and complete analysis of the situation. Congratulations!

Mind that ZFS pools are best kept below 80% occupancy.

You only have “bulk” storage, so a single raidz# pool makes more sense. Whether you further backup the photos only but not the backups is another matter.
Selectively spinning down two drives does not saves much. To keep power low, the key is to have few drives—so larger drives.
And then there’s resiliency. The PC backups possibly need no further backups. The photos do need further backup, especially if you go for raidz1 (or a mirror of 12 TB drives by the way…) because it is not that resilient (lose one drive and you’re at risk).
Since you cannot upgrade raidz level, the safest way is to start with raidz2, meaning more drives. ZFS forces to fronload costs…

An alternative would be to either keep the apps on a single drive (with frequent backup to the HDD pool), or switch to (a pair of) SATA SSDs so you can disable the x4 slot and enable all SATA ports. That would give 6-8 ports for the HDD pool.

With 1 GbE and a relatively small pool, 32 GB RAM is probably enough for storage and apps.

3

Thanks for the heads-up! Had forgotten about that part.

What do you mean with “bulk storage”? As compared to what other possible use cases for a NAS?

Photos will definitely be (and already are) backuped in different ways, both to cloud storage as well as local usb drives. Same holds for the computer backups: They are already on at minimum two different cloud storage solutions as well.

Yeah, I get that. But since I will need either four drives (the two I already got + two more) or scrap the two I already got and get like 3 * 8 TB to get the storgage I need, at this point in time, spinning down the photo pool and keep the backup pool up seems like a valid option? Getting rid of two drives with just a months usage seems like wasting money. :slight_smile:

I’m a bit hesitant about going raidz2. Everything on the NAS is backuped in various ways and nothing is time critical. Should a second drive fail during a resilvering of a raidz1 pool I can still restore from backups, only will take longer to get the system online again.

Really happy you mention this as a possible option as well! That’s something I’ve consider in a more distant future if/when I need more than 4 SATA ports. But right now I can manage with 3-4 HDDs.

A somewhat related side-note:

I ended up with my current selection of disks, two NVMe and two HDD, because I thought I would go with Unraid. When I read up on the alternatives, I saw many, many posts claiming that TrueNAS is more complex than Unraid. So I got what would make sense for Unraid.

But after realising that Unraid was quite simple to get up and running and that my use cases (protecting my own valuable data rather than hoarding terrabytes of tv shows) made zfs the most sensible choice for file system, I decided I should at least give TrueNAS a go, judging wether I would find it that much harder than TrueNAS. Turned out it wasn’t.

Yes, there are things to understand when it comes to zfs, but those are the same no matter what OS you run. The only true difference between Unraid and TrueNAS I ran into for my use cases was the permission part. But that’s not that hard either.

1 Like
4

As opposed to “block storage” (iSCSI, or zvols for VMs) where the host provide raw blocks for the guest to manage its storage rather than serving files. :wink:

If you understand the increased risk, and have a solid recovery strategy, then going for raidz1 may be a perfectly valid decision.

In my (not so) humble opinion, spinning drives under 10 TB are now a total waste of money in any scenario.
To add capacity, you may use raidz expansion to add more small drives… until you cannot for lack of ports. Then you have to replace ALL drives in the vdev with larger ones; the wider you grew through expansion, the more costly the replacement will be. So better start with few large drives.

Again, congratulations on your reading efforts.
Another relevant difference between Unraid and TrueNAS (ZFS, really) is that the former is very welcoming to “bring more drives as you go, whatever comes at hand” while ZFS requires planning in advance, extension by full sets of homogenous drives, and, as much as possible, frontloading storage costs.

5

Out of curiosity: How would the need for zvols or iSCSI change how one should think out one or more pools?

I have come to a similar conclusion. Wish I had had a better understanding when getting the two 4 TB drives back in April. :slight_smile:

I’ve double-checked the size of my current backup sets as well as my photo library. They clock in at 6 TB. The backup’s are fairly static in size, and I’m currently purging my photo library so that might actually decrease a bit near-time.

This mean I could probably to with a 3 x 4TB radiz1 for a file. If I put my photos on the NVMe mirror, I need around 5 TB storage for backups, which is within the 80 percent usage limit I would get out of that vdev. But there is a non-negligible probability that I would sooner rather than later find that I need to expand that vdev.

Would it cause any trouble to get a 10 TB drive for now, to get the raidz1 pool started with 10 + 4 + 4 TB drives, and replace to smaller ones when I need more storage?

6

And just to double-check: One way to increase the capacity of a vdev is to replace each drive with a bigger, one-by-one, where after they all have been replaced the vdev capacity will match the new, combined raw capacity of the new drives?

7

Block storage (and databases) is all about small transactions. Raidz is very inefficient at that, due to stripe width and padding. So if you had any of these, you’d want a mirror pool for it, alongside a raidz pool for sharing (big) files through SMB/NFS.

No. Vdevs are limited by the smallest member, so you need to replace both 4 TB drives to expand but there’s no drawback (other than not using the capacity of the largest members).
CORE expands automatically; CE now forces you to press a button.

8

Of course you could - but the 10TB drive would count as a 4TB drive until the other two get replaced.

9

Great! I had understood that the smallest drive in a vdev is the limiting factor, but not if it would cause any problem to mix drive sizes. Thanks!

10

After further research and consideration, my plan is to purchase 2×8TB drives and create a four-wide RAIDZ2 array. Combined with my existing two 4TB drives, this would provide approximately 5.5TB of effective storage space after accounting for the 20 percent ZFS overhead.

Together with my 1TB NVMe pool, I’ll have sufficient space for now, with a clear expansion strategy:

  1. Acquire two additional 8TB drives and upgrade the RAIDZ2 pool to 4×8TB for approximately 11+1TB.
  2. Remove the NVMe drive occupying the PCIe slot and reinstall the two 4TB drives as a mirrored pool, resulting in 11+3+1TB (while maintaining frequent snapshots of the app data from the NVMe to one of the redundant pools).
  3. Further expand the RAIDZ2 pool if storage requirements exceed 15TB in the future.

When considering price per terabyte, performance, power efficiency, and recommendations for RAIDZ levels relative to drive sizes, 8TB drives appear optimal for my requirements.

While I’ll pay slightly more per TB compared to 10-12TB drives, larger drives typically necessitate RAIDZ2 configuration, and four drives exceeding 8TB would provide more storage than I’m likely to need while doubling energy consumption. I don’t require the performance of 7200 RPM drives, which typically consume around 10W when active and 8W when idle, whereas 5400 RPM drives use approximately 5W and 4W respectively.

Have I overlooked any factors that might alter my approach?

11

Well, you can spin down the single pool too.
Most likely, your server will not be accessed by anybody for example during the nights.
(I assume, your windwos backups happen during business hours.)
SO after you go to sleep, you can set up TN to spin down all drives until the morning. (like give them a 1 hour time-out before spin down.)
If you still need access, the HDDs will spin up upon request.

12

…your second backup can also fail during recovery…

13

There are no longer any 5400 rpm NAS drives.
WD’s marketing, however, may want to sell you 7200 rpm drives as “5400 rpm-class” thanks (?) to a performance-limiting firmware.
And make sure you do NOT get SMR drives in these capacities.

14

There are both Ironwolfs (like ST8000VN002) and WD Red Plus (WD80EFPX) with 5400 RPM available in Sweden at least. Or is there something with these models I miss?

1 Like
15

Red Plus is good. Just be aware that there won’t be any 5400 rpm option moving to larger capacities—as should inevitably happen…

16

I think I’m good with a four and eventually five wide raidz2 built with 8 TB drives – even long term.

The easiest and cheapest option right now, given that I already have 2 x 4 TB, would be to get two more of them, for a 4 wide raidz1. But that would paint me into a corner with raidz1. How big drives do one dare to have before raidz2 is the sane alternative?

Or keeping my 2x4 TB mirror and adding a new pool with a 2x8 TB mirror and live with a more complex expansion sometime in the future.

A 5 wide 4TB raidz2 could also be an alternative, but that would as, previously discussed, mean I would have to get rid of my NVMe mirror for appdata. Could replication from the NVMe to the spinning pool like once every 30 minutes compensate for that?

This decision is causing me headaches right now… :slight_smile: