Currently I have a primary system with 2 x 6-way Z2 vdevs with 4TB disks. I went this route originally as I had a lot of spare 4TBs sat around and it gave me good storage size with efficient block usage (record padding).
My secondary (backup) system is 1 x 6-way Z2 with 8TB disks, and I’m wondering whether I should move my primary system to a similar setup, albeit moving from the 4TB disks to 12-18TB ones (price and availability in my country).
My reasons for doing so are…
I’m getting closer to full than I’d like - a couple of years tops before it gets to “too full” - so need to think about expanding
Updating or replacing 6 drives is more palatable than 12
12 drives is double the drive power usage - ball park 30-40W all day every day
12 drives is a lot more heat to dissipate
A lower drive count gives more case options, albeit nice smaller cases would run hotter.
Also, in a natural disaster (we’ve had them in the region) of flooding it would be nice to just grab a 6 drive system and load it in the car than unrack a 12 drive system and have to get help carrying it. B2 backup is for vitals and nobody lives in a suitable location to put the secondary offsite. Secondary is for convenience.
Both systems have mirrored a SSD pool where container, database and VM data is stored. The primary system has 128GB RAM and a i5-14500. The secondary has 64GB RAM and a i7-7700k.
The HDD storage is used for photos, videos, media, and general files.
What impacts would I notice moving from 2x6 Z2 to 1x6 Z2 with larger drives? Guessing I’d notice a rebuild with 4TB vs 8TB vs 12TB disks.
Should I consider a different layout like 1x8 Z2 with 8TB? It is cheaper than a 1x6 Z2 with 12TB disks for instance. Would the block padding bite me there? Does the 128GB RAM negate many negative effects?
Trying, like everyone I guess, to find the sweet spot between performance, price, capacity etc.
Taken as read that the simplest solution to expansion is always “stop hoarding”.
This will mostly depend on your workload. For a simple file share: Probably not much difference, 2x6 Z2 isn’t really a “fast” configuration either.
128GB RAM is enough for an L2ARC, so you could try to go down to 1x6 Z2 12TB and if you’re not happy (and arc_summary says you have too many misses) add some L2ARC later (maybe ~2TB, possibly set to MFU only).
A special device is also an option, but again if you’re mostly happy with 2x6 Z2 the difference between 200-300 and 100-150 IOPS isn’t that big. So that’d only be a recommendation if you’re currently unhappy with metadata responsiveness (setting permissions, working with tons of small files, BRT/DDT, etc.)
As for 6-wide vs 8-wide: If your primary goal is space and power savings a 6-wide pool shouldn’t be so much more expensive that it wouldn’t be worth it. There are some ITX cases that comfortably fit 6x3.5" and even some SSDs. 8x3.5" is often more expensive, larger or very crammed. 1x8 Z2 padding is slightly less optimal, but with 1M records not by much. The primary reason to go for 8-wide is if you really want as much capacity as can (reasonably) fit in a “normal form-factor” desktop/tower style case.
That usually happens so infrequently with such a small number of drives that I would ignore it.
As I understand it, the issue of block padding and the “ideal” configs now carry much less weight than they used to. I would focus instead on the record size of each data set and ensure that the right size is used by use case - i.e. I use 1M for media files, for example.
Your point re: portability, noise, etc. is well taken. You could consider a setup where the disks are stored externally in a dedicated 6-drive JBOD where every drive has a external eSATA port or there is a real card on the inside. Shy away from jMicron and like port multiplier setups, however. Those drives have to be exposed to TrueNAS for scrubs, SMART, and so on to work.
I would go with fewer, larger drives for the reasons you raise, ideally filled with He to lower power needs, heating, and increase lifespan. one VDEV vs. two is unlikely to make a significant difference in a home setting. My rig gobbles up media files at 250MB/s and it’s a single HDD VDEV with optimized record sizes supplemented by a sVDEV that deals with small blocks and metadata.
