First - apologies for the length post. I’m new here, and it’s my hope that the additional info will prove useful to get the correct recommendations/configuration advice!
A bit about myself, to “set the stage”, so to speak. Hopefully this helps with where my strengths are, and where my knowledge gaps are, so we can use some common vocabulary to find the answers to my questions.
I’m a long-time enterprise IT professional and consultant. Almost 30 years in. I’ve worked with everything from old school 50-pin SCSI up through U320, twin-tailed SCSI enclosures for clustering, Fibre Channel, iSCSI, hyperconverged systems including Simplivity, Nutanix, and HyperFlex, and just about everything in between. Built my first PC in the mid-90s with a Pentium 133, built my first personal SMP system with a Tyan board and a pair of Pentium 200s. Most familiar with IBM and Cisco server hardware, but a good bit of HP as well. Can’t stand Dell (sorry!). Arrays include IBM, EMC, 3PAR, Nimble, Pure, NetApp, and others.
The use case:
Home use. I record and edit videos as a hobby, and that obviously generates a lot of files - some large, some small, but between that and storing my NVR footage, I consume 6-8TB/yr pretty easily, and that’s with retention turned on in the NVR. My editing system has 6TB of NVME drives in it, and I offload them currently to an OLD Buffalo NAS that I bought through a previous employer for a song. The drives have been upgraded over the years to increase capacity, but it’s long in the tooth. Having recently undergone a network upgrade here at home, it’s time for a NAS upgrade.
I briefly considered several “off-the-shelf” solutions from QNAP and Asustor, among others. For the price of the models that would do what I want (compression & tiering), it would appear that I can roll my own for the same money and get more storage (or at the very least - grow into more storage).
I’ve watched some YouTube videos, I’m read a bunch of stuff, and I have a “loose” understanding of TrueNAS - but the finer points elude me - including hardware selection. I’ll admit to a good bit of unfamiliarity when using “consumer grade” or “white-label” server hardware - and I’m also quite confused about SLOGs, vdevs, pools, and the myriad types of “unconventional” RAID setups that are available. I know that more memory is better, and I know that more processing is better, and of course, SSD is fastest - but I want capacity as well, so I’m not planning to buy all SSD (plus - the cost!). I’m most confused about where to place these various system partitions/arrays, how many disks to use for each, what capacity they should be,
My budget is what I would consider “healthy”, but certainly not a “money is no object” type of mindset.
(edit because I’m able to post links now)
This is the system I’m looking at:
Trying to decide between the 2x Xeon E5-2697 v4 (2.3Ghz 18-core) and 2x Xeon E5-2697A v4 (2.6Ghz 16-core).
For compression/deduplication, do I want more cores, or higher clock speeds? Uncertain which processor to pick. Price difference is negligible, so it’s a design choice, not a cost consideration.
For RAM, I’m planning 256GB (4x64GB) ECC. This should leave plenty of room for expansion later - though I’ve read I only need 1GB per TB? Is this not the same as traditional systems where more is better? If no one sees a scenario where I need more than 256GB, then is there a performance gain to be had with 16x16GB sticks vs 4x64GB?
Will I need a storage controller, or will the 10 on-board SATA ports suffice? Available from the same source is a “12Gbps IT mode PCIe Storage Controller (flashed to IT mode; No RAID Pass-Thru Only)”. I don’t know if those on-board SATA ports can correctly drive the hot-swap backplanes? I’m especially confused by this storage/controller bit since with conventional, “datacenter” servers, you use a RAID controller (or FC HBS or iSCSI TOE adapter), so software RAID/JBOD is foreign to me.
I’ve read that compression is good on TrueNAS, but deduplication is bad, unless I can store the table in RAM or something like that? Is this accurate? The discussion was in reference to performance. I’m also interested in resiliency - like, in the event of mainboard failure or some other catastrophic failure, if I’m doing compression and/or deduplication, will I be able to simply replace the failed component or will this require a restore of all data? Is there any single component (or set of components) which could fail that would result in data loss? If so, how to plan around it?
My primary interest is in data integrity/resiliency, and my secondary interest is in speed. If possible, I’d like to be able to directly edit video across the wire without having to copy it back and forth.
Additionally, I plan to use something like FolderSync to automatically sync photos and videos from family devices for backup purposes. Aside from the FolderSync connections, there will only be 2-3 concurrent connections to this new NAS.
Since the mainboard in that server supports bifurcation, I plan to add at least 4, but as many as 8 or 10 M.2 drives.
I think I need some type of SSD for boot, yes? Would this best be 2.5" SSDs in one of the regular drive cages, or M.2 drives? If M.2, one on each of two PCIe adapters, I would presume? If 2.5" SSDs, one in front and one in back, to cover backplane failures?
Do I need an SLOG for this config? I’ve read/seen conflicting information. If I do, that should be SSD as well, right? How big does that need to be? 1TB? 2? 4? I see recommendations for SLOG mirrors, and even multiple mirrors. So, 4x1TB? etc.
I definitely want a hot spare for the pool. My understanding is that vdevs are combined to create the pool, and hot spares can be shared across all vdevs in the pool, right?
I saw a video where a guy espoused the fact that the ideal vdev size is 10 drives. Is this accurate? With a possible 36 3.5" drives, I could do 3 10-drive vdevs, and that would leave me six 3.5" bays for hot spare, and whatever else, and I’d still have the M.2 drives for SLOG, boot, and ZVOL (whatever that is)?
What I have to work with:
10gig SFP+ switch with both copper SFPs and DAC cables
Rackmount UPS with plenty of capacity
No fear of learning/tinkering
I’ve read/heard/seen that additional vdevs can be added later to expand capacity, so to start with my understanding is to buy 10 drives, create a vdev using all 10, then add that vdev to a pool and viola! I have 8/10 drives worth of capacity available, right? Specifically, I’m thinking 10 CMR drives that are 6TB ea, SATA 6GB/s, 7200 RPM and 512MB cache. How many hot spares? Can I add additional hot spares later as I add more vdevs, or do I have to define the number of hot spares when the pool is created? If I can do it as I go, then I’d do 1 per vdev as I build them. If I have to define them all up front, that will prove more problematic as the next vdev might use larger capacity drives, and so on with the third…
What about all the other things that I’ll need? SLOG, boot, ZVOL, ARC, L2ARC… Do I want to create a VDEV of just SSD and add that to the VDEV of spinning disk for tiering/caching?
I’m currently consuming ~30TB of storage, so the above ~48TB from the ten 6TB drives will buy me about another 2 years or so before I need to add another vdev.
Please, help me gurus: Given the chosen system above, what procs, how much memory, and what controller would you choose (and why)? What types of drives do I need for all these other functions besides just capacity, and should they be 2.5" SSD or NVME, and how many of each and what capacity do they need to be?
I do not plan to run any VMs on this, or use it for any other purpose than file sharing (SMB - POSSIBLY iSCSI in the future, but no immediate plans).
Also, if anyone knows of any easily-consumable resources for learning the finer points of truenas, that would be great. I don’t need something as detailed and dry as say, Cisco documentation, but I’d also prefer something a bit more detailed than “bUiLd ThIs!” I say easily consumable since I’m usually quite pressed for time, and usually can’t dedicate more than 15-20 minutes at a time to reading/watching something without interruption.
If you’ve made it this far - thank you! I’m excited to hear any/all recommendations the forum may have for me.