I’m building my first Truenas (scale) system. Hardware consists of an old Dell Optiplex with 4th gen i5 and 2 PCIe slots, 16GB of RAM, 3 SATA III ports, and an LSI SAS card connected to an external drive cabinet with 16 x 8TB drives. The SAS card is in one PCIe slot and in the other I have a 500GB NVME SSD. So the system (except for the external cabinet; I still have 8 slots available) is essentially maxed out (I could theoretically upgrade to 32GB RAM but I don’t think I want to spend any more ATM). I’ve done a test install and the system works.
My initial thought is to install the OS on one 120GB SATA SSD (I have larger SSDs but I don’t know if there’s any point for the boot drive), mirror two 240GB SATA SSDs for applications and data, and use the NVME as a cache for the external drives. Since the hardware is pretty old I’m thinking I probably don’t want to load it down with too much; just DLNA, SMB and basic maintenance stuff. Might consider using Jellyfin for audio; I don’t plan on streaming video outside of my LAN, so DLNA should suffice.
It is helpful to also list your desired / suggested pool layout. Some people select too wide RAID-Zx stripes, like >12 disks. Your 16 disks would be a bit wide for RAID-Zx, but using 2 x 8 disk RAID-Z2 seems a bit too conservative.
However, you mention that you have 8 additional disk slots in the external enclosure. That would be perfect for growth. So either of these could be your starting point:
Symmetrical:
2 x 8 disk RAID-Z2 vDevs, where you can later add disks to both, alternating
Asymmetrical:
1 x 12 disk RAID-Z2 vDev
1 x 4 disk RAID-Z2 vDev that you can later add disks to
Now the symmetrical layout would give the best performance, both initially and over time. However, the 4 disk RAID-Z2 would expand faster than an 8 disk RAID-Z2, at least initially.
What kind of cache? L2ARC, SLOG, Special VDEV? Generally, unless you have a specific usecase (or have been running the server & see something like ARC hit ratio being low) there is no real benefit/a chance to degrade performance/single point of failure for the entire pool.
Avoid the cache unless you know you need it. This isn’t as simple as ‘more hardware, more betterer’ that many initially assume.
The rule of thumb in terms of increasing performance without a specific scenario is that increasing RAM is the best option for zfs.
Techno Tim (look on YouTube) is deep into truenas and he can explain his layouts and why he’s doing it that way. He’s leveraging some nvme drives within vdevs to help his pools. With his stuff, it makes sense. But for most of us nvmes are a waste of time and money.
Yes max out the ram. The zfs cache will take most of it, speeding up redundant transfers. 32gb of whatever slow ram feeds that processor should be cheap.
For your boot drives I recommend mirrored ssd’s. They don’t need to be large, a pair of 128’s is more than plenty. Use the motherboard sata ports.
Until you have a 10g or faster lan, all your stuff will be fast enough to keep the pipe full.
I would skip the 2x 240GB SATA SSDs and use the NVMe drive for your apps pool (and replicate it to HDD as a backup).
My vote for the HDDs would be 2x 8-wide RAIDZ2 which you can either expand one-by-one to 2x 12-wide or add another 8-wide vDev of a different sized drive.
You will be surprised by how well a 4th gen i5 will perform. You can certainly do more with this than you think (e.g. run Plex rather than just dlna).
I have a lower powered 2-core system which runs plex and my network management just fine, and gets 99.8% ARC hit rate despite only having 10GB of memory.
