Hello there, after testing TrueNAS for some time I specced out a build and am looking for a second set of eyes to review this to see if I missed anything.
Requirements
This is intended for mixed personal and homelabbing use. Think your typical media and file storage on the personal side. And a small k8s cluster using this as object storage on the homelab side.
- Shared drive as a replacement for Google Drive. Small but should be reliable and fast file storage for family and company NAS use.
- Object storage An S3 interface for object storage. Must be able to saturate a clustered read up to 10Gbit. Should process 1TiB in <15mins. large sequential reads.
- Fast storage for applications. This one is not super important. Would like to move over some small applications that use things like Redis and Postgres to this machine if possible. Not a hard requirement though.
- Media storage mainly for large files and linux ISOs. 250MB/sec (2.5Gbit) single client read is sufficient
General Idea
My thinking on what I should prioritise, in order of importance:
- ARC & memory. Providing zfs with as much ARC as possible should speed up things. I’m looking at ECC memory. I’ve seen people do builds without ECC memory, but was thinking it would be risky since zfs caches lots of things in RAM. Adding ECC to the requirements list had big budget implications though.
- Special VDEV & NVMe SSDs. My intention is to put metadata and small files on two mirrored nvme SSDs. Currently I have a network share on spinning rust and waiting for directory contents to load takes ages.
- SLOG: Wasn’t 100% sure about this one. If I exclude requirement (3)(database storage) from my requirements there are no workloads that require synchronous writes or fast random i/o. Planning to add two mirrored nvme SSDs as SLOG, to speed up Redis and Postgres writes in case I want to migrate those services to this machine.
- L2ARC: Not planning on adding this. I’m banking on 6x spinning rust HDDs rated at ~250mb/s sequential read to saturate large sequential reads on my object store at 10Gbit.
- Storage: 6x spinning rust HDDs in RAID 10 or Z2
Planned build
| type / VDEV | Part | layout | expectation | cost (total, EU-based) |
|---|---|---|---|---|
| Storage | 4x Seagate Exos X18 (SATA, Standard model), 16TB + 2x Seagate Exos X16 (SATA, Standard model), 16TB | z2 | can saturate 10gbit sequential read of large files. | none. have these lying around |
| RAM / ARC | 4x Lenovo 46W0833 DDR4 32GB ECC 2400 MHz | / | slow, but really cheap and speed does not matter as much? | used. EUR 160 |
| special vdev | 2x Samsung PM983 2,5" 960GB U.2 | mirror | speed up document share | used. EUR 200 |
| ZIL | 2x Samsung PM983 2,5" 960GB U.2 | mirror | speed up sync database writes | used. EUR 200 |
| Boot | 2x slow m2 ssd | mirror | / | ? |
| Motherboard | ASRock Rack X570D4U-2L2T/BCM | / | / | new. EUR 500 |
| CPU | AMD Ryzen 5 Pro 4655G | / | pci3 support for nvme ssds / ecc support / am4 socket / Looking at PRO series since they claim lower idle power usage | new. EUR 180 |
| Network | 2x Ethernet 1Gbps, 2x Ethernet 10G/1G onboard | will try to link / aggregate 2 10gbit rj45 → 2x sfp+ on switch | / | / |
| NVME SSD Carrier Card | Delock PCI Express 4.0 x16 Card to 4 x internal U.2 NVMe SFF-8639 - Bifurcation | pci16 → 4x4x4x4 bifurcation for 4x u2 drives | new. EUR 140 |
Other considerations
- I opted for a last-gen am4 platform since ecc ddr5 memory is really expensive. the am4 mobo is the same price as it’s am5 counterpart but the ddr5 hurts my wallet.
- Trying to keep this build under 1.500 eur including VAT. trying to get refurbished or used where possible
- Will I get away with slow ddr4 ram?
- Total current cost: 1.380
- Other things I considered:
- Intel N100 and N150 boards: Cheap and low idle power. There’s even ones with onboard 10gbit ethernet. I struggled with adding nvme ssds to these though. It would include unorthodox m2 → u2 adapters
- EPYC platform: expensive and high idle power. dont intend to run VMs on this
- AM5 asrock rack platform: ddr5 seemed really expensive.
Thanks!