Planned first NAS build (virtualized?) for VM Storage

TrueNAS General
,
1

Hi all!

I’m looking into building my first NAS. Use case is not typical file storage, but instead VM image storage via NFS for my homelab XCP-ng cluster. Goals are to slowly teach myself physical server hardware, networking, virtualization, containerization, etc.

What matters:

  • Low Power: Ideally low power draw at idle, mainly to minimize heat output
  • Endurance: More enterprise-y TDW both for peace of mind
  • Performance: Random IOPS

What doesn’t really matter:

  • Capacity: I don’t need much capacity, since not storing large files
  • Performance: Streaming Bandwidth would only really matter during VM migration, which would be infrequent

Current plan:

  • OS: TrueNAS Scale - to better learn ZFS, seems straightforward to setup
  • ZFS Pool architecture: mirrored VDEVs, with a mirrored NVMe SLOG for fast writes
  • Hypervisor: current plan is to virtualize TrueNAS on top of XCP-ng (and pass through the disks for direct access).
    • Allows for a “single pane of glass”, assuming I do the same on my other hosts
    • Ease of backing up the OS, and restoring it if needed
    • Ease of migrating to beefier hardware in the future (I believe this should be the case)
  • Hardware:
    • Motherboard: Supermicro X11SDV-4C-TP8F
      • CPU: D-2123IT Xeon D Skylake, 4C / 8T, 2.2Ghz / 3GHz turbo
      • Networking: 2x 10Gbe RJ45, 2x SFP+, 4x 1G
      • Storage/expansion connectivity: 12x SATA 6 Gbps ports, 1x M.2, 1x PCIe 3.0 16x, 1x PCIe 3.0 8x
    • RAM: 256GB ECC Registered DDR4
    • Storage:
      • OS: Pair of SATADOMs in RAID1
      • Data VDEV: 4x enterprise SATA SSDs (perhaps each with ~1.92/2TB capacity) in a mirrored VDEV striped 2 VDEVs wide
      • SLOG: 2x mirrored used enterprise NVMe SSDs (super small capacity needed here, if I did my research correctly)

Open Questions:

  1. Are used enterprise SATA SSDs a good call here? Seems like not much of a premium over consumer SATA SSDs, and should give good resiliency and performance.
  2. Should I instead go for two larger drives? E.g. instead of getting 4x 2TB drives, get 2x 4TB drives? I’m unsure if there is a “size sweet spot” for the prices of used drives; will have to do some more digging here.
  3. Any particular brands or models of enterprise SATA SSDs to prefer, or avoid?
  4. Is the CPU (4C/8T, 2.2->3.0GHz) sufficient to not be a big bottleneck? I don’t fully understand what CPU speeds are needed for fast network transfer. E.g. - if I had something in ARC, and put a 25GbE card in here - would I be able to transfer at full line rate, or would the CPU (or something else entirely) hamstring things?
  5. TrueNAS Core vs Scale: nothing against BSD per se, but feels like standardizing on Linux might be prudent?
  6. If I know I’m going to want much larger “storage focused” NAS (for media / pictures / etc.) at 50TB+ in the future - is it silly to build this small stopgap NAS in the interim, or might it work well in that I could eventually backup this small NAS to the big one?

Appreciate any thoughts on the above, as it’s still taking form! :slight_smile: