(yet another) First TrueNAS CE build advice, home server

[professionaltart]

Hello, I have spent the last 6 months procrastinating on re-building my old Windows desktop into the fileserver it was always meant to be, and am hoping for some advice and pointers (to help get me out of my head and actually finish the rebuild)

Base hardware:

• AMD Ryzen 3700x
• 64GB RAM (will eventually go back up to 96GB when some loan DIMMs come home to roost) non-ECC
• Asus X570-E
• Fractal Design R6

Storage:

• 2x NVMe
  • WD SN850x 2TB
  • Crucial T500 2TB
• 4x SAS SSD
  • HGST 100GB (HUSSL4010BSS600)
• 9x SATA HDD
  • all 16TB, mix of Seagate EXO/Iron Wolf, and WD DC HC550

Misc:

• Seagate: 1x 6, 1x10 TB
• Samsung: 1x 1TB 860 EVO, 1x 250GB 850 EVO
• Intel I211-AT 1GbE, Realtek RTL8125-CG 2.5GbE
• Nvidia GTX970

re: Hardware:

• I’m aware of the GPU drivers from TrueNAS 25 onwards no longer supporting a card that old (and of the workaround).
• I’m hoping to purchase a HBA to make the storage side of things work, and potentially down the road a new NIC (that supports 5/2.5/1GbE) and maybe a PCIe card/switch (if I end up finding good deals on u.2 drives).
  • short of ‘probably a Broadcom-based 9305-16i or 9400-16i’, I haven’t really been able to narrow down what HBA to actually purchase, or if any of my existing drives are going to cause problems - only that whatever I get needs to support TRIM for flash storage

re: Software:

• I’m hoping to be able to setup some VMs for Windows/Linux/FreeBSD desktops/servers - is this likely to be an issue with how things stand on 25/26?

re: ZFS:

• my broad hope is to try and set the 16TB drives up in a (quite wide) raidz2 vdev, and the 2TB nvme drives in a mirrored pair.
• I got the 4 SAS drives on a whim earlier in the year when the price looked right; assuming they actually work when connected to a HBA, I was thinking of testing them out for a few months as a two-or-four-way mirrored L2ARC or ZIL set to make sure they’re reliable, and if so, look at using them as a four-way mirror special vdev to host metadata for the HDD vdev, to speed things up a little
• I was thinking of trying to re-use the 250GB EVO disk as a boot drive, knowing it can safely sit on one of the motherboard’s SATA ports

That box - sans SAS drives, and with only 6 of the 14TB drives connected (only have 8 SATA ports onboard) is currently running Win11 pro, with a few Linux VMs (and HomeAssistant) running under Hyper-V, and running natively under Windows I have Plex (media streaming) and Dropbox/Onedrive (backup local copy of some documents and photos that other laptops/phones in the house normally save in the cloud). I have about 40TB of data currently spread across the 6+10TB drives and two 16TB drives, with a full backup living in a ReFS/Storage Space pool I created on the spare 16TB drives while waiting to order a HBA.

My intention is to wipe the Storage Space array once I get a HBA, connect all the drives, and build a vdev out of the empty ones and start copying over data, and then expand the vdev 1-2 times as I’m able to add the last remaining drives. I technically have a box of retired drives of assorted sizes that I can pull back into temporary service while ferrying data around, if it will definitely be a lot better to not expand the vdev/that many times. I’m vaguely aware that 9 isn’t a great number as far as building out a vdev with how stripes/parity are spread across disks; given one is out of action until I can hopefully get it RA’d for warranty, I’m open to it being 8-wide with maybe a hot/cold spare.

I can see a lot of things now available via Catalog | TrueNAS Apps Market but I’m honestly happy to run things inside of VMs, as long as standing up the VMs themselves isn’t too complicated.

Any thoughts/advice? I’m based in Australia and don’t see myself travelling to the US anytime soon, so my hardware options (especially on the secondhand enterprise scene) are a little more restricted/expensive than otherwise - but I’m happy to import where I need to.

I appreciate I’ve not spoken about actual pool/dataset layout, and TBH I’ve spent most of my available time trying to get the hardware in place before I look into the best ways to structure that data.

(edited to add (forgot earlier): I figure HDDs for bulk media storage and device backups, and NVMe for virtualised servers and services. I don’t think anything I have qualifies as supporting true PLP, but the device is sitting on a 2RU 1500Va UPS that only needs to run for 5 minutes before the generator autostarts. I am mildly concerned that my (higher-end but consumer) non-SAS SSDs are going to get destroyed under a COW filesystem)

[Farout]

My 2 cents:

With gaming motherboards you might run into problems with passing through PCI devices to VMs, because often the IOMMU groups are not seperated enough.

Forget realtek NICs. If you want to go faster than 1G, then switch directly to 10G. Intel/chelsio and connectX cards have a proven track record.

Im not a fan of using gaming rigs for a server. There are excellent server boards for AM4 CPUs. Albeit they can be a bit hard to find and/or expensive.

[etorix]

@professionaltart

will eventually go back up to 96GB when some loan DIMMs come home

Oh dear… RAM is now so precious that it gets its own financial system, with loans and compound interests.

@professionaltart

1x 250GB 850 EVO

Here is your boot device.

@professionaltart

potentially down the road a new NIC (that supports 5/2.5/1GbE)

Make that 10G, so an Intel X550 or X710 if you want NBase-T as well. But SFP+ would be even better, network architecture allowing.

@professionaltart

short of ‘probably a Broadcom-based 9305-16i or 9400-16i’, I haven’t really been able to narrow down what HBA to actually purchase

You can actually make do with a 9300-8i and plug the remaining drives to the motherboard.

@professionaltart

setup some VMs for Windows/Linux/FreeBSD desktops/servers

Here go your pair of 2 TB NVMe as a mirror.

@professionaltart

I got the 4 SAS drives on a whim earlier

Well, there’s really no obvious purpose for these.
L2ARC, asssuming you have a use case for it, would best be NVMe.
And SLOG, assuming you do need it (which does not appear to be the case) should really be Optane (DC) or some write-intensive low-latency DC SSD.

@professionaltart

I appreciate I’ve not spoken about actual pool/dataset layout

One 8- or 9-wide raidz2 for storage, a NVMe mirror pool for apps/VMs and you’re set.

@professionaltart

I’m vaguely aware that 9 isn’t a great number as far as building out a vdev with how stripes/parity are spread across disks

9 is not better or worse than 8 or 10. ZFS has long dropped any “magic number” (2^n + p).

@professionaltart

I have Plex (media streaming)

If that includes transcoding, you should look for a cheap Intel ARC dGPU. If not, the sole purpose of the GTX970 dGPU is to let your consumer motherboard boot (TrueNAS itself would happily run headless), and replacing your 3700X by an AM4 APU may simplify the build and free a PCIe slot.

[professionaltart]

Thank you both!

@Farout

With gaming motherboards you might run into problems with passing through PCI devices to VMs, because often the IOMMU groups are not seperated enough.

There isn’t a lot I think I want to be passing through, honestly - I’ve only considered it previously if I ran Proxmox baremetal and wanted to pass the HBA through to TrueNAS.

@Farout

Forget realtek NICs. If you want to go faster than 1G, then switch directly to 10G. Intel/chelsio and connectX cards have a proven track record.

@etorix

Make that 10G, so an Intel X550 or X710 if you want NBase-T as well. But SFP+ would be even better, network architecture allowing.

The Intel is what I expect to rely on for now; I was only thinking of sub-10G because my desktop has a RTL8126 (for upto 5G), to run a cable directly between them - my switches are all old and 1G max (and no SFP). Thank you for those two suggestions, the best I had figured so far was a later revision i226

@Farout

Im not a fan of using gaming rigs for a server. There are excellent server boards for AM4 CPUs. Albeit they can be a bit hard to find and/or expensive.

My goal was to repurpose what I have lying around/working for now - in a year or two when storage prices come back down to Earth, I intend to turn this current box into a backup store, and try to get some actual workstation/server equipment for day-to-day storage and virtual compute.

@etorix

You can actually make do with a 9300-8i and plug the remaining drives to the motherboard.

Good catch - I was still in the mindset of needing a bigger HBA to pass through to TrueNAS I think I’d settled on 9305 over 9300 just for the reduced power draw/heat production.

@etorix

@professionaltart

I got the 4 SAS drives on a whim earlier

Well, there’s really no obvious purpose for these.
L2ARC, asssuming you have a use case for it, would best be NVMe.
And SLOG, assuming you do need it (which does not appear to be the case) should really be Optane (DC) or some write-intensive low-latency DC SSD.

They were relatively cheap for 2026 and I think I read they had quite decent write endurance on them - if for nothing else I was going to use them as a location to download files onto before they’ve moved elsewhere, to try and reduce write amplification impacts on the nvme drives specifically. I don’t think I have a meaningful use-case for L2ARC or a SLOG, hence thinking of using them as a metadata special vdev, or small file cache for the spinners. I’d have preferred to have Optane to play around with, but I missed the train on cheap drives and don’t really have lots of free PCIe lanes available to make use of the small 32GB Optane drives.

[etorix]

@professionaltart

my switches are all old and 1G max (and no SFP)

Make your life easier and upgrade your switch rather than running specific cables at specific speeds between specific devices.

@professionaltart

I think I’d settled on 9305 over 9300 just for the reduced power draw/heat production.

The 9305 is a good choice if you need 16 ports; if you only need 8, a 9300 will do.

@professionaltart

a metadata special vdev

This has its uses, but is a liability and you may run out of PCIe lanes. (SATA SSDs as specicail vdev and/or app pool could be a solution.)

@professionaltart

make use of the small 32GB Optane drives

The smaller M10 make fine boot devices but are somewhat limited in bandwidth and endurance compared to their bigger siblings. For SLOG, you’d want a 900p or P4801X—but I also do not think you have the use case for it.