Hello all, thank you for this community, reading posts here helped a lot in planning my build. I pulled the trigger on a bunch of parts and want to make sure I’m not missing anything.
I will be running Proxmox with TrueNAS Scale, Home Assistant, a debian VM for Jellyfin, Immich, Paperless, and the rest of my 30+ services currently living on various VPS’s, and possibly a couple other VMs (dedicated Orcaslicer VM?).
I plan to use the 4x NVMe in Raidz1 for the other Proxmox VMs and the 6x HDD in Raidz2 for bulk storage (media, backups, etc).
Parts list:
Case: Fractal Design Node 804
CPU: Ryzen 7 Pro 5750G
MoBo: Asrock X570M
RAM: Micron 32 GB DDR4-3200 UDIMM ECC x 4
Boot drive: Patriot P220 128GB SATA SSD
NVMe riser: Asrock Hyper Quad PCIe to 4x M.2
NVMe: WD SN850X 2TB NVME x 4
Power supply: Corsair HX750
HDD: Seagate Exos X18 14TB (OEM) x 6
Is there any downside to not using an HBA for the HDDs other than having to pass them through proxmox individually? My motherboard has 8 SATA ports so I’m not constrained there (yet).
Also, if I wanted to add a GPU later for local AI (LLM, paperless-AI, etc) how much will i suffer with it on PCIe x4? My primary slot is used by the NVMe adapter, since it can bifurcate, and that limits the 2nd slot to 4 lanes.
You absolutely should pass through the controller as a PCIe device—and then your consumer motherboard may not have optimal IOMMU grouping for that. Passing individual disks to ZFS through a hypervisor is known to result in data loss.
Thank you! I was able to find an IOMMU dump for the X570M Pro4 (doesn’t let me post the link here?). It looks like the SATA ports are split between 2 controllers. Also noticed that my bifurcation is limited to 8x4x4, so new plan:
Ditch the 120GB SATA boot drive
Use 1 motherboard M.2 for small boot drive
Passthrough both SATA controllers
Passthrough 2nd motherboard M.2 for one of the TrueNAS NVMe drives
Passthrough PCIe x16 slot for M.2 Adapter with remaining 3 M.2 drives for TrueNAS
To bump your trust level, take the tutorial with the forum bot.
Bifurcation is limited to x8x4x4 by your APU; use a Ryzen (desktop) CPU for x4x4x4x4 (but then the consumer motherboard may refuse to boot without a GPU).
Yeah, the APU is limiting me there The Asrock rack or MC12-LE0 boards are going for ~400-500 euro right now so that’s a bit out of reach. I can live with an empty NVMe on the expansion board, since it seems like the IOMMU groups are well split and will let me pass through one of the motherboard slots to give me a 4th drive.
Maybe i should just ignore my bank statements and go with an asrock rack board with a non-APU 5000 series CPU… any downside when i plan on it being headless anyway? No windows VM is what comes to mind
Proxmox will eventually eat those WD SN850X drives as well as the Patriot 220. They don’t have the durability (DWPD) to withstand the write amplification of ZFS for very long. You should expect them to wear out in a year or so, maybe less. In addition, I am personally very wary of using drives that don’t have power loss protection in Proxmox. I am also not sure I would use raidz1 for the boot drive or the VMs. Mirrored VDEVS will allow Proxmox to keep running if a drive fails. I am pretty sure raidz1 won’t.
My server is similar to the one you are building. I also run Proxmox with TrueNAS virtualized. I run two mirrored VDEVs (sort of the ZFS version of raid 10) with four 1TB Samsung sm863a enterprise SSDs that I picked up off of ebay for $43 each. I use that for both my boot drive and all of my VMs. It works quite well and is plenty fast enough for running the VMs. That being said, I don’t run that many VMs: I have one for TrueNAS, two for docker containers (one on my trusted VLAN and one on my externally facing VLAN), one for my Discourse forum and two more for Wordpress websites. I run something like 24 different docker containers. But my VMs are all very small because I keep my data on a separate NAS using NFS mounts. I think my largest VM is 64gb, so there’s no way I would ever use more than 1TB of storage for VMs, if that.
Ok so I swapped out the x570m Pro4 for a x570d4u and the 5750G for a 5950x. I will set up the NVMe drives in a mirror.
@Louie1961 - so your SSDs are setup as storage in Proxmox then, not TrueNAS? My plan is to pass them through to TrueNAS then create a bridge network back to Proxmox to mount as NFS. Won’t actually touch the NIC so should be as fast as the drives themselves. That way I have everything (except the small boot drive) in TrueNAS.
I have SSDs on both sides. There are four that Proxmox owns, then I pass through a complete 6 port M.2 to SATA adapter to TrueNAS and the SSDs that TrueNAS uses are connected to that. I keep them separated so that if TrueNAS goes down (or I take it down for some reason) my Proxmox and all the other VMs will keep running.
Ok, after reading all these horror stories about proxmox grabbing zfs pools before truenas boots, I’m going to run truenas baremetal. I’ll suffer through any performance penalty on VMs running inside truenas for the peace of mind with my data. Thanks everyone! Hardware is slowly arriving, looking forward to putting it all together!
The Asrock rack or MC12-LE0 boards are going for ~400-500 euro right now so that’s a bit out of reach. I can live with an empty NVMe on the expansion board, since it seems like the IOMMU groups are well split and will let me pass through one of the motherboard slots to give me a 4th drive.