I’m planning on switching to a new system for my homelab/media server. For context, I am a software engineer working on macOS but leaning more and more towards Linux. Currently using a Synology but starting to feel limited by the OS and also I gained experience over the years so this no longer fits my needs, which is why I naturally had a look at TrueNAS and it seems like an obvious choice for my future build.
The plan is to run Proxmox VE as the host, with as main VMs:
TrueNAS SCALE
A linux VM to run my Docker stack (Plex, the *arrs stack, torrenting)
Another VM for Home Assistant
Maybe a couple of small VMs to test linux distros, but nothing really heavy.
I know I can run Docker inside TrueNAS, but I prefer to have a dedicated VM with a full Linux OS dedicated to it. I know I can run containers inside Proxmox directly but I’m used to Docker and so I don’t think I need to switch to Proxmox containers.
This is the setup that I found will bring me the more flexibility, and I believe it is not uncommon for people to run TrueNAS through Proxmox.
Would love to hear any pros and cons or things I should be aware of when going with this setup.
Regarding the hardware, here’s what I’m looking at:
Proxmox OS: mirrored NVMe on the 2 on-board M.2 slots
TrueNAS OS: NVMe added via a PCIe → M.2 adapter (or SATA if simpler?)
Docker: 1–2 TB NVMe (mirrored?) on the same adapter (or another PCIe slot?)
Temporary download folder: I want to have my temporary downloading folder (incomplete) on NVMe (the completed files will be on the HDDs), not sure if this can be on the same physical drive as my Docker stack?
Potential VMs: Dedicated NVM.e drive or not?
HDDs: starting with 3–4 × WD Red Pro drives (on the 8 on-board SATA ports), eventually growing up with an HBA later (the Define 7XL can handle up to 20 HDDs)
My main questions or points of attention are the following:
I picked this board and CPU mainly for **expandability and future-proofing. I wanted something on the SP5 platform with 10 GbE built in that I can keep upgrading over time. I have a lot of PCIe slots so I’ll even be able to add an Intel Arc GPU later if I need it for transcoding for example. Same for the Case which can contain a lot of HDDs, the plan is to keep this setup for many years and add drives, RAM and update the CPU when needed.
Realistically how much RAM can I start with, given my use cases? I understand this is an ambitious setup and also a pretty expansive one, so I would love not to have to start with 128GB.
The storage plan is a bit messy at the moment, I’m not exactly sure what should be isolated on the hardware side, I believe it’s best to have Proxmox OS and TrueNAS OS on 2 dedicated drives but not 100% sure, same for the Docker/Temporary ownload folder thing. I’m thinking it is probably best to have everything mirror when it comes to the NVM.e but again I’m not sure at all, maybe it’s not needed and it is overkill?
Lastly, since it will be my first build first using TrueNAS (and Proxmox too for the occasion), any overall remarks, feedback or advice regarding this potential build would be highly appreciated.
I’ve read about HBA Passthrough but my logic was “I will get an HBA when I have more drives than I have SATA slots in my board” but it seems I was wrong and should consider the HBA directly to configure the main ZFS Pool in TrueNAS with HBA Passthrough.
This is helping me quite a bit for the “messy” Storage plan.
Let me know if this looks better:
TrueNAS Data Pool: I want the ZFS Pool that will manage my main data storage from all my HDDs to go through the HBA Passthrough.
Proxmox OS: goes on the 2 onboards M.2 slots and use ZFS mirror (small size M.2s)
TrueNAS OS: goes on a simple PCIe→M.2 adapter as virtual disks assigned to the TrueNAS VM (managed by Proxmox) (small sized M.2s too)
VMs / Docker / Temporary download: double larger M.2 drives added to Proxmox as a separate local ZFS pool?
Does this make more sense? Am I thinking this correctly?
TrueNAS OS can be a small virtual drive alongside other VMs, no need for a dedicated drive.
In theory, passing through the SATA controller is a valid alternative to using a dedicated HBA—assuming, of course, that no SATA devace is used by the host.
TrueNAS OS can be a small virtual drive alongside other VMs, no need for a dedicated drive.
In theory, passing through the SATA controller is a valid alternative to using a dedicated HBA—assuming, of course, that no SATA devace is used by the host.
Okay, so it’s even simpler? I got my ZFS Data Pool with HDDs which is managed by TrueNAS and my ZFS NVM.e Pool where I handle my TrueNAS boot / Docker / VMs etc managed by Proxmox ?
Why SP5 and 9000 series over SP6 and 8000?
As I understand it from my researches, the SP5 platform offers 12-channel memory and more PCIe lanes than the SP6. So while SP6 would allow me to save costs (I believe it will be overall cheaper), my max expandability (in memory bandwidth, PCIe lanes) will be somewhat reduced compared to a SP5 platform.
The question is: is SP5 overkill for my use case or not? As I said, I plan to expand and add more drive + potentially a GPU for transcoding. But in terms of VMs, I don’t plan have tons of them running constantly.
I was thinking about what I said there and I think for temporary download (torrents), it should be a 2nd Pool in TrueNAS.
I know about ARC and the L1 and L2 cache in terms of read but I think it still make sense to have a NVM.e Pool in terms of I/O and write, no?
Also, for boot drives (Proxmox and TrueNAS), is it best to mirror both or not needed? + Are NVM.e preferred? From my researches, I think I saw that Proxmox would benefit from NVM.e but TrueNAS not so much.
